Diploma In Apache Kafka

Kafka is an open-source stream processing platform. Kafka can be integrated with Spark, Storm and Hadoop. Learn about Kafka Architecture, setup Kafka Cluster, understand Kafka Stream APIs, and implement Twitter Streaming with Kafka, Flume, Hadoop and Storm.

Who Should Do?

Bachelor Of Technology, Bachelor Of Engineering, Bachelor Of Science, Bachelor Of Computer Application, Master Of Technology, Master Of Engineering, Master Of Science, Master Of Computer Application.

Modules Covered:

• MODULE 1 – Introduction to Big Data and Apache Kafka
• Introduction to Big Data
• Big Data Analytics
• Need for Kafka
• What is Kafka?
• Kafka Features
• Kafka Concepts
• Kafka Architecture
• Kafka Components
• ZooKeeper
• Where is Kafka Used?
• Kafka Installation
• Kafka Cluster
• Types of Kafka Clusters
• MODULE 2 – Kafka Producer
• Configuring Single Node Multi Broker Cluster
• Constructing a Kafka Producer
• Sending a Message to Kafka
• Producing Keyed and Non-Keyed Messages
• Sending a Message Synchronously & Asynchronously
• Configuring Producers
• Serializers
• Serializing Using Apache Avro
• Partitions
• MODULE 3 – Kafka Consumer
• Consumers and Consumer Groups
• Standalone Consumer
• Consumer Groups and Partition Rebalance
• Creating a Kafka Consumer
• Subscribing to Topics
• The Poll Loop
• Configuring Consumers
• Commits and Offsets
• Rebalance Listeners
• Consuming Records with Specific Offsets
• Deserializers
• MODULE 4 -Kafka Internals
• Cluster Membership
• The Controller
• Replication
• Request Processing?
• Physical Storage
• Reliability
• Broker Configuration
• Using Producers in a Reliable System
• Using Consumers in a Reliable System
• Validating System Reliability
• Performance Tuning in Kafka
• MODULE 5 - Kafka Cluster Architectures & Administering Kafka
• Use Cases - Cross-Cluster Mirroring
• Multi-Cluster Architectures
• Apache Kafka’s MirrorMaker
• Other Cross-Cluster Mirroring Solutions
• Topic Operations
• Consumer Groups
• Dynamic Configuration Changes
• Partition Management
• Consuming and Producing
• Unsafe Operations
• MODULE 6 - Kafka Monitoring and Kafka Connect
• Considerations When Building Data Pipelines
• Metric Basics
• Kafka Broker Metrics
• Client Monitoring
• Lag Monitoring
• End-to-End Monitoring
• Kafka Connect
• When to Use Kafka Connect?
• Kafka Connect Properties
• MODULE 7 - Kafka Stream Processing
• Stream Processing
• Stream-Processing Concepts
• Stream-Processing Design Patterns
• Kafka Streams by Example
• Kafka Streams: Architecture Overview
• MODULE 8 - Integration of Kafka With Hadoop, Storm and Spark
• Apache Hadoop Basics
• Hadoop Configuration
• Kafka Integration with Hadoop
• Apache Storm Basics
• Configuration of Storm
• Integration of Kafka with Storm
• Apache Spark Basics
• Spark Configuration
• Kafka Integration with Spark
• MODULE 9 - Integration of Kafka With Talend and Cassandra
• Flume Basics
• Integration of Kafka with Flume
• Cassandra Basics such as and KeySpace and Table Creation
• Integration of Kafka with Cassandra
• Talend Basics
• Integration of Kafka with Talend