Event driven Architecture with Kafka
Program Details
Topic Apache Kafka & Streaming Distributed Messaging Training
Scope Confluence Kafka Enterprise 5.x Kafka
2.x Version
Level Intermediate and Advanced
Languages Java 1.8 64 bit
Lab Setup
Host Machine 64bit Windows/Linux/Mac [Linux is preferred]
Virtual 64bit Linux OS [Mint/Ubuntu OS Based]
Machine
Host Machine 16 GB if developer wants to use Virtual Machine
RAM
RAM 8 GB if developer can use Linux/Mac machine directly without Host Machine
Setup 16 GB RAM
Requirements 40-50 GB GB Free harddrive
OS: Windows 10/Ubuntu Desktop/Mac
Cloud Trainer provides cloud machine for each participant with 160 GB SSD Harddrive, 4
Machine vCPU and 8 GB RAM for participants, cost about 5 Euros per participant. The VM shall
be only for training period. Maximum 15 VMs provided, team need to share
themselves.
For Cloud, the setup shall be done by trainer.
Lap Setup 1. Java 1.8 Java SDK from
https://www.oracle.com/technetwork/java/javase/downloads/jdk8downloads-
2133151.html
2. Intellij Community Edition for Java Development
https://www.jetbrains.com/idea/download/
3. https://www.putty.org/ or https://gitforwindows.org/
4. https://www.confluent.io/download download the Confluent platform free,
need to provide email id.
Event driven Architecture with Kafka
Participants Pre-requites
JDK Good Knowledge on Using JDK, Compiler Setup & Java.
Good Working knowledge on Java /Exposure to Reactive
Programming for Streaming
Maven & POM XML Good Knowledge on Maven & POM XML files handling
Highspeed Internet Access to Highspeed Internet
Editor Good expertise in using IntelliJ/Eclipse Editor
Preferred, but not High level understanding of AMQP
mandatory Understanding Producer and Consumer Design Patterns
Message Queuing
Distributed Systems
Stream vs Batch Processing
SSH Gitbash for windows
SSH client for Linux/Mac/Windows
Day 1
Event driven Architecture with Kafka
Introduction Introduction
Kafka Introduction
Kafka Features
Kafka vs AMQP
Kafka Features
Kafka Real Use Cases
Enterprise Software Digital Software Transformation From Monolithic to
Architecture MicroServices to Serverless Applications
Event-Driven Architecture ( EDA )
- MicroServices
- Serverless
- FaaS
- Streaming
- Event Sourcing
- CQRS
Reactive Manifesto
Kafka Case Studies - MicroServices Log Analysis
- Invoice Processing
- Distributed Transaction Management with SAGA
Pattern
- Data Migration between databases
Kafka Architecture Elements of Kafka [Consumer, Producer, Broker, ZooKeeper,
Cluster]
Kafka Architecture
Kafka Overall Messaging Architecture
Kafka Streaming Architecture
Kafka Consumer and Producer Architecture
Kafka Persistence Architecture
Kafka Clusters
Kafka Consumer Workers Group
Event driven Architecture with Kafka
Kafka Distribution Apache Kafka
Confluence Kafka Open Source
Confluence Kafka Enterprise
Kafka Setup Java JDK
Kafka Setup
[Hands-on] Zookeeper Configuration
Single Broker Configuration
Understanding build-in tools, Kafka server, test publisher, test
consumer
Configuration Kafka Single Node Broker
Start ZooKeeper
Start Kafka Server
Using Example console consumer and publisher
Broker, Consumer, Producer Configuration
Zookeeper connection
Broker ID
Logs
Port
Consumer Configuration
Producer Configuration
Kafka Administrator List all topics
Commands List all brokers Create Topics
Change Topics
[Hands-on] Delete Topics
Topic Description
Topics & Partitions Understanding Topics
Understanding Partitions
Understanding Replications
Event driven Architecture with Kafka
Producer Kafka Producer Introduction
Kafka Producer API
[Hands-on] Writing First Producer
Kafka Producer Class
Build a Kafka Producer
Configure Producers
Kafka Topics, Partitions Configuration
Send messages to Kafka
Send messages Synchronously
Send Message Asynchronously and process the async
acknowledgement
Producing Keyed and Non-Keyed Messages
Serializers, Key/Value Serializer, ProducerRecord
Serialize Using Apache Avro, JSON
Working with Multiple Brokers
Consumer [Hands-on] Kafka Consumer Introduction
Consumers and Consumer Groups
Subscribe for Topics
Consumer Records
Achieving higher performance with Consumer group De-
serialize key and values
Consumer Groups and Partition Rebalance
Creating a Kafka Consumer
The Poll Loop
Configuring Consumers
Commits and Offsets
Rebalance Listeners
Consuming Records with Specific Offsets
Kafka Internal Architecture, Cluster Membership
Storages, Cluster, Partitions The Controller
in Depth Replication workflow, Replication Internals
Request Processing
Physical Storage
Reliability
Partition allocation and number of partitions
Broker Configuration
Using Producers in a Reliable System
Using Consumers in a Reliable System
Validating System Reliability
Performance Tuning in Kafka
Event driven Architecture with Kafka
Day 2
Cluster Architecture and Revising Cluster, Producer, Consumer, Broker, ZooKeeper
Workflow
Data Flow Between all actors in Kafka Systems
Creating Clusters with Multiple Brokers
Broker IDs and Master/Slave Brokers
Fail-over handling in clusters
Understanding Leader Broker
Balancing leadership
Cluster Administration and Configure Cluster
Configuration Replication
[Hands-on] High Availability
Multi-Cluster Architectures
Apache Kafka’s MirrorMaker
Topic Operations
Partition in Depth [Hands-on] Topics and Partitions
Understanding partition numbers
Default partitions
Explicit Partitions
Under the hood partitions
Replications Understanding Replications
Replication Factors and High Availability
Messages in Details Message Models
Message Serialize
[Hands-on] JSON handling
Message IDs
Day 3
Persistence [Hands-on] Persisting Messages on File System
Offline Consumer and Subscription handling
Durability of Messages
Failover and Persistence
Event driven Architecture with Kafka
Kafka Streaming Stream Processing
Stream-Processing Concepts
[Hands-on] Stream-Processing Design Patterns
Kafka Streams by Example
Kafka Streams: Architecture Overview
KStream
KTable
GlobalKTable
KStream to KTable
KTable to KSTream
Stateful/Stateless stream operation
KSQL Kafka and KSQL
KStream and KTable
Creating and running persistent and non-persistent quries
Zoo Keeper Administration Managing Zookeeper Cluster
Zookeeper lead, configurations
[Hands-on]
Day 4
Kafka Monitoring Considerations When Building Data Pipelines
Metric Basics
Kafka Broker Metrics
Client Monitoring
Lag Monitoring
End-to-End Monitoring
Kafka Connect APIs + Event Kafka Connect API overview Kafka
Sourcing + Elastic Search Connect
When to Use Kafka Connect?
[Hands-on] Kafka Connect Properties
Perform File source and sink using Kafka Connect
Kafka Connect vs Consumer/Producer model
Kafka Connect with JDBC [Requires PostgreSQL/MySQL]
Using Kafka with ElasticSearch
Event sourcing using Debizium with PostgreSQL
Kafka Connect architecture and design
Connectors, Configurations, Tasks, Workers
Stand-alone vs distributed mode
Event driven Architecture with Kafka
REST Proxy Managing REST Proxy Configuration
[Hands-on]
Kafka Connect Configuration Managing Kafka Connect
& Administration Managing Sink/Sources
Configuration and settings
[Hands-on]
Confluence Control Centre Introduction to Confluent Control Center
Managing Control Center and Dashboard Kafka
[Hands-on] Monitoring
Schema Registry Managing Schema Registry services
Configuration &
Administration
[Hands-on]
Kafka with Spring Spring-Kafka module
Configuring KafkaTemplate
Produce & Consume Messages with Kafka Template
Spring-cloud-stream with Kafka binder
KStream
KTable
Production Deployment
Configuration, cleaner installation
