KEMBAR78
Kafka 101 and Developer Best Practices | PDF
confluent, profile picture
Uploaded byconfluent
1,883 views

Kafka 101 and Developer Best Practices

The document provides an overview of Kafka, highlighting its developer best practices and the event streaming paradigm that enables efficient data integration. It details the architecture of Kafka, including producers, consumers, brokers, and topics, and emphasizes the importance of using schemas and configuring producer properties for data consistency and performance. Confluent is positioned as a key player in the enterprise event streaming market, enhancing Kafka's capabilities for large-scale deployments.

Technology◦
Related topics:
Apache Kafka•
In this document
Powered by AI

Overview of Kafka, its purpose, and agenda covering Kafka 101 and best practices for producers and consumers.

Comparing batch processing and messaging systems; highlighting drawbacks of batch such as high cost, data loss, while emphasizing advantages of messaging.

Introduction to event streaming, encouraging a shift in perspective towards data as continuous streams instead of stored records.

Describes applications like Customer 360 and Financial Fraud Detection; Confluent's solutions enable event streaming success.

Basics of Kafka architecture including brokers, consumers, producers, and its distributed nature.

How to create topics in Kafka and the basics of topic management such as partitions and replication factors.

Insight into data production to Kafka, including producer design, serialization, and importance of record keys for partitioning.

Utilizing schema registries for managing data encode evolution and ensuring compatibility within services.

Delves into producer properties and guarantees like acks, idempotence, and handling duplicates for consistent data delivery.

Overview of transactional producers in Kafka; how to set up transactions for reliable message delivery.

How consumers retrieve data from Kafka topics, illustrating single and grouped consumers for load balancing.

Usage of interceptors for producers and consumers for enhanced observability; advises against pooling connections.

Provides additional learning resources about Kafka and event processing; concludes with thank you.

Kafka 101 & Developer Best Practices
Agenda ● Kafka Overview ● Kafka 101 ● Best Practices for Writing to Kafka: A tour of the Producer ● Best Practices for Reading from Kafka: The Consumer ● General Considerations
3 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
4 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
5 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records
6 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records Both of these are a complete mismatch to how your business works.
7 ETL/Data Integration Messaging Transient Messages Stored records ETL/Data Integration Messaging Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm High Throughput Durable Persistent Maintains Order Fast (Low Latency)
8 Fast (Low Latency) Event Streaming Paradigm To rethink data as not stored records or transient messages, but instead as a continually updating stream of events
9 Fast (Low Latency) Event Streaming Paradigm
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. ... Device Logs ... ... ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Customer 360 Financial Fraud Detection Real-time Risk Analytics Real-time Payments Machine Learning Models ... Event-Streaming Applications Universal Event Pipeline Amazon S3 SaaS apps Confluent: Central Nervous System For Enterprise
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Confluent uniquely enables Event Streaming success Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS Confluent founders are original creators of Kafka Confluent team wrote 80% of Kafka commits and has over 1M hours technical experience with Kafka Confluent helps enterprises successfully deploy event streaming at scale and accelerate time to market Confluent Platform extends Apache Kafka to be a secure, enterprise-ready platform
Kafka 101
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Scalability of a Filesystem Guarantees of a Database Distributed By Design Rewind and Replay
15 Kafka Broker Kafka Broker Kafka Broker Kafka Broker Kafka Producer Kafka Producer Kafka Producer Kafka Consumer Kafka Consumer (grouped) Kafka Consumer /Producer Kafka Broker Writers Readers Kafka Cluster
KAFKA A MODERN, DISTRIBUTED PLATFORM FOR DATA STREAMS
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Creating a topic $ kafka-topics --zookeeper zk:2181 --create --topic my-topic --replication-factor 3 --partitions 3 Or use the AdminClient API!
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time C C C
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
Producing to Kafka
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Clients - Producer Design Producer Record Topic [Partition] [Key] Value Serializer Partitioner Topic A Partition 0 Batch 0 Batch 1 Batch 2 Topic B Partition 1 Batch 0 Batch 1 Batch 2 Kafka Broker Send() Retry ? Fail ? Yes No Can’t retry, throw exception Success: return metadata Yes
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer Kafka doesn’t care about what you send to it as long as it’s been converted to a byte stream beforehand. JSON CSV Avro Protobufs XML SERIALIZERS 01001010 01010011 01001111 01001110 01000011 01010011 01010110 01001010 01010011 01001111 01001110 01010000 01110010 01101111 01110100 ... 01011000 01001101 01001100 (if you must) Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer private Properties kafkaProps = new Properties(); kafkaProps.put(“bootstrap.servers”, “broker1:9092,broker2:9092”); kafkaProps.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”); kafkaProps.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); kafkaProps.put("schema.registry.url", "https://schema-registry:8083"); producer = new KafkaProducer<String, SpecificRecord>(kafkaProps); Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Record Topic [Partition] [Key] Value Record keys determine the partition with the default kafka partitioner If a key isn’t provided, messages will be produced in a round robin fashion partitioner Record Keys and why they’re important - Ordering
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] AAA Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] BBB Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] CCC Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] DDD Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Record Keys and why they’re important - Key Cardinality Consumers Key cardinality affects the amount of work done by the individual consumers in a group. Poor key choice can lead to uneven workloads. Keys in Kafka don’t have to be primitives, like strings or ints. Like values, they can be be anything: JSON, Avro, etc… So create a key that will evenly distribute groups of records around the partitions. Car·di·nal·i·ty /ˌkärdəˈnalədē/ Noun the number of elements in a set or other grouping, as a property of that grouping.
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101” } You don’t have to but... use a Schema! Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send JSON “Where’s record.City?” Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
Schema Registry: Make Data Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic ! Schema Registry Open Source Feature
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101”, “City”: “NA”, “State”: “NA” } Avro allows for evolution of schemas Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send AvroRecord Schema Registry Version 1 Version 2 Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
Developing with Confluent Schema Registry We provide several Maven plugins for developing with the Confluent Schema Registry ● download - download a subject’s schema to your project ● register - register a new schema to the schema registry from your development env ● test-compatibility - test changes made to a schema against compatibility rules set by the schema registry Reference https://docs.confluent.io/current/schema-registry/docs/maven-plugin.html <plugin> <groupId>io.confluent</groupId> <artifactId>kafka-schema-registry-maven-plug <version>5.0.0</version> <configuration> <schemaRegistryUrls> <param>http://192.168.99.100:808 </schemaRegistryUrls> <outputDirectory>src/main/avro</outp <subjectPatterns> <param>^TestSubject000-(key|valu </subjectPatterns> </configuration> </plugin>
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Use Kafka’s Headers Reference https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers Producer Record Topic [Partition] [Timestamp] Value [Headers] [Key] Kafka Headers are simply an interface that requires a key of type String, and a value of type byte[], the headers are stored in an iterator in the ProducerRecord . Example Use Cases ● Data lineage: reference previous topic partition/offsets ● Producing host/application/owner ● Message routing ● Encryption metadata (which key pair was this message payload encrypted with?)
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=0 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 ack Producer Properties acks=1 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234 data: abcd} - offset 3345 Failed ack Successful write Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234, data: abcd} - offset 3345 {key: 1234, data: abcd} - offset 3346 retry ack dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - with exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties enable.idempotence=true max.inflight.requests.per.connection=5 acks = “all” retries > 0 (preferably MAX_INT) (pid, seq) [payload] (100, 1) {key: 1234, data: abcd} - offset 3345 (100, 1) {key: 1234, data: abcd} - rejected, ack re-sent (100, 2) {key: 5678, data: efgh} - offset 3346 retry ack no dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Transactional Producer Producer T1 T1 T1 T1 T1 KafkaProducer producer = createKafkaProducer( “bootstrap.servers”, “broker:9092”, “transactional.id”, “my-transactional-id”); producer.initTransactions(); -- send some records -- producer.commitTransaction(); Consumer KafkaConsumer consumer = createKafkaConsumer( “bootstrap.servers”, “broker:9092”, “group.id”, “my-group-id”, "isolation.level", "read_committed"); Reference https://www.confluent.io/blog/transactions-apache-kafka/
Consuming from Kafka
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. A basic Java Consumer final Consumer<String, String> consumer = new KafkaConsumer<String, String>(props); consumer.subscribe(Arrays.asList(topic)); try { while (true) { ConsumerRecords<String, String> records = consumer.poll(100); for (ConsumerRecord<String, String> record : records) { -- Do Some Work -- } } } finally { consumer.close(); } }
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Single Consumer C
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C1 C C C2
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C C
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0, 3 1 2 3
Kafka’s Interceptors ProducerInterceptor onSend(ProducerRecord<K, V> record) Returns ProducerRecord<K, V> . Called from send() before key and value get serialized and partition is assigned. This method is allowed to modify the record. onAcknowledgement(RecordMetadata metadata, java.lang.Exception exception) This method is called when the record sent to the server has been acknowledged, or when sending the record fails before it gets sent to the server. Used for observability and reporting. ConsumerInterceptor onConsume(ConsumerRecords<K,V> records) Called just before the records are returned by KafkaConsumer.poll() This method is allowed to modify consumer records, in which case the new records will be returned. onCommit(Map<TopicPartition,OffsetAndMetada ta> offsets) This is called when offsets get committed. Used for observability and reporting Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/producer/ProducerInterceptor.html Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/consumer/ConsumerInterceptor.html
Should I pool connections? NO! Since Kafka connections are long-lived, there is no reason to pool connections. It’s common to keep one connection per thread.
Use a good client! Clients ● Java/Scala - default clients, comes with Kafka ● C/C++ - https://github.com/edenhill/librdkafka ● C#/.Net - https://github.com/confluentinc/confluent-kafka-dotnet ● Python - https://github.com/confluentinc/confluent-kafka-python ● Golang - https://github.com/confluentinc/confluent-kafka-go ● Node/JavaScript - https://github.com/Blizzard/node-rdkafka (not supported by Confluent!) New Kafka features will only be available to modern, updated clients!
Resources Free E-Books from Confluent! I Heart Logs: https://www.confluent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/ Kafka: The Definitive Guide: https://www.confluent.io/resources/kafka-the-definitive-guide/ Designing Event Driven Systems: https://www.confluent.io/designing-event-driven-systems/ Confluent Blog: https://www.confluent.io/blog Thank You!
Thank you! pascal@confluent.io

More Related Content

PDF
Benefits of Stream Processing and Apache Kafka Use Cases
byconfluent
 
PDF
ksqlDB: A Stream-Relational Database System
byconfluent
 
PPTX
Kafka and Avro with Confluent Schema Registry
byJean-Paul Azar
 
PPTX
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
byJean-Paul Azar
 
PPTX
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
PDF
Getting Started with Confluent Schema Registry
byconfluent
 
PDF
From Zero to Hero with Kafka Connect
byconfluent
 
ODP
Stream processing using Kafka
byKnoldus Inc.
 
Benefits of Stream Processing and Apache Kafka Use Cases
byconfluent
 
ksqlDB: A Stream-Relational Database System
byconfluent
 
Kafka and Avro with Confluent Schema Registry
byJean-Paul Azar
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
byJean-Paul Azar
 
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
Getting Started with Confluent Schema Registry
byconfluent
 
From Zero to Hero with Kafka Connect
byconfluent
 
Stream processing using Kafka
byKnoldus Inc.
 

What's hot

PDF
Fundamentals of Apache Kafka
byChhavi Parasher
 
PDF
Apache Kafka - Martin Podval
byMartin Podval
 
PPTX
Kafka 101
byClement Demonchy
 
PPTX
Introduction to Apache Kafka
byJeff Holoman
 
PPTX
Kafka Tutorial - basics of the Kafka streaming platform
byJean-Paul Azar
 
PPTX
Apache kafka
byKumar Shivam
 
PPSX
Microservices, DevOps & SRE
byAraf Karsh Hamid
 
PDF
Unified Big Data Processing with Apache Spark (QCON 2014)
byDatabricks
 
PPTX
Introduction to Apache Kafka
byAIMDek Technologies
 
PDF
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
PPTX
The Top 5 Apache Kafka Use Cases and Architectures in 2022
byKai Wähner
 
PPTX
Deep Dive into Apache Kafka
byconfluent
 
PDF
효과적인 NoSQL (Elasticahe / DynamoDB) 디자인 및 활용 방안 (최유정 & 최홍식, AWS 솔루션즈 아키텍트) :: ...
byAmazon Web Services Korea
 
PPTX
A visual introduction to Apache Kafka
byPaul Brebner
 
PPTX
Envoy and Kafka
byAdam Kotwasinski
 
PPTX
Exactly-once Stream Processing with Kafka Streams
byGuozhang Wang
 
PPTX
Apache Kafka 0.8 basic training - Verisign
byMichael Noll
 
PPTX
kafka
byAmikam Snir
 
PPSX
Docker Kubernetes Istio
byAraf Karsh Hamid
 
PDF
Best Practices for Middleware and Integration Architecture Modernization with...
byClaus Ibsen
 
Fundamentals of Apache Kafka
byChhavi Parasher
 
Apache Kafka - Martin Podval
byMartin Podval
 
Kafka 101
byClement Demonchy
 
Introduction to Apache Kafka
byJeff Holoman
 
Kafka Tutorial - basics of the Kafka streaming platform
byJean-Paul Azar
 
Apache kafka
byKumar Shivam
 
Microservices, DevOps & SRE
byAraf Karsh Hamid
 
Unified Big Data Processing with Apache Spark (QCON 2014)
byDatabricks
 
Introduction to Apache Kafka
byAIMDek Technologies
 
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
byKai Wähner
 
Deep Dive into Apache Kafka
byconfluent
 
효과적인 NoSQL (Elasticahe / DynamoDB) 디자인 및 활용 방안 (최유정 & 최홍식, AWS 솔루션즈 아키텍트) :: ...
byAmazon Web Services Korea
 
A visual introduction to Apache Kafka
byPaul Brebner
 
Envoy and Kafka
byAdam Kotwasinski
 
Exactly-once Stream Processing with Kafka Streams
byGuozhang Wang
 
Apache Kafka 0.8 basic training - Verisign
byMichael Noll
 
kafka
byAmikam Snir
 
Docker Kubernetes Istio
byAraf Karsh Hamid
 
Best Practices for Middleware and Integration Architecture Modernization with...
byClaus Ibsen
 

Similar to Kafka 101 and Developer Best Practices

PDF
What is Apache Kafka and What is an Event Streaming Platform?
byconfluent
 
PPTX
Westpac Bank Tech Talk 1: Dive into Apache Kafka
byconfluent
 
PDF
Devoxx university - Kafka de haut en bas
byFlorent Ramiere
 
PDF
Virtual Bash! A Lunchtime Introduction to Kafka
byJason Bell
 
PDF
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
PPTX
Kafka overview v0.1
byMahendran Ponnusamy
 
PDF
Kafka Deep Dive
byKnoldus Inc.
 
PDF
Apache KAfka
byPedro Alcantara
 
PDF
Developing Realtime Data Pipelines With Apache Kafka
byJoe Stein
 
PDF
Apache Kafka - From zero to hero
byApache Kafka TLV
 
PDF
Kafka zero to hero
byAvi Levi
 
PDF
Kafka syed academy_v1_introduction
bySyed Hadoop
 
PDF
DevOps Fest 2020. Сергій Калінець. Building Data Streaming Platform with Apac...
byDevOps_Fest
 
PDF
Apache Kafka Scalable Message Processing and more!
byGuido Schmutz
 
PPTX
Fundamentals and Architecture of Apache Kafka
byAngelo Cesaro
 
PDF
Kafka online course | Kafka training
byAccentfuture
 
PPTX
Streaming in Practice - Putting Apache Kafka in Production
byconfluent
 
PDF
Apache Kafka Women Who Code Meetup
bySnehal Nagmote
 
PDF
Introduction to Kafka and Event-Driven
byarconsis
 
PPTX
Introduction to Kafka and Event-Driven
byDimosthenis Botsaris
 
What is Apache Kafka and What is an Event Streaming Platform?
byconfluent
 
Westpac Bank Tech Talk 1: Dive into Apache Kafka
byconfluent
 
Devoxx university - Kafka de haut en bas
byFlorent Ramiere
 
Virtual Bash! A Lunchtime Introduction to Kafka
byJason Bell
 
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
Kafka overview v0.1
byMahendran Ponnusamy
 
Kafka Deep Dive
byKnoldus Inc.
 
Apache KAfka
byPedro Alcantara
 
Developing Realtime Data Pipelines With Apache Kafka
byJoe Stein
 
Apache Kafka - From zero to hero
byApache Kafka TLV
 
Kafka zero to hero
byAvi Levi
 
Kafka syed academy_v1_introduction
bySyed Hadoop
 
DevOps Fest 2020. Сергій Калінець. Building Data Streaming Platform with Apac...
byDevOps_Fest
 
Apache Kafka Scalable Message Processing and more!
byGuido Schmutz
 
Fundamentals and Architecture of Apache Kafka
byAngelo Cesaro
 
Kafka online course | Kafka training
byAccentfuture
 
Streaming in Practice - Putting Apache Kafka in Production
byconfluent
 
Apache Kafka Women Who Code Meetup
bySnehal Nagmote
 
Introduction to Kafka and Event-Driven
byarconsis
 
Introduction to Kafka and Event-Driven
byDimosthenis Botsaris
 

More from confluent

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
PDF
Migration, backup and restore made easy using Kannika
byconfluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
PDF
Unlocking value with event-driven architecture by Confluent
byconfluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
PDF
Building API data products on top of your real-time data infrastructure
byconfluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
Migration, backup and restore made easy using Kannika
byconfluent
 
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
Unlocking value with event-driven architecture by Confluent
byconfluent
 
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
Building API data products on top of your real-time data infrastructure
byconfluent
 
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 

Recently uploaded

PDF
Getting the Best of TrueDEM – October News & Updates
bypanagenda
 
PDF
Atlantix is an AI-first venture studio, building companies with AI at the core
byadmin625551
 
PDF
How to Measure Microsoft 365 Copilot Success and Manage AI Risk: Advanced Str...
byNikki Chapple
 
PDF
Data Structure - 12 Graph
byAndiNurkholis1
 
PPTX
Agentic AI Solutions and Services | Aeologic
byankitaeologic
 
PDF
A fool with a tool is still a fool - Plone Conference 2025
byAndreas Jung
 
PPTX
"Daily workflows with Cursor IDE", Maksym Anisimov
byFwdays
 
PDF
Unlocking Full-Stack Observability for AIOps: A Precisely Ironstream for Big ...
byPrecisely
 
PDF
NDP Act GAID Simplified: From Confusion to Compliance
byMuiz Adeleke
 
PDF
Dragino商品カタログ 2025.7 LoRaWAN・NB-IoT・LTE-M(LTE Cat.M1)対応センサーリスト
byCRI Japan, Inc.
 
PPTX
"Towards React Server Components in Clojure", Roman Liutikov
byFwdays
 
PPTX
Hybrid Active Directory Cyber Resiliency
byFrancesco Faenzi
 
PDF
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
PDF
2025 October Patch Tuesday
byIvanti
 
PDF
Cassandra vs. ScyllaDB: Evolutionary Differences
byScyllaDB
 
PDF
Meta and Apple close to settling EU cases.pdf
byLUMINATIVE MEDIA/PROJECT COUNSEL MEDIA GROUP
 
PDF
From Data Chaos to Copilot Confidence: A Step-by-Step Microsoft 365 Copilot R...
byNikki Chapple
 
PDF
NSSHOEU-J 4x35mm² Cable Specification.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
PPTX
CBD Belgium 2025 - The adventures of a Microsoft 365 Platform Owner.pptx
byJasper Oosterveld
 
PDF
Session 2 - Agentic Orchestration with UiPath Maestro
byDianaGray10
 
Getting the Best of TrueDEM – October News & Updates
bypanagenda
 
Atlantix is an AI-first venture studio, building companies with AI at the core
byadmin625551
 
How to Measure Microsoft 365 Copilot Success and Manage AI Risk: Advanced Str...
byNikki Chapple
 
Data Structure - 12 Graph
byAndiNurkholis1
 
Agentic AI Solutions and Services | Aeologic
byankitaeologic
 
A fool with a tool is still a fool - Plone Conference 2025
byAndreas Jung
 
"Daily workflows with Cursor IDE", Maksym Anisimov
byFwdays
 
Unlocking Full-Stack Observability for AIOps: A Precisely Ironstream for Big ...
byPrecisely
 
NDP Act GAID Simplified: From Confusion to Compliance
byMuiz Adeleke
 
Dragino商品カタログ 2025.7 LoRaWAN・NB-IoT・LTE-M(LTE Cat.M1)対応センサーリスト
byCRI Japan, Inc.
 
"Towards React Server Components in Clojure", Roman Liutikov
byFwdays
 
Hybrid Active Directory Cyber Resiliency
byFrancesco Faenzi
 
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
2025 October Patch Tuesday
byIvanti
 
Cassandra vs. ScyllaDB: Evolutionary Differences
byScyllaDB
 
Meta and Apple close to settling EU cases.pdf
byLUMINATIVE MEDIA/PROJECT COUNSEL MEDIA GROUP
 
From Data Chaos to Copilot Confidence: A Step-by-Step Microsoft 365 Copilot R...
byNikki Chapple
 
NSSHOEU-J 4x35mm² Cable Specification.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
CBD Belgium 2025 - The adventures of a Microsoft 365 Platform Owner.pptx
byJasper Oosterveld
 
Session 2 - Agentic Orchestration with UiPath Maestro
byDianaGray10
 

Kafka 101 and Developer Best Practices

  • 1.
    Kafka 101 & DeveloperBest Practices
  • 2.
    Agenda ● Kafka Overview ●Kafka 101 ● Best Practices for Writing to Kafka: A tour of the Producer ● Best Practices for Reading from Kafka: The Consumer ● General Considerations
  • 3.
    3 ETL/Data Integration Messaging Batch Expensive TimeConsuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
  • 4.
    4 ETL/Data Integration Messaging Batch Expensive TimeConsuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
  • 5.
    5 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficultto Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records
  • 6.
    6 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficultto Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records Both of these are a complete mismatch to how your business works.
  • 7.
    7 ETL/Data Integration Messaging TransientMessages Stored records ETL/Data Integration Messaging Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm High Throughput Durable Persistent Maintains Order Fast (Low Latency)
  • 8.
    8 Fast (Low Latency) EventStreaming Paradigm To rethink data as not stored records or transient messages, but instead as a continually updating stream of events
  • 9.
    9 Fast (Low Latency) EventStreaming Paradigm
  • 10.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. ... Device Logs ... ... ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Customer 360 Financial Fraud Detection Real-time Risk Analytics Real-time Payments Machine Learning Models ... Event-Streaming Applications Universal Event Pipeline Amazon S3 SaaS apps Confluent: Central Nervous System For Enterprise
  • 11.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Confluent uniquely enables Event Streaming success Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS Confluent founders are original creators of Kafka Confluent team wrote 80% of Kafka commits and has over 1M hours technical experience with Kafka Confluent helps enterprises successfully deploy event streaming at scale and accelerate time to market Confluent Platform extends Apache Kafka to be a secure, enterprise-ready platform
  • 12.
  • 13.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
  • 14.
    Scalability of a Filesystem Guaranteesof a Database Distributed By Design Rewind and Replay
  • 15.
    15 Kafka Broker Kafka Broker KafkaBroker Kafka Broker Kafka Producer Kafka Producer Kafka Producer Kafka Consumer Kafka Consumer (grouped) Kafka Consumer /Producer Kafka Broker Writers Readers Kafka Cluster
  • 16.
  • 17.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3
  • 18.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Creating a topic $ kafka-topics --zookeeper zk:2181 --create --topic my-topic --replication-factor 3 --partitions 3 Or use the AdminClient API!
  • 19.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time
  • 20.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time C C C
  • 21.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
  • 22.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
  • 23.
  • 24.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Clients - Producer Design Producer Record Topic [Partition] [Key] Value Serializer Partitioner Topic A Partition 0 Batch 0 Batch 1 Batch 2 Topic B Partition 1 Batch 0 Batch 1 Batch 2 Kafka Broker Send() Retry ? Fail ? Yes No Can’t retry, throw exception Success: return metadata Yes
  • 25.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer Kafka doesn’t care about what you send to it as long as it’s been converted to a byte stream beforehand. JSON CSV Avro Protobufs XML SERIALIZERS 01001010 01010011 01001111 01001110 01000011 01010011 01010110 01001010 01010011 01001111 01001110 01010000 01110010 01101111 01110100 ... 01011000 01001101 01001100 (if you must) Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
  • 26.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer private Properties kafkaProps = new Properties(); kafkaProps.put(“bootstrap.servers”, “broker1:9092,broker2:9092”); kafkaProps.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”); kafkaProps.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); kafkaProps.put("schema.registry.url", "https://schema-registry:8083"); producer = new KafkaProducer<String, SpecificRecord>(kafkaProps); Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
  • 27.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Record Topic [Partition] [Key] Value Record keys determine the partition with the default kafka partitioner If a key isn’t provided, messages will be produced in a round robin fashion partitioner Record Keys and why they’re important - Ordering
  • 28.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] AAA Value partitioner Record keys determine the partition with the default kafka partitioner
  • 29.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] BBB Value partitioner Record keys determine the partition with the default kafka partitioner
  • 30.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] CCC Value partitioner Record keys determine the partition with the default kafka partitioner
  • 31.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] DDD Value partitioner Record keys determine the partition with the default kafka partitioner
  • 32.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Record Keys and why they’re important - Key Cardinality Consumers Key cardinality affects the amount of work done by the individual consumers in a group. Poor key choice can lead to uneven workloads. Keys in Kafka don’t have to be primitives, like strings or ints. Like values, they can be be anything: JSON, Avro, etc… So create a key that will evenly distribute groups of records around the partitions. Car·di·nal·i·ty /ˌkärdəˈnalədē/ Noun the number of elements in a set or other grouping, as a property of that grouping.
  • 33.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101” } You don’t have to but... use a Schema! Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send JSON “Where’s record.City?” Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
  • 34.
    Schema Registry: MakeData Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic ! Schema Registry Open Source Feature
  • 35.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101”, “City”: “NA”, “State”: “NA” } Avro allows for evolution of schemas Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send AvroRecord Schema Registry Version 1 Version 2 Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
  • 36.
    Developing with ConfluentSchema Registry We provide several Maven plugins for developing with the Confluent Schema Registry ● download - download a subject’s schema to your project ● register - register a new schema to the schema registry from your development env ● test-compatibility - test changes made to a schema against compatibility rules set by the schema registry Reference https://docs.confluent.io/current/schema-registry/docs/maven-plugin.html <plugin> <groupId>io.confluent</groupId> <artifactId>kafka-schema-registry-maven-plug <version>5.0.0</version> <configuration> <schemaRegistryUrls> <param>http://192.168.99.100:808 </schemaRegistryUrls> <outputDirectory>src/main/avro</outp <subjectPatterns> <param>^TestSubject000-(key|valu </subjectPatterns> </configuration> </plugin>
  • 37.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Use Kafka’s Headers Reference https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers Producer Record Topic [Partition] [Timestamp] Value [Headers] [Key] Kafka Headers are simply an interface that requires a key of type String, and a value of type byte[], the headers are stored in an iterator in the ProducerRecord . Example Use Cases ● Data lineage: reference previous topic partition/offsets ● Producing host/application/owner ● Message routing ● Encryption metadata (which key pair was this message payload encrypted with?)
  • 38.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=0 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 39.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 ack Producer Properties acks=1 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 40.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
  • 41.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
  • 42.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234 data: abcd} - offset 3345 Failed ack Successful write Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 43.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234, data: abcd} - offset 3345 {key: 1234, data: abcd} - offset 3346 retry ack dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 44.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - with exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties enable.idempotence=true max.inflight.requests.per.connection=5 acks = “all” retries > 0 (preferably MAX_INT) (pid, seq) [payload] (100, 1) {key: 1234, data: abcd} - offset 3345 (100, 1) {key: 1234, data: abcd} - rejected, ack re-sent (100, 2) {key: 5678, data: efgh} - offset 3346 retry ack no dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 45.
    Transactional Producer Producer T1 T1T1 T1 T1 KafkaProducer producer = createKafkaProducer( “bootstrap.servers”, “broker:9092”, “transactional.id”, “my-transactional-id”); producer.initTransactions(); -- send some records -- producer.commitTransaction(); Consumer KafkaConsumer consumer = createKafkaConsumer( “bootstrap.servers”, “broker:9092”, “group.id”, “my-group-id”, "isolation.level", "read_committed"); Reference https://www.confluent.io/blog/transactions-apache-kafka/
  • 46.
  • 47.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. A basic Java Consumer final Consumer<String, String> consumer = new KafkaConsumer<String, String>(props); consumer.subscribe(Arrays.asList(topic)); try { while (true) { ConsumerRecords<String, String> records = consumer.poll(100); for (ConsumerRecord<String, String> record : records) { -- Do Some Work -- } } } finally { consumer.close(); } }
  • 48.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Single Consumer C
  • 49.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C1 C C C2
  • 50.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C C
  • 51.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
  • 52.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
  • 53.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0, 3 1 2 3
  • 54.
    Kafka’s Interceptors ProducerInterceptor onSend(ProducerRecord<K, V>record) Returns ProducerRecord<K, V> . Called from send() before key and value get serialized and partition is assigned. This method is allowed to modify the record. onAcknowledgement(RecordMetadata metadata, java.lang.Exception exception) This method is called when the record sent to the server has been acknowledged, or when sending the record fails before it gets sent to the server. Used for observability and reporting. ConsumerInterceptor onConsume(ConsumerRecords<K,V> records) Called just before the records are returned by KafkaConsumer.poll() This method is allowed to modify consumer records, in which case the new records will be returned. onCommit(Map<TopicPartition,OffsetAndMetada ta> offsets) This is called when offsets get committed. Used for observability and reporting Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/producer/ProducerInterceptor.html Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/consumer/ConsumerInterceptor.html
  • 55.
    Should I poolconnections? NO! Since Kafka connections are long-lived, there is no reason to pool connections. It’s common to keep one connection per thread.
  • 56.
    Use a goodclient! Clients ● Java/Scala - default clients, comes with Kafka ● C/C++ - https://github.com/edenhill/librdkafka ● C#/.Net - https://github.com/confluentinc/confluent-kafka-dotnet ● Python - https://github.com/confluentinc/confluent-kafka-python ● Golang - https://github.com/confluentinc/confluent-kafka-go ● Node/JavaScript - https://github.com/Blizzard/node-rdkafka (not supported by Confluent!) New Kafka features will only be available to modern, updated clients!
  • 57.
    Resources Free E-Books fromConfluent! I Heart Logs: https://www.confluent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/ Kafka: The Definitive Guide: https://www.confluent.io/resources/kafka-the-definitive-guide/ Designing Event Driven Systems: https://www.confluent.io/designing-event-driven-systems/ Confluent Blog: https://www.confluent.io/blog Thank You!
  • 58.