KEMBAR78
Apache Kafka 0.11 の Exactly Once Semantics | PDF
Apache Kafka 0.11
Exactly Once Semantics
/ @laclefyoshi / ysaeki@r.recruit.co.jp
•
• Kafka
• Exactly Once :
•
2
• 2011/04
• 2015/09
•
• Druid (KDP, 2015)
• RDB NoSQL ( , 2016; : HBase )
• ESP8266 Wi-Fi IoT (KDP, 2016)
•
• (WebDB Forum 2014)
• Spark Streaming (Spark Meetup December; 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
• Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016)
• Apache Spark ( Geek Night #11; 2016)
• (BigData-JAWS #6; 2017)
3
Kafka
Kafka
5
Kafka
ACK
→
ACK
→
6
Kafka
Broker
→
7
Kafka
→
→
8
Kafka
At Least Once At Most Once
9
Kafka
At Least Once At Most Once Exactly Once
9
Exactly Once :
Kafka 0.11.0.0
• 2017/06/28
• KIP-129 → KAFKA-4923: Add Exactly-Once Semantics to Streams
• https://issues.apache.org/jira/browse/KAFKA-4923
• Confluent Blog: Exactly-once Semantics are Possible: Here’s How
Kafka Does it
• https://www.confluent.io/blog/exactly-once-semantics-are-possible-
heres-how-apache-kafka-does-it/
• “In the following posts in this series, we will go into more details …”
• Exactly Once Delivery and Transactional Messaging in Kafka
• https://docs.google.com/document/d/
11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/
• KIP-98 → KAFKA-5059: Implement Transactional Coordinator
11
Kafka
13
Kafka : Exactly Once
14
Producer Exactly Once
15
Producer Exactly Once
1.
2.
3. x N
4. →
16
Producer Exactly Once
1.
2.
3. x N
4.
→
→
→ 1
17
Producer Exactly Once
5.Broker Commit Marker 

COMMITTED ABORTED
↓ ↓
18
Consumer Exactly Once
19
Consumer Exactly Once
6. 

↓
READ_COMMITTED
↓
READ_UNCOMMITTED
20
Consumer Exactly Once
7.
8. 

- COMMITTED 

- ABORTED Transaction List 

21
• Kafka API RecordBatch
• Kafka < 0.11 Message MessageSet
API
• Kafka 0.11 MessageSet/Message RecordBatch/Record
•
• Producer Broker Consumer
ID PID Producer ID
22
Exactly Once
• : Exactly Once
• Producer
• Broker
• Commit Marker
• Consumer
•
→ OSS
• Exactly Once
• Producer Broker
• Consumer
• 3%
•
• Confluent
• OSS
Kafka
• Exactly Once
24

Apache Kafka 0.11 の Exactly Once Semantics