![29Confidential
Stream Processing App #1: Anomaly Detection / Alerting
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 10 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;
● Use Kafka Streams
and/or additional
input streams in
more sophisticated
algorithm
● possible_fraud
is a change log
stream where key is
[card_number,
window_start]
authorization_attempts
possible_fraud
SMS Gateway](https://image.slidesharecdn.com/tourofkafka-bangkokandsingapore-181105170304/85/A-Tour-of-Apache-Kafka-29-320.jpg)
![29Confidential
Stream Processing App #1: Anomaly Detection / Alerting
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 10 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;
● Use Kafka Streams
and/or additional
input streams in
more sophisticated
algorithm
● possible_fraud
is a change log
stream where key is
[card_number,
window_start]
authorization_attempts
possible_fraud
SMS Gateway](https://image.slidesharecdn.com/tourofkafka-bangkokandsingapore-181105170304/75/A-Tour-of-Apache-Kafka-29-2048.jpg)
Uploaded byconfluent
A Tour of Apache Kafka
The document provides an overview of Apache Kafka, a streaming platform widely used by Fortune 500 companies for real-time data processing. It highlights Kafka's strengths in scalability, durability, and integration capabilities, as well as its applications in microservices and data integration. It also discusses specific use cases, Kafka clients, and how it can serve as a commit log for organizations.
More Related Content
![UiPath Automation Suite Installation (Hands-On) [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/automationsuitecommunitysession2-251015095633-a6d862f1-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
A Tour of Apache Kafka
- 1. 1Confidential A Tour ofApache Kafka Matt Howlett Engineer, Confluent Inc.
- 2. 2Confidential Agenda 1. Technical Overviewof Apache Kafka 2. Use Cases
- 3. 3Confidential What is ApacheKafka? Kafka is a streaming platform. A distinct tool in your toolbox, like a relational database or a traditional messaging system. A streaming platform encourages architectures that have an emphasis on events and changes to data (not data at rest). Widely applicable. E.g. consider Walmart.
- 4. 4Confidential Who Uses KafkaToday? ● 35% of Fortune 500 + thousands of companies world wide use Kafka ● Across all industries ● High growth of usage within companies
- 5. 5Confidential Core Kafka Pt.1 Traditional messaging: move data Kafka: make data available
- 6. 6Confidential ● Simple: ○ Highperformance ○ Robust horizontal scalability ● Suitable for real-time, streaming and batch operations ● Ad-hoc consumption & reprocessing ● Immutable: ○ Easier to debug/reason about v.s. ephemeral data ○ Auditable by default Core Kafka Pt. 2: Why Logs?
- 7. 7Confidential Core Kafka Pt.3 - Scaling Notes: ordering per partition only re-partitioning Key Value Message Kafka topics are partitioned logs
- 8. 8Confidential Core Kafka Pt.4 - Durability Kafka topics are replicated partitioned logs Notes: all reads and writes are to leader replica
- 9. 9Confidential Core Kafka Pt.5: How Scalable is Kafka? ● No bottleneck! ○ Many brokers ○ Many producers ○ Many consumers ● Limits? ○ Internet giants are driving the limits higher - you won’t need to worry. ○ e.g. LinkedIn > 1 trillion messages / day through Kafka clusters. ○ 100 brokers / 2 billion messages a day is “straightforward” to operate ○ Don’t over partition ~< 100k partitions producers brokers consumers
- 10.
- 11. 11Confidential Kafka Clients Use cases: ●Integration with custom applications ○ log application events ○ Invoke REST API ● Stateless stream processing (filter, transform) Confluent supported clients: ● Java ● C (librdkafka) ○ C++ ○ Python ○ C# / .NET ○ Go ● REST Proxy
- 12. 12Confidential ● Off-the-shelf connectors ○Confluent Hub ● Standardized framework ○ Scalable ○ Fault Tolerant Kafka Connect ● Stateless workers ○ Un-opinionated deployment ● REST API ● Transforms ● Exactly Once
- 13. 13Confidential Kafka Streams ● Justa library! A library that makes it easy to do stateful operations (joins, aggregations, windowing). ● Elastically scalable ○ distributed! ● Fault tolerant ● Un-opinionated deployment ● State backed by Kafka used as a changelog ● Exactly once processing ● Record-at-a-time processing ● Complex topologies ○ (but keep it simple) ● JVM only (Java, Scala, etc.)
- 14. 14Confidential Confluent: A MoreComplete Streaming Platform +
- 15.
- 16. 16Confidential When Should YouUse Kafka? Scalability ● Quantity of Data ○ Simple Applications (or not) ● Complexity ○ Architectural ○ Organizational
- 17. 17Confidential Buffering Pt. 1 Kafkais a very good buffer: ● Write optimized ● Highly reliable ● Tolerate data spikes ● Tolerate downstream outages ● Used by KStreams (no back pressure problems) Other examples:
- 18. 18Confidential Buffering Pt. 2 Movedata to multiple locations
- 19. 19Confidential Data Integration Explosion ofData Sources and Processing Frameworks
- 20.
- 21. 21Confidential … Hub-and-Spoke ● #connections: O(N) vs potentially O(N^2) ● standardized, reliable data transport ● standardized data format
- 22. 22Confidential Dual Writes Problem Withouta log, there’s potential consistency problem:
- 23. 23Confidential Data Integration -Eventual Consistency
- 24.
- 25. 25Confidential You can thinkof Kafka as a commit log for your entire organization “turning the database inside out”
- 26. 26Confidential Advanced ETL #1:PII Data Filter ● Kafka security: TLS encryption, flexible authentication + authorization (secured) kafka streams
- 27. 27Confidential Advanced ETL #2:Enriching stream data
- 28. 28Confidential Advanced ETL #2:Stream / Table Join in KSQL CREATE STREAM enriched_weblog AS SELECT ip, text, g.location AS location FROM weblog w LEFT JOIN geo g ON w.ip = g.ip; ● Query is long running on a KSQL cluster. ● Create weblog stream and geo table (backed by kafka topics) first. ● Currently, KSQL can interpret AVRO, JSON and CSV.
- 29. 29Confidential Stream Processing App#1: Anomaly Detection / Alerting CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 10 SECONDS) GROUP BY card_number HAVING count(*) > 3; ● Use Kafka Streams and/or additional input streams in more sophisticated algorithm ● possible_fraud is a change log stream where key is [card_number, window_start] authorization_attempts possible_fraud SMS Gateway
- 30. 30Confidential Microservices What are Microservices? ●Independently deployable, small units of functionality ○ (not a formal definition) ○ Primary motivation: decouple teams (scale in people terms) ○ Usually REST endpoints + commands/queries Microservices can also be built on a backbone of events: ○ PII Filter ○ Weblog enricher ○ SMS fraud alert notifier ○ ... just the start
- 31.
- 32. 32Confidential Microservices - CommandsPt. 2 Adding new functionality that depends on placing orders requires changes to the Orders Service.
- 33. 33Confidential Microservices - ReceiverDriven Flow Control ● Pricing Service team does not need to talk to Orders Service team ● Trade off: no statement of overall behavior
- 34. 34Confidential Microservices - Queries Note:Eventual Consistency
- 35. 35Confidential Microservices ● Decreased coupling- Orders Service materializes the view it requires (Kafka can even act as system of record) ● Often appropriate at larger scales
- 36. 36Confidential Microservices - DataDichotomy The Data Dichotomy “Data systems are about exposing data. Services are about hiding it”
- 37.