Kafka based Global Data Mesh at Wix

@NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Kafka-based Global Data
Mesh at Wix
Natan Silnitsky Backend Infra TL, Wix.com
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* keep simple even. Migrated, disributed

@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* messaging platform

@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* tackled blockers

@NSilnitsky
@NSilnitsky
Event-driven
architecture
Kafka
Growing
Pains
Large scale
traffic
Hundreds of
developers
Diverse
tech stack
Cluster
overload
DC
awareness

@NSilnitsky
Kafka Broker
You Know
Kafka
Kafka
Consumer
Kafka Broker
Service A Service B
Kafka
Producer

@NSilnitsky
If You Don’t
Know Kafka
Kafka Broker
Kafka Broker
Topic
Partition

@NSilnitsky
Kafka
Broker Topic
Partition
Kafka
Producer
0 1 2 3 4 5
Service A

@NSilnitsky
Kafka
Broker Topic
Partition
Kafka
Producer
0 1 2 3 4 5
Service A
Service B
Kafka
Consumer

@NSilnitsky
@NSilnitsky
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka
Growing
Pains
Cluster
overload
DC
awareness

@NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Greyhound*
Wraps
Kafka
* open source software
Kafka Broker
Kafka
Producer
Service A Service B
Kafka
Consumer

@NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Kafka Broker
Kafka
Producer
Service A Service B
Kafka
Consumer

@NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
val producer = GreyhoundBuilder.resilientProducer(topicName)

@NSilnitsky
✔ Parallel Consumption
✔ Batch Consumer
✔ Consumer Retry Strategies
✔ Resilient Producer
✔ Context Propagation
✔ Metrics reporting
Self Service 🟊
Greyhound
Kafka Broker

@NSilnitsky
Scala NodeJS / Python / Go
Greyhound
Kafka Broker
I’m a container
gRPC + proto
sidecar
JVM library

@NSilnitsky
Kafka Broker
Domain Events
ETL to DWH
Task Scheduling
Async Queue
...
We use it
for...
Infra

@NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness

@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Kafka Broker
Single
Cluster
Overload
* intra-dc

@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Single
Cluster
Overload
A
A
A
A
B
Kafka Cluster (by SLAs)
B
Multi
Cluster
C

@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
A
A
A
A
B
To which
Cluster?
Kafka Cluster (by SLAs)
B
I want to produce
a domain event.
C

@NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
.ClusterA
val producer = GreyhoundBuilder.resilientProducer(topicName,
_.ClusterB
* wix-layer

@NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Master Active
Master Passive
Read-Only DBs
Http/gRPC
To which
DC?
I want to
process the event
and save the data.

@NSilnitsky
@NSilnitsky
Kafka + Greyhound
+ Cross DC services
Growing
Pains
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster

@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka

@NSilnitsky
Replicate !
DC1 DC2
across DCs
Greyhound
producer
Greyhound
consumer
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
A

@NSilnitsky
Greyhound
Batch
Consumer
Greyhound
Producer
Batch #
Replicate !
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
DC1 DC2
How?
Greyhound
producer
Greyhound
consumer
A
Cross DC
Replication
Service

@NSilnitsky
Replicate !
DC1 DC2
a Compact Topic for Policy
A
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer

@NSilnitsky
Replication
service
Replicate !
DC1 DC2
How?
A
KVStoreReader
In-Memory Map
Key Value
Topic1 AllDCs
Topic2 Custom
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer

@NSilnitsky

@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. Produce to Main DC
Greyhound
Kafka

@NSilnitsky
Produce !
DC1 DC2
To main DC
Read-Only DB /
Master Passive DB
Master Active
A
Kafka
Broker
Cluster
Greyhound
producer

@NSilnitsky
main-dc-producer
service
Produce !
DC1 DC2
How?
Master Active
Greyhound
gRPC
producer
producer(topicName, _.toMainDc)

@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. ✔ Produce to Main DC
Greyhound
Kafka

@NSilnitsky
DC1 DC2
To where needed
Greyhound
producer
Greyhound
consumer
Master Active

@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
To where needed
Master Active

@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Master Active
Greyhound
producer
Greyhound
consumer
To where needed
Replication service
* reintroduce decoupling

@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
A
A
A
A
B
B
Just Produce ❤
To a Single Logical
Kafka Cluster.
Kafka Broker
Cluster
DB

@NSilnitsky
@NSilnitsky
Any
Tech Stack
Greyhound
Anywhere
You Need
developer

@NSilnitsky
developer
github.com/wix/greyhound

@NSilnitsky
@NSilnitsky
Thank
You!
👉 slideshare.net/NatanSilnitsky
Any questions?

@NSilnitsky
@NSilnitsky
Q&A
👉 slideshare.net/NatanSilnitsky
Any questions?

Kafka based Global Data Mesh at Wix

More Related Content

What's hot

Similar to Kafka based Global Data Mesh at Wix

More from Natan Silnitsky

Recently uploaded

Kafka based Global Data Mesh at Wix