KafkaとAWS Kinesisの比較

KafkaとAWS Kinesisの比較

• 1.
  Kafka AWS Kinesis /@laclefyoshi / ysaeki@r.recruit.co.jp
• 2.
  • • Kafka Kinesis • • • • 2
• 3.
  • 2011/04 • 2015/09 • •Druid (2015; Kindle Direct Publishing) • RDB NoSQL (2016; ; HBase ) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) 3
• 4.
  Kafka Kinesis • Apache •Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
• 5.
  Kafka Kinesis • Apache •Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
• 6.
  : 5 : <100ms : ~8000msgs/sec Kinesis
• 7.
  [Kafka] [Kinesis] 6
• 8.
  [Kafka] [Kinesis] 6
• 9.
  [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS APIGateway HTTP API ETL ETL 7
• 10.
  [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS APIGateway HTTP API ETL ETL OSS •Kafka Streams •PipelineDB AWS •Kinesis Analytics 7
• 11.
  [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ←→ ← → ← → 8
• 12.
  [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ←→ ← → ← → 8
• 13.
  [ ] http://insightdataengineering.com/blog/ingestion-comparison/ X
• 14.
  Kafka Kinesis • Kafkaor Kinesis • Kafka OSS Kinesis • 9
• 15.
  Kafka Kinesis • (<5000msgs/sec) Kinesis • Kafka • Kinesis • Hadoop Kafka • Kinesis • Kafka 10
• 16.
  • OSS →Kafka • → Kinesis • API • • SNS Bot IoT • 11