KEMBAR78
Kafka slideshare | PDF
Kafka
ํ•œ๊ตญํด๋ฆฌํ…๋Œ€ํ•™
์Šค๋งˆํŠธ๊ธˆ์œต๊ณผ
Kafka?
๏ต
๏ต
image source : https://mesosphere.com/blog/kafka-dcos-tutorial/
Kafka Major Concepts
Topic
Producer
Producer
Consumer
Consumer
<Broker>
Kafka Major Concepts
๋Œ€์šฉ๋Ÿ‰ ์‹ค์‹œ๊ฐ„ ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ + ๋ถ„์‚ฐ๋œ ๋ฐ์ดํ„ฐ ํŒŒ์ดํ”„๋ผ์ธ ํ‘œ์ค€ํ™”
๋†’์€ ์ฒ˜๋ฆฌ๋Ÿ‰, ๋น ๋ฅธ ๋ฉ”์‹œ์ง€ ์ „์†ก, ์šด์˜ ํšจ์œจํ™”๋ฅผ ์œ„ํ•ด
๋ถ„์‚ฐ์‹œ์Šคํ…œ, ํŽ˜์ด์ง€์บ์‰ฌ, ๋ฐฐ์น˜์ „์†ก์ฒ˜๋ฆฌ
Kafka Major Concepts
image source : https://www.tutorialspoint.com/apache_kafka/apache_kafka_cluster_architecture.htm
โ€ข broker : ์นดํ”„์นด ์„œ๋ฒ„๋ฅผ ๊ฐ€๋ฆฌํ‚ด
โ€ข zookeeper : ์นดํ”„์นด ์„œ๋ฒ„ (+ํด๋Ÿฌ์Šคํ„ฐ) ์ƒํƒœ๋ฅผ ๊ด€๋ฆฌํ•˜๊ณ 
โ€ข cluster : ๋ธŒ๋กœ์ปค๋“ค์˜ ๋ฌถ์Œ
Kafka Major Concepts
image source : https://kafka.apache.org/documentation/
โ€ข topic : ๋ฉ”์‹œ์ง€ ์ข…๋ฅ˜
โ€ข partitions : topic ์ด ๋‚˜๋ˆ ์ง€๋Š” ๋‹จ์œ„
โ€ข Log : 1๊ฐœ์˜ ๋ฉ”์„ธ์ง€
โ€ข offset : ํŒŒํ‹ฐ์…˜ ๋‚ด์—์„œ ๊ฐ ๋ฉ”์‹œ์ง€๊ฐ€ ๊ฐ€์ง€๋Š” unique id
Kafka working mechanism
image source : https://fr.wikipedia.org/wiki/Fichier:Overview_of_Apache_Kafka.svg
โ€ข ์ •ํ•ด์ง„ topic์— producer๊ฐ€ ๋ฉ”์‹œ์ง€๋ฅผ ๋ฐœํ–‰ํ•ด ๋†“์œผ๋ฉด
consumer๊ฐ€ ํ•„์š”ํ•  ๋•Œ ํ•ด๋‹น ๋ฉ”์‹œ์ง€๋ฅผ ๊ฐ€์ ธ๊ฐ„๋‹ค.
Kafka working mechanism
image source : https://www.tutorialspoint.com/apache_kafka/apache_kafka_cluster_architecture.htm
zookeeper ๊ฐ€ kafka ์˜ ๋ถ„์‚ฐ ๋ฉ”์‹œ์ง€ ํ์˜ ์ •๋ณด๋ฅผ ๊ด€๋ฆฌํ•ด์ค€๋‹ค.
Kafka working mechanism
์—ฌ๋Ÿฌ ๊ฐœ์˜ ์นดํ”„์นด ์„œ๋ฒ„๋กœ ๊ตฌ์„ฑํ•ด ๋™์ผํ•œ ์—ญํ• ์„ ํ•˜๋Š” ์„œ๋ฒ„๋ฅผ ์ถ”๊ฐ€ํ•ด ๋ถ€ํ•˜๋ฅผ ๋ถ„์‚ฐํ•  ์ˆ˜ ์žˆ๋Š” ๊ฒƒ
ํ•˜์ง€๋งŒ ์„œ๋ฒ„๋งŒ ์ถ”๊ฐ€ํ•˜๋‹ค ๋ณด๋ฉด ๋ถˆํ•„์š”ํ•œ ๋น„์šฉ์ด ์ฆ๊ฐ€
LinkedIn์—์„œ ๊ฐ€์žฅ ์‚ฌ์šฉ๋Ÿ‰ ๋†’์€ ํด๋Ÿฌ์Šคํ„ฐ๋Š” 60๋Œ€์˜ ๋ธŒ๋กœ์ปค๋ฅผ ์‚ฌ์šฉํ•จ
image source : http://blog.mmlac.com/log-transport-with-apache-kafka/
ํ•œ ๋ฒˆ ์ฝ์€ ํŒŒ์ผ์˜ ๋‚ด์šฉ์„ ์ด ํŽ˜์ด์ง€ ์บ์‹œ ์˜์—ญ์— ์ €์žฅํ•˜๊ณ , ๊ฐ™์€ ํŒŒ์ผ์˜ ์ ‘๊ทผ์ด ์ผ์–ด๋‚˜๋ฉด
๋””์Šคํฌ์—์„œ ์ฝ์–ด์˜ค๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ ํŽ˜์ด์ง€ ์บ์‹œ์—์„œ ์ฝ๋Š”๋‹ค.
์นดํ”„์นด๋Š” ํŽ˜์ด์ง€ ์บ์‰ฌ๋ฅผ ์ด์šฉํ•ด ๋น ๋ฅธ ์•ก์„ธ์Šค๊ฐ€ ๊ฐ€๋Šฅํ•จ.
์ €๋ ดํ•œ SATA๋””์Šคํฌ๋ฅผ ์‚ฌ์šฉํ•ด๋„ ๋ฌด๋ฐฉ
Kafka working mechanism
source : https://www.slideshare.net/DmitryTolpeko/apache-kafka-messagingsystem/14
Kafka working mechanism
source : https://www.slideshare.net/DmitryTolpeko/apache-kafka-messagingsystem/14
source : https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600
๋ฐ์ดํ„ฐ๋ฅผ ์ฃผ๊ณ  ๋ฐ›๋Š” ๊ณผ์ •์—์„œ ์ž‘์€ I/O ๋นˆ๋ฒˆํ•˜๊ฒŒ ์ผ์–ด๋‚˜๋ฉด ์†๋„ ์ €ํ•˜ ๋จ
์นดํ”„์นด๋Š” ์ž‘์€ I/O๋ฅผ ๋ฌถ์–ด์„œ ์ฒ˜๋ฆฌํ•˜์—ฌ ๋„คํŠธ์›Œํฌ ์˜ค๋ฒ„ํ—ค๋“œ๋ฅผ ์ค„์ด๊ณ  ๋ฐฐ์น˜ ์ „์†ก์ฒ˜๋ฆฌ ์†๋„ ํ–ฅ์ƒ
Kafka working mechanism
Kafka Use cases
๋ฐ์ดํ„ฐ ๋ถ„์„ ๊ฒฐ๊ณผ๋กœ ์„œ๋น„์Šค ํŽ˜์ด์ง€์— ์žˆ๋Š” ์ปจํ…์ธ ๊ฐ€ ์‚ฌ์šฉ์ž์—๊ฒŒ
์–ผ๋งˆ๋‚˜ ์˜ํ–ฅ์„ ์ฃผ๋Š”์ง€ ์‹ค์‹œ๊ฐ„์œผ๋กœ ๊ฐ์ง€
์•ž์˜ ์ˆซ์ž๋Š” ๊ธฐ์‚ฌ์˜ ์ˆœ์œ„
๋’ค์˜ ์ˆซ์ž๋Š” ๊ธฐ์‚ฌ์˜ ์†Œ๋น„์ง€์ˆ˜
source : โ€œkafka exerciseโ€ presented by Dongkeun Lee, Kangyong kim, student at dept. of smart finance, Korea Polytechnics in 2018
Kafka Use cases
์นด์นด์˜คํ†ก ์•ฑ์—์„œ 10๋ถ„ ๊ฐ„ ์†Œ๋น„ํ•œ ๋„ค์ด๋ฒ„ ์—ฐ์•  ๊ธฐ์‚ฌ ์†Œ๋น„ ์ง€์ˆ˜ ๋ถ„์„
source : โ€œkafka exerciseโ€ presented by Dongkeun Lee, Kangyong kim, student at dept. of smart finance, Korea Polytechnics in 2018
Kafka Installation
VM ์„ค์น˜
Java 8.0, ZooKeeper, Kafka ์„ค์น˜
master slave1 slave2
ZooKeeper ensemble
master slave1 slave2
Kafka Cluster
quorum
์‹ค์Šตํ™˜๊ฒฝ ๊ฐœ์š”
๊ฐ€์ƒ ๋จธ์‹ (VM)
โ€ข VirtualBox ์ด์šฉ
โ€ข ์ค€๋น„ ๋ฐ ์ ๊ฒ€ ์‚ฌํ•ญ
- VirtualBox Latest Ver.
- CentOS 7 (DVD Ver.)
- JDK 1.8
- ๋ชจ๋“  ์„ค์น˜ ๋ฐ
ํ™˜๊ฒฝ์„ค์ •์€ root๊ถŒํ•œ์œผ๋กœ
์„ค์น˜ํ•จ
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
๊ฐ€์ƒ ๋จธ์‹ (VM) ์„ค์น˜ ๋ฐ ์„ค์ •
1. ๊ฐ€์ƒ๋จธ์‹  ์ด๋ฆ„ ์„ค์ • 2~6. ๊ฐ€์ƒ๋จธ์‹  ๋ฉ”๋ชจ๋ฆฌ / ํ•˜๋“œ ์„ค์ •
2
4
3
5 6
1
5 6
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
๊ฐ€์ƒ ๋จธ์‹ (VM) ์„ค์น˜ ๋ฐ ์„ค์ •
1. ์ƒ์„ฑ๋œ master ๋จธ์‹ ์˜ ๋„คํŠธ์›Œํฌ ์„ค์ •์œผ๋กœ ๋“ค์–ด๊ฐ„๋‹ค
2. ์–ดํƒญํ„ฐ 2์—๋Š” ์–ด๋Œ‘ํ„ฐ์˜ ๋ธŒ๋ฆฌ์ง€๋ฅผ ์„ ํƒ
3. ์–ดํƒญํ„ฐ 3์—๋Š” ํ˜ธ์ŠคํŠธ ์ „์šฉ ์–ด๋Œ‘ํ„ฐ ์„ ํƒ
1 2
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
Cent OS 7 ์„ค์น˜
1. GNOME ํ…Œ์Šคํฌํƒ‘ ์„ ํƒ
2. ๋กœ์ปฌ ํ‘œ์ค€ ๋””์Šคํฌ ์„ ํƒ
1 2
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
Cent OS 7 ์„ค์น˜
1. Root ๋น„๋ฐ€๋ฒˆํ˜ธ ์„ค์ •
2. ์‚ฌ์šฉ์ž ID, PW์„ค์ •
1
2
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
๋„คํŠธ์›Œํฌ ์„ค์ •
๏ต Network ๊ณ ์ • IP ์„ค์ •(Static)
* vi /etc/sysconfig/network-scripts/ifcfg-enp0s8
BOOTPROTO=static
์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€
ONBOOT=yes
NETMASK=255.255.255.0
IPADDR=192.168.0.30
GATEWAY=10.0.0.2
๋„คํŠธ์›Œํฌ ๋Œ€์—ญ๋Œ€๋Š” ๋ณ€๊ฒฝ ๊ฐ€๋Šฅํ•จ
์„ค์ • ์™„๋ฃŒ ํ›„
systemctl restart network
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
๋„คํŠธ์›Œํฌ ์„ค์ •
๏ต Host ์„ค์ •
vi /etc/hosts
master
192.168.0.30 master.kopo
slave1
192.168.0.31 slave1.kopo
slave2
192.168.0.32 slave2.kopo
์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
vi /etc/hosts
๋„คํŠธ์›Œํฌ ์„ค์ •
๏ต Host name ์„ค์ •
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=master.kopo
์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€
--๋‚˜๋จธ์ง€ VM๋„ ๋ณ€๊ฒฝ--
์˜ˆ์‹œ)
NETWORKING=yes
HOSTNAME=slave1.kopo
[root@localhost ~]# hostnamectl set-hostname master.kopo
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
๋„คํŠธ์›Œํฌ ์„ค์ •
๏ต ๋ฐฉํ™”๋ฒฝ ํ•ด์ œ
- ๋ฐฉํ™”๋ฒฝ ๋„๊ธฐ
systemctl stop firewalld
- ํ”„๋กœ๊ทธ๋žจ ์ž๋™ ์‹œ์ž‘ ํ•ด์ œ
systemctl disable firewalld
-๋ฐฉํ™”๋ฒฝ ํ•ด์ฒด ํ›„ ping test
ping โ€“c 3 8.8.8.8
- CentOS์˜ ๊ฒฝ์šฐ OS๋ฅผ ์ฒ˜์Œ ์„ค์น˜ํ–ˆ์„ ๋•Œ ๊ธฐ๋ณธ ํฌํŠธ๋ฅผ ์ œ์™ธํ•œ ๋ชจ๋“  ํฌํŠธ๋ฅผ ๋ฐฉํ™”๋ฒฝ์—์„œ ๋ง‰๊ณ  ์žˆ์–ด ์‚ฌ์šฉํ•  ํฌํŠธ๋ฅผ ๊ฐœ๋ฐฉํ•ด ์ฃผ์–ด์•ผํ•จ.
- ํฌํŠธ ์ถ”๊ฐ€
firewall-cmd --permanent --zone=public --add-port=ํฌํŠธ๋ฒˆํ˜ธ/tcp
- ๋ฐฉํ™”๋ฒฝ ์žฌ์‹œ์ž‘
firewall-cmd --reload
- ์™„์ „ํžˆ ๋ฐฉํ™”๋ฒฝ์„ ํ•ด์ฒดํ•˜๊ณ  ์‹ถ์€ ๊ฒฝ์šฐ
- ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ๋ชจ๋“  ์„œ๋น„์Šค/ํฌํŠธ ๋ชฉ๋ก์„ ์กฐํšŒ
firewall-cmd --list-all
- ํ—ˆ์šฉํ•œ ํฌํŠธ ๋ชฉ๋ก
firewall-cmd --list-ports
- ๋ฐฉํ™”๋ฒฝ ์ƒํƒœ ํ™•์ธ
firewall-cmd โ€“state
# ์ฃผํ‚คํผ ํด๋ผ์ด์–ธํŠธ ์ ‘์†์šฉ ํฌํŠธ
firewall-cmd --permanent --zone=public --add-port=2181/tcp
firewall-cmd --reload
firewall-cmd --list-ports
nc -z master 2181
#์ฃผํ‚คํผ ์•™์ƒ๋ธ” ํฌํŠธ
firewall-cmd --permanent --zone=public --add-port=2888/tcp
firewall-cmd --permanent --zone=public --add-port=3888/tcp
firewall-cmd --reload
firewall-cmd --list-ports
#์นดํ”„์นด ์ ‘์† ํฌํŠธ
firewall-cmd --permanent --zone=public --add-port=9092/tcp
firewall-cmd --reload
firewall-cmd --list-ports
yum install nc-1.84-24.el6.x86_64.rpm
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
Java ์„ค์น˜
๏ต Java ์„ค์น˜
1.๋‹ค์šด๋กœ๋“œ
Java jdk 1.8์„ ๋ฆฌ๋ˆ…์Šค ์„œ๋ฒ„์— ๋‹ค์šด๋กœ๋“œ ํ›„
ํŠน์ • ํด๋”์— ์ €์žฅ
2. ์••์ถ•ํ•ด์ฒด
tar xzf jdk-8u181-linux-x64.tar.gz ๋ช…๋ น์–ด๋กœ
์••์ถ• ํ•ด์ฒด
3. ํด๋” ์ด๋™
mv jdk1.8.0_181 /usr/local
๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด ํด๋” ์ด๋™
4. ๋งํฌ ์„ค์ •
-local ํด๋”๋กœ ์ด๋™
cd /usr/local
-๋งํฌ ์„ค์ •
ln โ€“s jdk1.8.0_181 java
๋˜๋Š” yum โ€“y install java-1.8.0-openjdk
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
์ฃผํ‚คํผ ์„ค์ •
cd /usr/local
wget http://apache.mirror.cdnetworks.com/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz
tar zxf zookeeper-3.4.13.tar.gz
ln -s zookeeper-3.4.13 zookeeper
mkdir -p /data
cd /data
echo 1 > /data/myid
zookeeper node
[root@master conf]# ll
total 16
-rw-r--r--. 1 501 games 535 Jun 29 2018 configuration.xsl
-rw-r--r--. 1 501 games 2161 Jun 29 2018 log4j.properties
-rw-r--r--. 1 root root 167 Nov 8 23:38 zoo.cfg
-rw-r--r--. 1 501 games 922 Jun 29 2018 zoo_sample.cfg
[root@master conf]# pwd
/usr/local/zookeeper/conf
[root@master conf]# more zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data
clientPort=2181
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888
maxClientCnxns=200
์ฃผํ‚คํผ ์„ค์ •
vi zoo.cfg๋กœ ์‹คํ–‰ํ•˜์—ฌ
์ขŒ์ธก๊ณผ ๊ฐ™์ด ์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€ํ•œ๋‹ค.
tickTime : ์ฃผํ‚คํผ๊ฐ€ ์‚ฌ์šฉํ•˜๋Š” ์‹œ๊ฐ„์— ๋Œ€ํ•œ ๊ธฐ๋ณธ ์ธก์ • ๋‹จ์œ„(๋ฐ€๋ฆฌ์ดˆ)
initLimit : Follower๊ฐ€ Leader์™€ ์ดˆ๊ธฐ์— ์—ฐ๊ฒฐํ•˜๋Š” ์‹œ๊ฐ„์— ๋Œ€ํ•œ TimeOut tick ์ˆ˜
syncLimit : Follower๊ฐ€ Leader์™€ ๋™๊ธฐํ™” ํ•˜๋Š” ์‹œ๊ฐ„์— ๋Œ€ํ•œ TimeOut tick ์ˆ˜
dataDir : ์ฃผํ‚คํผ์˜ ํŠธ๋žœ์žญ์…˜ ๋กœ๊ทธ์™€ ์Šค๋ƒ…์ƒท์ด ์ €์žฅ๋˜๋Š” ๊ฒฝ๋กœ
clientPort : ์ฃผํ‚คํผ ์‚ฌ์šฉ TCP ํฌํŠธ
server.* : ์ฃผํ‚คํผ ์•™์ƒ๋ธ” ๊ตฌ์„ฑ์„ ์œ„ํ•œ ์„œ๋ฒ„ ์„ค์ •, server.myid ํ˜•์‹์œผ๋กœ ์‚ฌ์šฉ
Java ์„ค์น˜
๏ต JAVA Path ์„ค์ • (๏ƒ  Optional for Kafka)
vi /etc/profile
Shift + g ์ œ์ผ ๋ฐ‘์œผ๋กœ ์ด๋™
- PATH ์ถ”๊ฐ€ -
export JAVA_HOME=/usr/local/java
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=โ€œ.โ€
Esc ํ›„ shift + : wq ์„ค์ • ์ €์žฅ
- ํ™˜๊ฒฝ๋ณ€์ˆ˜ ๋ฐ˜์˜
source /etc/profile
๏ต java -versionjava ์„ค์น˜ ํ™•์ธ
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
VM ๋ณต์ œ ๋ฐ ์„ค์ •
- 1๊ฐœ์˜ VM(Master)์— ๋ชจ๋“  ์ž‘์—…์„ ํ•œ ํ›„ VM ๋ณต์ œ๋ฅผ ํ†ตํ•ด datanode๋ฅผ ๊ตฌ์ถ•ํ•œ๋‹ค.
slave1
slave1, slave2 2๊ฐœ์˜ VM๋ณต์ œ๊ฐ€ ํ•„์š”ํ•จ
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
VM ๋ณต์ œ ๋ฐ ์„ค์ •
Slave1
Salve2
๋ชจ๋‘ ์ž‘์—…
- ๋ณต์ œ๋œ VM์€ ๋„คํŠธ์›Œํฌ ์ดˆ๊ธฐํ™”๋ฅผ ์•ˆ ํ–ˆ๊ธฐ ๋•Œ๋ฌธ์— ์›๋ณธ VM๊ณผ ๋™์ผํ•œ MAC ์ฃผ์†Œ๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ๋‹ค.
- ๋์ž๋ฆฌ๋ฅผ ๋ณ€๊ฒฝํ•ด์„œ MAC ์ฃผ์†Œ๋ฅผ ๋‹ค๋ฅด๊ฒŒ ํ•ด ์ค€๋‹ค.
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
VM ๋ณต์ œ ๋ฐ ์„ค์ •
Slave1
Salve2
๋ชจ๋‘ ์ž‘์—…
- ๋ณต์ œ๋œ VM network ๋ณ€๊ฒฝ vi/etc/sysconfig/network-scripts/ifcfg-enp0s8
- IPADDR=192.168.0.31 ~ 32 ๋ณ€๊ฒฝ
- systemctl restart network ์‹คํ–‰
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
VM ๋ณต์ œ ๋ฐ ์„ค์ •
Slave1
Salve2
๋ชจ๋‘ ์ž‘์—…
- network ๋ณ€๊ฒฝ vi/etc/sysconfig/network
- NETWORKING=yes
HOSTNAME=slave1~2.kopo ์œผ๋กœ ๋ณ€๊ฒฝ
[root@localhost ~]# hostnamectl set-hostname master.kopo
vi /etc/sysconfig/network
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
๋งˆ์Šคํ„ฐ ๋ฐ ์Šฌ๋ ˆ์ด๋ธŒ SSH ์„ค์ •
- SSH๋ž€ (Secure Shell)๋Š” ์›๊ฒฉ ๋จธ์‹ ์— ๋กœ๊ทธ์ธํ•˜์—ฌ ๋ช…๋ น๋“ค์„ ์‹คํ–‰ํ•˜๋Š” ํ”„๋กœ๊ทธ๋žจ.
- openssh๊ฐ€ ๊ธฐ๋ณธ ์„ค์น˜๋˜์–ด ์žˆ์–ด openssh clients๋งŒ ์ถ”๊ฐ€๋กœ ์•„๋ž˜์™€ ๊ฐ™์ด ์„ค์น˜ํ•œ๋‹ค.
๋งŒ์•ฝ ์‹คํ–‰ ์‹œ package openssh-clinets.. already installed and latest version ์ด๋ผ๋Š” ๋ฌธ๊ตฌ๊ฐ€
์ถœ๋ ฅ๋˜๋ฉด ํ˜„์žฌ๋‹จ๊ณ„๋Š” ๋„˜์–ด๊ฐ„๋‹ค.
yum install openssh-clients
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
๋งˆ์Šคํ„ฐ ๋ฐ ์Šฌ๋ ˆ์ด๋ธŒ SSH ์„ค์ •
- key-gen ์ƒ์„ฑ
Master์—์„œ๋งŒ ssh-keygen ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ RSA ํ‚ค ํŽ˜์–ด๋ฅผ ์ƒ์„ฑํ•œ๋‹ค.
cd
ssh-keygen ์ž…๋ ฅ ํ›„ ์—”ํ„ฐ 3๋ฒˆ
ssh-copy-id โ€“i ~/.ssh/id_rsa.pub root@slave1.kopo
ssh-copy-id โ€“i ~/.ssh/id_rsa.pub root@slave1.kopo
์ž…๋ ฅ ํ›„ Are you sure you want
continue connection ? yes ์ž…๋ ฅ ํ›„
๊ฐ slave ์„œ๋ฒ„ root ๋น„๋ฐ€๋ฒˆํ˜ธ ์ž…๋ ฅ
source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
[root@slave1 data]# ll
total 4
-rw-r--r--. 1 root root 2 Oct 30 03:08 myid
drwxr-xr-x. 2 root root 131 Nov 27 02:57 version-2
[root@slave1 data]# pwd
/data
[root@slave1 data]# more myid
2
[root@slave1 data]#
๋ณต์ œ ๋…ธ๋“œ(slave1, slave2)์ฃผํ‚คํผ ์„ค์ •
vi myid๋กœ id๋ฅผ ์ˆ˜์ •ํ•œ๋‹ค.
slave1 : 2
slave2 : 3
์ฃผํ‚คํผ ์‹œ์ž‘๊ณผ ์ข…๋ฃŒ, ์ƒํƒœํ™•์ธ
[root@slave2 ~]# sh /usr/local/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@slave2 ~]# sh /usr/local/zookeeper/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
[root@slave2 ~]#
[root@master conf]# sh /usr/local/zookeeper/bin/zkCli.sh
์นดํ”„์นด ์„ค์น˜
https://kafka.apache.org/downloads
wget http://apache.mirror.cdnetworks.com/kafka/1.0.0/kafka_2.11-1.0.0.tgz
tar zxf kafka_2.11-1.0.0.tgz
ln -s kafka_2.11-1.0.0 kafka
scp kafka_2.11-1.0.0.tgz root@slave1:/usr/local
scp kafka_2.11-1.0.0.tgz root@slave2:/usr/local
vi /usr/local/kafka/config/server.properties
์นดํ”„์นด ์„ค์น˜
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# A comma seperated list of directories under which to store log files
log.dirs=/kdata1,/kdata2
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=master:2181,slave1:2181,slave2:2181/kopo-kafka
slave1 : 2
slave2 : 3
mkdir /kdata1
mkdir /kdata2
์‹ค ์šด์˜์‹œ ๋ณ„๋„ ํŒŒํ‹ฐ์…˜์œผ๋กœ ์„ค์ •
for more information:
https://kafka.apacheaorg/documentation/#brokerconfigs
* ์ฃผํ‚คํผ๋…ธ๋“œ๋ช… ๋ฏธ์ž‘์„ฑๆ™‚
์ฃผํ‚คํผ ๋ฃจํŠธ๋…ธ๋“œ์— ์ €์žฅ
์ด ์˜ˆ์ œ์—์„  kopo-kafka๋กœ ์„ค์ •ํ•˜์˜€์Œ
์นดํ”„์นด ์‹คํ–‰
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
/usr/local/kafka/bin/kafka-server-stop.sh
#...์ค‘๋žตโ€ฆ(๋‹ค๋Ÿ‰์˜ ๋ฉ”์‹œ์ง€ ํŒ์—… ํ›„)
#[kafkaServer id=1] started (kafka.server.KafkaServer) ๊ฐ€ ๋œจ๋ฉด ์„ฑ๊ณต
#์ฃผํ‚คํผ ํฌํŠธ ํ™•์ธ
โ€ข netstat โ€“ntlp | grep 2181
#์นดํ”„์นด ํฌํŠธ ํ™•์ธ
โ€ข netstat โ€“ntlp | grep 9092
์นดํ”„์นด ์‹คํ–‰
#์ฃผํ‚คํผ ์ง€๋…ธ๋“œ๋ฅผ ์ด์šฉํ•˜์—ฌ ์นดํ”„์นด ์ •๋ณด ํ™•์ธ
โ€ข cd /usr/local/zookeeper/bin
โ€ข ./zkCli.sh
#์ฃผํ‚คํผ ์ถ”๊ฐ€ ๋…ธ๋“œ ํ™•์ธ
โ€ข ls /
#ํด๋Ÿฌ์Šคํ„ฐ ๋…ธ๋“œ ์—ฐ๊ฒฐ ํ™•์ธ
โ€ข ls /kopo-kafka/brokers/ids
*ํ™•์ธ ํ›„ quit ์ž…๋ ฅ
์นดํ”„์นด ์‹คํ–‰
์ฐธ๊ณ > ์นดํ”„์นด ์„œ๋ฒ„ ๋กœ๊ทธ ํ™•์ธ
cat /usr/local/kafka/logs/server.log
์นดํ”„์นด ํ…Œ์ŠคํŠธ
์นดํ”„์นด ํ† ํ”ฝ ์ƒ์„ฑ
โ€ข Kafka-topics.sh ์ด์šฉํ•˜์—ฌ kopo-topic ์ƒ์„ฑ
โ€ข cd /home/kafka/bin
โ€ข ./kafka-topics.sh --zookeeper
master:2181,slave1:2181,slave2:2181/kopo-kafka --replication-factor 1
--partitions 1 --topic kopo-topic --create
[root@master ~]# /usr/local/kafka/bin/kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka
--replication-factor 1 --partitions 1 --topic ds-topic โ€“create
Created topic "ds-topic".
*ํ”„๋กœ๋“€์„œ ์ ‘์† ์‹œ > ํ”„๋กฌํ”„ํŠธ blink ํ™•์ธ ํ›„ ๋ฉ”์‹œ์ง€(Hello
World!) ์ž…๋ ฅ
์นดํ”„์นด ํ…Œ์ŠคํŠธ
์นดํ”„์นด ํ”„๋กœ๋“€์„œ
โ€ข Kafka-console-producer.sh ์ด์šฉํ•˜์—ฌ ํ”„๋กœ๋“€์„œ ์ ‘์†
โ€ข ./kafka-console-producer.sh --broker-list
master:9092,slave1:9092,slave2:9092 --topic kopo-topic
์นดํ”„์นด ํ…Œ์ŠคํŠธ
์นดํ”„์นด ์ปจ์Šˆ๋จธ ์ƒ์„ฑ
โ€ข Kafka-console-consumer.sh ์ด์šฉํ•˜์—ฌ kopo-topic ๋ฉ”์‹œ์ง€ ํ™•์ธ ๊ฐ€๋Šฅ
โ€ข ./kafka-console-consumer.sh --bootstrap-server
master:9092,slave1:9092,slave2:9092 --topic kopo-topic --from-
beginning
Kafka Data Model
Kafka Topic
source : https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85
image source : https://kafka.apache.org/documentation/
โ€ข topic : ๋ฉ”์„ธ์ง€ ์ข…๋ฅ˜
โ€ข partitions : topic ์ด ๋‚˜๋ˆ ์ง€๋Š” ๋‹จ์œ„
โ€ข Log : 1๊ฐœ์˜ ๋ฉ”์„ธ์ง€
โ€ข offset : ํŒŒํ‹ฐ์…˜ ๋‚ด์—์„œ ๊ฐ ๋ฉ”์‹œ์ง€๊ฐ€ ๊ฐ€์ง€๋Š” unique id
Topic
์นดํ”„์นด ํด๋Ÿฌ์Šคํ„ฐ๋Š” ํ† ํ”ฝ์—๋‹ค๊ฐ€ ๋ฐ์ดํ„ฐ๋ฅผ ์ €์žฅํ•จ.
์นดํ”„์นด์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ตฌ๋ถ„ํ•˜๊ธฐ ์œ„ํ•œ ๋‹จ์œ„
Partition
ํ•˜๋‚˜์˜ ํ† ํ”ฝ์„ ๋ถ„ํ•  ํ•œ ๊ฒƒ (๋ณ‘๋ ฌ์ฒ˜๋ฆฌ!)
ํ•˜๋‚˜์˜ ํŒŒํ‹ฐ์…˜ VS ๋ณต์ˆ˜์˜ ํŒŒํ‹ฐ์…˜
Kafka Topic
Offset
Unique, Sequential Number in the Partition
์˜คํ”„์…‹์„ ์ด์šฉํ•ด ๋ฉ”์‹œ์ง€์˜ ์ˆœ์„œ๋ฅผ ๋ณด์žฅํ•จ
Kafka Partition
ํŒŒํ‹ฐ์…˜ ์ˆ˜์˜ ์ฆ๊ฐ€
1. ํŒŒ์ผ ํ•ธ๋“ค๋Ÿฌ์˜ ๋‚ญ๋น„
๊ฐ ํŒŒํ‹ฐ์…˜์€ ๋ธŒ๋กœ์ปค์˜ ๋””๋ ‰ํ† ๋ฆฌ์™€ ๋งคํ•‘ ๋˜๊ณ , ์ €์žฅ๋˜๋Š” ๋ฐ์ดํ„ฐ์— ๋Œ€ํ•œ ํŒŒ์ผ ํ•ธ๋“ค ์ˆ˜๊ฐ€ ๋งŽ์•„์ ธ ๋ฆฌ์†Œ์Šค ์ฆ๊ฐ€
2. ์žฅ์•  ๋ณต๊ตฌ ์‹œ๊ฐ„ ์ฆ๊ฐ€
ํ† ํ”ฝ์€ ์—ฌ๋Ÿฌ ๊ฐœ์˜ ํŒŒํ‹ฐ์…˜์œผ๋กœ ๋‚˜๋ˆ ์ง€๊ณ  ๋ธŒ๋กœ์ปค์—๋Š” ์—ฌ๋Ÿฌ ๊ฐœ์˜ ํŒŒํ‹ฐ์…˜์กด์žฌ
๏ƒ  ํ”„๋กœ๋“€์„œ, ์ปจ์Šˆ๋จธ์˜ ๋ชฉํ‘œ ์ฒ˜๋ฆฌ๋Ÿ‰๊ณผ ์ฒ˜๋ฆฌ ์†๋„๋ฅผ ๊ธฐ์ค€์œผ๋กœ ์ ์ ˆํ•œ ์ˆ˜์˜ ํŒŒํ‹ฐ์…˜์ˆ˜๋ฅผ ์„ค์ •ํ•ด์•ผํ•จ
โ€ป ํŒŒํ‹ฐ์…˜์ˆ˜ ์ฆ๊ฐ€๋Š” ๊ฐ€๋Šฅํ•˜๋‚˜ ์ค„์ด๋Š” ๊ฒƒ์€ ๋ถˆ๊ฐ€ํ•จ
Consumer
Consumer
Consumer
์„œ๋ฒ„ ์ˆ˜ ์„œ๋ฒ„๋‹น ๋ฉ”์‹œ์ง€ ์ „์†ก ์ˆ˜ ํ•ฉ๊ณ„ ํ•„์š” ํŒŒํ‹ฐ์…˜ ์ˆ˜
Producer 3 10 msg /sec 3*10 msg /sec 3
Consumer 6 5 msg /sec 6*5 msg /sec ?
partition
Producer
Producer
Consumer
Consumer
<Broker>
Producer
partition
partition
Consumer
Kafka Partition
์ฒ˜๋ฆฌ๋Ÿ‰ 10 msg /sec
์ฒ˜๋ฆฌ๋Ÿ‰ 10 msg /sec
์ฒ˜๋ฆฌ๋Ÿ‰ 10 msg /sec
Consumer
Consumer
Consumer
์„œ๋ฒ„ ์ˆ˜ ์„œ๋ฒ„๋‹น ๋ฉ”์‹œ์ง€ ์ „์†ก ์ˆ˜ ํ•ฉ๊ณ„ ํ•„์š” ํŒŒํ‹ฐ์…˜ ์ˆ˜
Producer 3 10 msg /sec 3*10 msg /sec 3
Consumer 6 5 msg /sec 6*5 msg /sec 6
partition
Producer
Producer
Consumer
Consumer
<Broker>
Producer
partition
partition
Consumer
partition
partition
partition
Kafka Partition
ํ•˜๋‚˜์˜ consumer๊ฐ€ ์ˆ˜์‹ ํ•  ์ˆ˜ ์žˆ๋Š” ์ฒ˜๋ฆฌ๋Ÿ‰์ด 5 msg / sec ์ด๋ฏ€๋กœ
partition ์ˆซ์ž๋ฅผ 2๋ฐฐ๋กœ ๋Š˜๋ ค์ค˜์•ผ ํ•จ
Kafka Partition & Topic
News-Topic
Partition01
News-Topic
Partition01
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition02
News-Topic
Partition02
Sports-Topic
Partition01
Sports-Topic
Partition01
Sports-Topic
Partition01
Replication-factor : 3
Kafka Replication
๏ต ์žฅ์•  ๋Œ€์‘ ๊ธฐ๋Šฅ
๏ต ํŒŒํ‹ฐ์…˜์„ ๋ณต์ œํ•˜๋Š” ๊ธฐ๋Šฅ
News-Topic
Partition01
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition02
Sports-Topic
Partition01
Sports-Topic
Partition01
Replication-factor : 2
Kafka Replication
๏ต kafka-topics.sh -โ€“zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka
-โ€“topic kopo-topic -โ€“describe
๏ต Leader ๋ฒˆํ˜ธ๊ฐ€ ํŒŒํ‹ฐ์…˜์˜ ๋ฆฌ๋” ๋ธŒ๋กœ์ปค ๋ฒˆํ˜ธ
๏ต Replicas ๋ฒˆํ˜ธ : ํŒŒํ‹ฐ์…˜์— ์ฐธ์—ฌํ•œ ๋ธŒ๋กœ์ปค ๋ฒˆํ˜ธ
Replication-factor ํ† ํ”ฝ์„ ์ƒ์„ฑํ•  ๋•Œ ์ ์šฉ ์šด์˜ ์ค‘์—๋„ ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ํŒฉํ„ฐ ๊ฐ’์€ ๋ณ€๊ฒฝ๊ฐ€๋Šฅ
ํ† ํ”ฝ์ด ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ ํŒฉํ„ฐ 1๋กœ ์„ค์ •๋˜์–ด ์žˆ์œผ๋ฉด ํ•˜๋‚˜์˜ ๋ธŒ๋กœ์ปค์—๋งŒ ๋ฉ”์‹œ์ง€๋ฅผ ์ €์žฅํ•˜๋Š” ์ƒํƒœ
Kopo-topic(๋ฆฌ๋”) , kopo-topic(ํŒ”๋กœ์›Œ) ์ฝ๊ธฐ, ์“ฐ๊ธฐ๋Š” ๋ฆฌ๋”๋ฅผ ํ†ตํ•ด์„œ๋งŒ ์ผ์–ด๋‚จ
Kafka ISR
๏ต ISR : In Sync Replica
๏ต ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋˜๊ณ  ์žˆ๋Š” ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ ๊ทธ๋ฃน์„ ์˜๋ฏธํ•จ
๏ต ISR๊ทธ๋ฃน์˜ ๊ตฌ์„ฑ์›๋งŒ์ด ๋ฆฌ๋”์˜ ์ž๊ฒฉ์„ ๊ฐ–์Œ
๏ต ๋ฆฌ๋”(Leader)๋งŒ ์ฝ๊ณ  ์“ฐ๊ธฐ, Follower๋Š” Leader์˜ ๋ฐ์ดํ„ฐ๋ฅผ ์ฝ์–ด์„œ ๋™๊ธฐํ™”(Sync)
๏ต ๋ฆฌ๋”์™€ ๋ฐ์ดํ„ฐ ๋™๊ธฐํ™”์ž‘์—…์„ ํ•˜๊ณ  ์ด๊ฒƒ์„ ์œ ์ง€ํ•˜์—ฌ ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜์˜ ์‹ ๋ขฐ์„ฑ์„ ๋†’์ž„.
News-Topic
Partition01
News-Topic
Partition01
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition02
News-Topic
Partition02
Kafka ISR
ISR Leader Follower Follower
ISR Leader FollowerFollower
์žฅ์•  ๋ฐœ์ƒ์‹œ
ํ•ด๋‹น ISR์—์„œ ์ถ”๋ฐฉ๋จ
์žฅ์•  ๋ฐœ์ƒ์‹œ
ํ•ด๋‹น ISR์—์„œ ์ถ”๋ฐฉ๋˜๊ณ  ๋‚˜๋จธ์ง€
Follower ์ค‘์—์„œ Leader๋ฅผ ์ž๋™ ์„ ์ถœํ•จ
Leader๋Š” Follower๋“ค์ด ์ฃผ๊ธฐ์ ์œผ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ์š”์ฒญํ•˜๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธ
๋งŒ์•ฝ, ์„ค์ •๋œ ์ผ์ •์ฃผ๊ธฐ(replica.lag.time.max.ms)๋งŒํผ ํ™•์ธ ์š”์ฒญ์ด
์˜ค์ง€์•Š๋Š”๋‹ค๋ฉด, Leader ๋Š” ํ•ด๋‹น Follower์˜ ์ด์ƒ์„ ๊ฐ์ง€ํ•˜๊ณ , ISR์—์„œ ์ถ”๋ฐฉ
Kafka ISR
๏ต ํด๋Ÿฌ์Šคํ„ฐ๋‚ด์˜ ๋ชจ๋“  ๋ธŒ๋กœ์ปค๊ฐ€ ๋‹ค์šด์ด ๋œ๋‹ค๋ฉด?
1. ๋งˆ์ง€๋ง‰ ๋ฆฌ๋”๊ฐ€ ์‚ด์•„๋‚˜๊ธฐ๋ฅผ ๊ธฐ๋‹ค๋ฆฐ๋‹ค.
๏ƒ  ๋ฉ”์‹œ์ง€ ์†์‹ค์—†์ด ํ”„๋กœ๋“€์„œ์˜ ์š”์ฒญ์„ ์ฒ˜๋ฆฌํ•˜๋ฉด์„œ ์„œ๋น„์Šค ์ง€์†๊ฐ€๋Šฅ
2. ๋จผ์ € ์‚ด์•„๋‚˜๋Š” Broker๊ฐ€ ์ž๋™์œผ๋กœ ๋ฆฌ๋”๊ฐ€ ๋œ๋‹ค.
๏ƒ  ๋ฉ”์‹œ์ง€ ์ผ๋ถ€ ์†์‹ค๋  ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์ง€๋งŒ ์„œ๋น„์Šค๋Š” ์กฐ์†ํžˆ ์ •์ƒํ™”ํ•  ์ˆ˜ ์žˆ์Œ(downtime ์ตœ์†Œํ™”)
๏ต ๋ฐ์ดํ„ฐ์˜ ์™„์ „ํ•จ์„ ์ค‘์‹œํ•˜๋А๋ƒ, ์ง€์†์ ์ด ์„œ๋น„์Šค๋ฅผ ์ค‘์‹œ ํ•˜๋А๋ƒ์— ๋”ฐ๋ผ ์„ค์ •ํ•  ์ˆ˜ ์žˆ์Œ.
Unclean.leader.election.enable = false == > 1๋ฒˆ ์ผ๊ด€์„ฑ
Unclean.leader.election.enable = true == > 2๋ฒˆ ๊ฐ€์šฉ์„ฑ
Kafka Producer
์นดํ”„์นด ํ† ํ”ฝ ์ƒ์„ฑ
โ€ข Kafka-topics.sh ์ด์šฉํ•˜์—ฌ kopo-topic ์ƒ์„ฑ
โ€ข cd /home/kafka/bin
โ€ข ./kafka-topics.sh --zookeeper
master:2181,slave1:2181,slave2:2181/kopo-kafka --replication-factor 1
--partitions 1 --topic kopo-topic --create
*ํ”„๋กœ๋“€์„œ ์ ‘์† ์‹œ > ํ”„๋กฌํ”„ํŠธ blink ํ™•์ธ ํ›„ ๋ฉ”์‹œ์ง€(Hello
World!) ์ž…๋ ฅ
์นดํ”„์นด ํ”„๋กœ๋“€์„œ
โ€ข Kafka-console-producer.sh ์ด์šฉํ•˜์—ฌ ํ”„๋กœ๋“€์„œ ์ ‘์†
โ€ข ./kafka-console-producer.sh --broker-list master:9092,slave1:9092,slave2:9092 --topic
kopo-topic -โ€“request-required-acks 1
Kafka Producer
ํ”„๋กœ๋“€์„œ ์ฃผ์š” ์˜ต์…˜
Broker-list
์นดํ”„์นด ํด๋Ÿฌ์Šคํ„ฐ์— ์ฒ˜์Œ์—ฐ๊ฒฐ ํ•˜๊ธฐ ์œ„ํ•œ ํ˜ธ์ŠคํŠธ์™€ ํฌํŠธ ์ •๋ณด๋กœ ๊ตฌ์„ฑ๋œ ๋ฆฌ์ŠคํŠธ ์ •๋ณด ๋‚˜ํƒ€๋ƒ„
ํ˜ธ์ŠคํŠธ ํ•˜๋‚˜๋งŒ ์“ธ ์ˆ˜ ๋„ ์žˆ์ง€๋งŒ ์žฅ์• ๊ฐ€ ์ƒ๊ธธ ์ˆ˜ ์žˆ๊ธฐ๋•Œ๋ฌธ์— ๋ฆฌ์ŠคํŠธ ์ „์ฒด๋ฅผ ์ž…๋ ฅํ•˜๋Š” ๊ฑธ ๊ถŒ์žฅ
Acks
ํ”„๋กœ๋“€์„œ๊ฐ€ ์นดํ”„์นด ํ† ํ”ฝ์˜ ๋ฆฌ๋”์—๊ฒŒ ๋ฉ”์‹œ์ง€๋ฅผ ๋ณด๋‚ธ ํ›„ ์š”์ฒญ์„ ์™„๋ฃŒํ•˜๊ธฐ ์ „ ์Šน์ธ์˜ ์ˆ˜
ํ•ด๋‹น ์˜ต์…˜์˜ ์ˆ˜๊ฐ€ ์ž‘์œผ๋ฉด ์„ฑ๋Šฅ์ด ์ข‹์ง€๋งŒ, ๋ฉ”์‹œ์ง€ ์†์‹ค ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์Œ
Producer
๋ฉ”์‹œ์ง€๋ฅผ ์ƒ์‚ฐํ•ด์„œ ์นดํ”„์นด์˜ ํ† ํ”ฝ์œผ๋กœ ๋ฉ”์‹œ์ง€๋ฅผ ๋ณด๋‚ด๋Š” ์—ญํ• ์„ ํ•˜๋Š” ์„œ๋ฒ„, ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋“ฑ
์ฃผ์š”๊ธฐ๋Šฅ
1. ๊ฐ๊ฐ์˜ ๋ฉ”์‹œ์ง€๋ฅผ Topic์˜ Partition์— ๋งคํ•‘ํ•˜๊ณ  Partition์˜ ๋ฆฌ๋”์— ์š”์ฒญ์„ ๋ณด๋‚ด๋Š” ๊ฒƒ
2. key ๊ฐ’์„ ์ •ํ•ด ํ•ด๋‹น key ๋ฅผ ๊ฐ€์ง„ ๋ชจ๋“  ๋ฉ”์‹œ์ง€๋ฅผ ๋™์ผํ•œ ํŒŒํ‹ฐ์…˜์œผ๋กœ ์ „์†ก
๋งŒ์•ฝ key ๊ฐ’์„ ์ž…๋ ฅํ•˜์ง€์•Š์œผ๋ฉด Partition ๋ผ์šด๋“œ๋กœ๋นˆ ๋ฐฉ์‹์œผ๋กœ ํŒŒํ‹ฐ์…˜์— ๊ท ๋“ฑํ•˜๊ฒŒ ๋ถ„๋ฐฐ
Kafka Producer
from kafka import KafkaProducer
producer = kafkaProducer(acks=1, compresstion_type=โ€œgzaipโ€, bootstrap_server=โ€˜master:9092,slave1:9092,slave2:9092โ€™)
for i range(1,10):
if I % 2 == 1:
producer.send(โ€˜kopo-topicโ€™,key=โ€˜1โ€™, value=โ€˜Kopo Testโ€™)
else :
producer.send(โ€˜kopo-topicโ€™,key=โ€˜2โ€™, value=โ€˜Korea Testโ€™)
[Producer] Kafka-python https://github.com/dpkp/kafka-python
โ˜… pip install kafka-python ์œผ๋กœ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์„ค์น˜ (์‚ฌ์ „ ์กฐ๊ฑด : python 3.xx ์„ค์น˜)
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='master:9092,slave1:9092,slave2:9092')
future = producer.send('kopo-topic','HWY TEST')
result = future.get(timeout=60)
print(result)
print("END")
[root@slave1 pyKafkaTest]# python producer-option.py
RecordMetadata(topic='kopo-topic', partition=0, topic_partition=TopicPartition(topic='kopo-topic', partition=0),
offset=7, timestamp=1542334214067, checksum=None, serialized_key_size=-1, serialized_value_size=8)
END
์ด๋ฅผ ํƒ€ ์„œ๋ฒ„(slave2)์—์„œ consumer๋กœ ํ™•์ธํ•œ ๊ฒฐ๊ณผ๋Š” ์•„๋ž˜์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค.
[root@master ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092
--topic kopo-topic --from-beginning
HWY TEST
Kafka Consumer
์นดํ”„์นด ์ปจ์Šˆ๋จธ ์ƒ์„ฑ
โ€ข Kafka-console-consumer.sh ์ด์šฉํ•˜์—ฌ kopo-topic ๋ฉ”์‹œ์ง€ ํ™•์ธ ๊ฐ€๋Šฅ
โ€ข ./kafka-console-consumer.sh --bootstrap-server
master:9092,slave1:9092,slave2:9092 --topic kopo-topic --from-
beginning
from kafka import KafkaConsumer
consumer = KafkaConsumer('kopo-topic',group_id='kopo-consumer',bootstrap_servers='master:9092,slave1:9092,slave2:9092',enable_auto_commit=True,auto_offset_reset='latest')
for message in consumer:
print "Topic : %s, Partition: %d, Offset: %d, Key: %s, Value: %s" % (message.topic, message.partition, message.offset, message.key, message.value.decode('utf-8'))
https://github.com/dpkp/kafka-python
[Consumer] Kafka-python
โ˜… pip install kafka-python ์œผ๋กœ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์„ค์น˜ (์‚ฌ์ „ ์กฐ๊ฑด : python 3.xx ์„ค์น˜)
late response
[Consumer] Kafka-python https://github.com/dpkp/kafka-python
from kafka import KafkaConsumer
consumer = KafkaConsumer('kopo-topic',group_id='kopo-consumer',bootstrap_servers='master:9092,slave1:9092,slave2:9092โ€™,
enable_auto_commit=True,auto_offset_reset='latest',fetch_min_bytes=1)
while True:
message = consumer.poll(1.0)
#print (message.keys()) #print (message.values())
#print "Offset: %d, Key: %s, Value: %s" % (message.offset, message.key, message.value.decode('utf-8'))
for tp, mg in message.items():
for m in mg:
print ("%s : %d %d : key=%s value=%s" % ( tp.topic, tp.partition, m.offset, m.key, m.value))
consumer ์ฃผ์š” ์˜ต์…˜:
https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html#
https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html
Partition and Message
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
kafka-topics.sh โ€“zookeeper master:2181,slave1:2181,slave2:2181/news-kafka โ€“topic news-topic โ€“partitions 3 โ€“replication-factor 1 --create
kafka-console-producer.sh
โ€“broker-list master:9092,slave1:9092,slave2:9092
โ€“topic news-topic
1
2
3
4
5
1 4
2 5
3
kafka-console-consumer.sh
--bootstrap-server master:9092,slave1:9092,slave2:9092
โ€“topic news-topic โ€“from-beginning
1
4
2
5
3
kafka-console-consumer.sh
--bootstrap-server master:9092,slave1:9092,slave2:9092
โ€“topic news-topic โ€“partition 0 โ€“from-beginning
kafka-console-consumer.sh
--bootstrap-server master:9092,slave1:9092,slave2:9092
โ€“topic news-topic โ€“partition 1 โ€“from-beginning
kafka-console-consumer.sh
--bootstrap-server master:9092,slave1:9092,slave2:9092
โ€“topic news-topic โ€“partition 2 โ€“from-beginning
consumer๋Š” producer๊ฐ€ ์–ด๋–ค ์ˆœ์„œ๋กœ ๋ฉ”์‹œ์ง€๋ฅผ ๋ณด๋ƒˆ๋Š”์ง€ ๋ชจ๋ฆ„
Topic์˜ Partition์ด ๋ณต์ˆ˜์ธ ๊ฒฝ์šฐ, ๋ฉ”์‹œ์ง€์˜ ์ˆœ์„œ๋ฅผ ๋ณด์žฅํ•˜์ง€ ์•Š์Œ
(Partition๋‚ด์˜ ์ˆœ์„œ๋Š” ๋ณด์žฅ๋˜๋‚˜, Partition๊ฐ„ ์ˆœ์„œ๋Š” ๋ณด์žฅํ•˜์ง€ ์•Š์Œ)
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer
Consumer Group
Kafka๋Š” ํ•˜๋‚˜์˜ Topic์— ๋Œ€ํ•˜์—ฌ ์—ฌ๋Ÿฌ Consumer Group์ด ๋™์‹œ์— ์ ‘์†ํ•˜์—ฌ ๋ฉ”์‹œ์ง€๋ฅผ ์ˆ˜์‹ ํ•  ์ˆ˜ ์žˆ๋‹ค.
์ฒ˜๋ฆฌ๋Ÿ‰ โ†‘, ๋ถ€ํ•˜ โ†‘
consumer๋งŒ ์ถ”๊ฐ€ํ•  ๊ฒฝ์šฐ,
: ๊ธฐ์กด์˜ consumer์™€ offset์ •๋ณด๊ฐ€
์—‰์ผœ์„œ ๋ฉ”์‹œ์ง€ ์ค‘๋ณต์ฒ˜๋ฆฌ ๋ฐœ์ƒ
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer
Consumer
Consumer
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer
Consumer
Consumer
Consumer
Topic์˜ Partition์€ ํ•˜๋‚˜์˜ Consumer๋งŒ ์—ฐ๊ฒฐ ๊ฐ€๋Šฅ
Idle
Consumer Group
Partition์„ ์ถ”๊ฐ€ํ•ด์•ผ ํ•จ
Consumer Group
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer
Consumer
Consumer
Consumer sends HeartBeat to broker
- consumer polls
- message offset commit
* heartbeat.interval.ms = 3000 by default
* session.timeout.ms = 10000 by default
Consumer๊ฐ€ heartbeat.interval.ms ๋™์•ˆ heartbeat๋ฅผ
๋ณด๋‚ด์ง€ ์•Š๋Š”๋‹ค๋ฉด?
session.timeout.ms ์ดํ›„์— ๋ฐ”๋กœ rebalance ์‹œ์ž‘
(์ฆ‰, ํ•ด๋‹น consumer๋Š” out๋จ)
Dead
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer
Consumer
Consumer
Consumer Group
Consumer
Group
<kopo-
consumer2>
Consumer
Consumer
Consumer
consumer group
<kopo-consumer>
consumer group
<kopo-consumer2>
์ปจ์Šˆ๋จธ ๋ฆฌ์ŠคํŠธ ๋ช…๋ น์–ด
--bootstrap : ๋ธŒ๋กœ์ปค๋ฆฌ์ŠคํŠธ ์„ค์ •
--list : ์ปจ์Šˆ๋จธ ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ณด์—ฌ์คŒ.
./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:092,slave2:9092 --list
Kafka Consumer
[root@master bin]# ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:092,slave2:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=
Note: This will not show information about old Zookeeper-based consumers.
[2019-05-15 23:03:11,612] WARN Connection to node -2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kopo-consumer
Kafka Consumer
์ปจ์Šˆ๋จธ์˜ ์ƒ์„ธ ์ •๋ณด๋ฅผ ์กฐํšŒํ•จ
--bootstrap-server : ๋ธŒ๋กœ์ปค ๋ฆฌ์ŠคํŠธ ์ž…๋ ฅ
--group : ์ปจ์Šˆ๋จธ ๊ทธ๋ฃน ์ด๋ฆ„ ์„ค์ •
--describe : ์ƒ์„ธ์ •๋ณด ๋ณด๊ธฐ
*LAG : ํ˜„์žฌํ† ํ”ฝ์˜ ์ €์žฅ๋œ ๋ฉ”์‹œ์ง€์™€ ์ปจ์Šˆ๋จธ๊ฐ€ ๊ฐ€์ ธ๊ฐ„ ๋ฉ”์‹œ์ง€์˜ ์ฐจ์ด
ํ”„๋กœ๋“€์„œ์™€ ์ปจ์Šˆ๋จธ์˜ ์ฒ˜๋ฆฌ ์†๋„์ฐจ์ด๋ฅผ ์•Œ ์ˆ˜ ์žˆ์Œ
./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --group kopo-consumer --describe
[root@master bin]# ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --group kopo-consumer --describe
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Note: This will not show information about old Zookeeper-based consumers.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test-topic 0 21 21 0 kafka-python-1.4.3-3d11d79a-04f0-4da5-a1b9-808c34f99865/192.168.0.31 kafka-python-1.4.3
kopo-topic 0 425 426 1 - - -
[root@master bin]#
Consumer Commit
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer 1
Consumer 2
Consumer 3
Consumer๋Š” Matching Partition์— ๋Œ€ํ•ด
์ˆ˜์‹ ํ•œ ๋ฉ”์‹œ์ง€์˜ offset ์ •๋ณด๋ฅผ ๊ธฐ๋กํ•˜๊ณ  ์žˆ์Œ
์ด๋Ÿฌํ•œ ๋™์ž‘์„ Commit์ด๋ผ๊ณ  ํ•จ
โ€ข enable.auto.commit=true
โ€ข auto.commit.interval.ms= 5 (sec) by default
1 2 3 4 5 6 7 8 9 10Partition 03
Consumer 3 1 2 3 4
5 sec 5 sec
5
5๊นŒ์ง€ ์ฒ˜๋ฆฌํ–ˆ์œผ๋‚˜,
Commit์€ 4๊นŒ์ง€ ํ•œ ์ƒํ™ฉ
Consumer Commit
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer 1
Consumer 2
1 2 3 4 5 6 7 8 9 10Partition 03
Consumer 3 1 2 3 4
5 sec 5 sec
5 5๊นŒ์ง€ ์ฒ˜๋ฆฌํ–ˆ์œผ๋‚˜,
Commit์€ 4๊นŒ์ง€ ํ•œ ์ƒํ™ฉ
Consumer 2 5 6 Consumer 2๋Š” 5 ๋ถ€ํ„ฐ ์ฝ์–ด๋“ค์ž„
Consumer Commit
Manual Commit
๋ฉ”์‹œ์ง€๊ฐ€ ์ฒ˜๋ฆฌ๊ฐ€ ์™„๋ฃŒ๋  ๋•Œ๊นŒ์ง€ Commit์„ ๋ณด๋ฅ˜
โ€ข enable.auto.commit : false
โ€ข commitSync ํ•จ์ˆ˜ ํ˜ธ์ถœ
[์ฐธ๊ณ ] ํŠน์ • ํŒŒํ‹ฐ์…˜์ด๋‚˜ ํŠน์ • ์˜คํ”„์…‹๋ถ€ํ„ฐ ๊ฐ€์ ธ์˜ค๋Š” ๊ธฐ๋Šฅ๋„ ์žˆ์Œ
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
News-Topic
Partition01
News-Topic
Partition02
News-Topic
Partition03
Consumer
Group
<kopo-consumer>
Consumer 1
Consumer 2
์ฐธ๊ณ 
How to find Zookeeper Leader in
zookeeper ensemble?
To identify a Zookeeper leader/follower, there are few possible options. Mentioning 2 for keeping this document simple.
1. Check the zookeeper log file on each node, and grep as below:
# grep LEAD /var/log/zookeeper/zookeeper-zookeeper-server-xyz.out
2016-02-29 22:33:47,113 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer@829] - LEADING
2016-02-29 22:33:47,114 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:Leader@358] - LEADING - LEADER ELECTION TOOK - 9066
2. Use "nc" command to listen TCP communication on port 2181 and determine if the ZooKeeper server is a leader or a follower.
Bellow is a way to check the same:
# echo stat | nc localhost 2181
Zookeeper version: 3.4.6-3485--1, built on 12/16/2015 02:35 GMT
Clients:
/172.25.16.243:37619[1](queued=0,recved=54,sent=54)
/172.25.16.253:58745[1](queued=0,recved=53,sent=53)
Latency min/avg/max: 0/1/362
Received: 627
Sent: 626
Connections: 9
Outstanding: 0
Zxid: 0x400000003
Mode: leader
Node count: 182
Here, we see that the Mode is shown as "leader". If it is a follower, the Mode will be seen as "follower".
For example:
# echo stat | nc localhost 2181 | grep Mode
Mode: leader
# echo stat | nc localhost 2181 | grep Mode
Mode: follower
source : https://community.hortonworks.com/content/supportkb/49435/how-to-find-zookeeper-leader-in-zookeeper-ensemble.html
Why Kafka is so fast?
- page cache
source : https://www.slideshare.net/baniuyao/kafka-24299168
๊ธฐํƒ€ ๋ช…๋ น์–ด
ํ† ํ”ฝ ์ƒ์„ฑ ๋ช…๋ น์–ด
--zookeeper : ์ฃผํ‚คํผ ์ •๋ณด,
--replication-factor : ํ† ํ”ฝ ๋ณต์ œ ์„ค์ •,
--partitions : ํ† ํ”ฝ ๋‚ด ํŒŒํ‹ฐ์…˜์ˆ˜ ์„ค์ •
--topic : ํ† ํ”ฝ๋„ค์ž„ ์„ค์ •, --create : ํ† ํ”ฝ์ƒ์„ฑ
1.ํ† ํ”ฝ ์ƒ์„ฑ ๋ช…๋ น์–ด
./kafka-topics.sh โ€“zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka
-โ€“replication-factor 1 --partitions 1 -โ€“topic kopo-test --create
ํ† ํ”ฝ ๋ฆฌ์ŠคํŠธ ๋ช…๋ น์–ด
--zookeeper : ์ฃผํ‚คํผ ์ •๋ณด
--list : ํ•ด๋‹น ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ๊ฐ€์ง€๊ณ  ์žˆ๋Š” ํ† ํ”ฝ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ณด์—ฌ์คŒ
2.ํ† ํ”ฝ ๋ฆฌ์ŠคํŠธ ๋ช…๋ น์–ด
./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --list
ํ† ํ”ฝ ์ƒ์„ธ๋ณด๊ธฐ ๋ช…๋ น์–ด
--zookeeper : ์ฃผํ‚คํผ ์ •๋ณด, --topic : ์ƒ์„ธ์ •๋ณด๋ฅผ ์•Œ๊ณ  ์‹ถ์€ ํ† ํ”ฝ์ด๋ฆ„ ์„ค์ •
--describe : ํ•ด๋‹น ํ† ํ”ฝ์˜ ์ƒ์„ธ ์ •๋ณด๋ฅผ ํ™•์ธ
3.ํ† ํ”ฝ ์ƒ์„ธ๋ณด๊ธฐ ๋ช…๋ น์–ด
./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --topic test-topic --describe
Topic:test-topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
๋””์Šคํฌ ๊ณต๊ฐ„ ํ™•๋ณดํ•˜๋Š” ๊ฐ€์žฅ ์ข‹์€ ๋ฐฉ๋ฒ•์€ ๊ณต๊ฐ„์„ ๋งŽ์ด ์ฐจ์ง€ํ•˜๋Š” ํ† ํ”ฝ์˜ ๋ณด๊ด€์ฃผ๊ธฐ๋ฅผ ์ค„์ด๋Š” ๊ฒƒ
๋ธŒ๋กœ์ปค์˜ ๊ธฐ๋ณธ ์„ค์ • ๊ฐ’์€ ๋ณด๊ด€์ฃผ๊ธฐ 7์ผ๋กœ ์„ค์ • ๋˜์–ด์žˆ์Œ.
--zookeeper : ์ฃผํ‚คํผ ์ •๋ณด, --alter : ๋ณ€๊ฒฝ ์„ค์ •, --entity-type : topics๋ฅผ ์ถ”๊ฐ€,
--entity-name : ๋ณ€๊ฒฝํ•˜๊ณ ์ž ํ•˜๋Š” ํ† ํ”ฝ์ด๋ฆ„ ์„ค์ •
--add-config : ๋ณด๊ด€์ฃผ๊ธฐ๋ฅผ 1์‹œ๊ฐ„์„ ์˜๋ฏธํ•˜๋Š” retention.ms=360000๋ฅผ ์ž…๋ ฅ
*์ถ”๊ฐ€๋œ ์˜ต์…˜์„ ์‚ญ์ œํ•˜๊ณ ์ž ํ• ๋• -โ€“delete-config retention.ms๋ฅผ ์ž…๋ ฅํ•˜๋ฉด ๋Œ
4.ํ† ํ”ฝ์˜ ๋ฉ”์‹œ์ง€ ๋ณด๊ด€์ฃผ๊ธฐ ์„ค์ • ๋ณ€๊ฒฝ
./kafka-configs.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --alter --entity-type topics --entity-name test-topic --add-config retention.ms=3600000
./kafka-configs.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --alter --entity-type topics -entity-name test-topic --delete-config retention.ms
ํ† ํ”ฝ ๋ฉ”์‹œ์ง€ ๋ณด๊ด€์ฃผ๊ธฐ ๋ณ€๊ฒฝ
ํ† ํ”ฝ ๋ฉ”์‹œ์ง€ ๋ณด๊ด€์ฃผ๊ธฐ ์˜ต์…˜ ์‚ญ์ œ
ํ† ํ”ฝ์˜ ํŒŒํ‹ฐ์…˜ ์ˆ˜ ๋ณ€๊ฒฝ
ํ† ํ”ฝ์˜ ํŒŒํ‹ฐ์…˜์€ ์ฆ๊ฐ€๋Š” ๊ฐ€๋Šฅํ•˜์ง€๋งŒ, ๊ฐ์†Œ๋Š” ๋ถˆ๊ฐ€๋Šฅํ•จ
์ฒ˜์Œ ํ† ํ”ฝ์„ ์ƒ์„ฑํ•  ๋•Œ ํŒŒํ‹ฐ์…˜ ์ˆ˜๋ฅผ ์ฒ˜๋ฆฌ๋Ÿ‰์— ๋Œ€ํ•ด ๋ฏธ๋ฆฌ ํ™•์ธํ•˜๊ณ  ์„ค์ •!
--zookeeper : ์ฃผํ‚คํผ ์ •๋ณด
--alter : ๋ณ€๊ฒฝ ์˜ต์…˜ ์„ ํƒ
--topic : ํ•ด๋‹น ํ† ํ”ฝ ์„ค์ •
--partitions : ํŒŒํ‹ฐ์…˜ ์ˆ˜ ์„ค์ •
./kafka-topics.sh -zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka -alter -topic test-topic -partitions 2
5.ํ† ํ”ฝ์˜ ํŒŒํ‹ฐ์…˜ ์ˆ˜ ๋ณ€๊ฒฝ
[root@master bin]# ./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --topic test-topic --describe
Topic:test-topic PartitionCount:2 ReplicationFactor:1 Configs:retention.ms=3600000
Topic: test-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: test-topic Partition: 1 Leader: 2 Replicas: 2 Isr: 2
manager
์นดํ”„์นด ๋งค๋‹ˆ์ € ์„ค์น˜
์นดํ”„์นด ๋งค๋‹ˆ์ €๋ฅผ wget์œผ๋กœ ๋‹ค์šด์„ ๋ฐ›๋Š”๋‹ค.
wget https://github.com/yahoo/kafka-manager/archive/1.3.3.17.zip
Unzip ๋ช…๋ น์–ด๋กœ
์••์ถ•์„ ํ‘ผ๋‹ค.
Unzip 1.3.3.17.zip
Cd kafka-manager-1.3.3.17 ๋กœ ์ด๋™ ํ•œ ๋’ค
Zipํ˜•ํƒœ์˜ ๋ฐฐํฌ ํŒŒ์ผ์„ ์ƒ์„ฑํ•œ๋‹ค.
./sbt clean dist
[์ฐธ๊ณ ] ๋งŒ์•ฝ javac ๊ฐ€ ์—†์–ด์„œ ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•œ๋‹ค๋ฉด?
๋‹ค์Œ๊ณผ ๊ฐ™์ด ์„ค์น˜๊ฐ€๋Šฅํ•œ jdk๋ฅผ ํ™•์ธํ•œ๋‹ค.
# yum list java*jdk-devel
์„ค์น˜๊ฐ€๋Šฅํ•œ jdk ๋ฆฌ์ŠคํŠธ ์ค‘ jdk 1.8 ๋ฒ„์ „์„ ์„ค์น˜ํ•œ๋‹ค.(์•„๋ž˜ ์˜ˆ์‹œ)
# yum install java-1.8.0-openjdk-devel.x86_64
์„ค์น˜ ํ›„ java compiler ๋ฒ„์ „์„ ํ™•์ธํ•œ๋‹ค.
# javac โ€“version
๋‹ค์‹œ ./sbt clean dist ๋ฅผ ์‹คํ–‰ํ•œ๋‹ค.
์ถœ๋ ฅ๋ฌธ์˜ ๋งˆ์ง€๋ง‰ ๊ฒฝ๋กœ์˜ ํŒŒ์ผ์„ /usr/local๋กœ ๋ณต์‚ฌํ•œ๋‹ค.
๊ฒฝ๋กœํ™•์ธ
Cd /usr/local/ ๊ฒฝ๋กœ๋กœ ์ด๋™ํ•œ ๋’ค ๋ณต์‚ฌํ•œ zipํŒŒ์ผ์˜ ์••์ถ•์„ ํ‘ผ๋‹ค.
์••์ถ• ํ’€๊ณ  ๋‚œ ๋’ค ์ƒ๊ธด ๋””๋ ‰ํ† ๋ฆฌ์—์„œ
vi conf/application.conf
์ฃผํ‚คํผ ์ฃผ์†Œ๋ฅผ ์„ค์ •ํ•ด์ค€๋‹ค.
์„ค์ •์„ ์™„๋ฃŒ ํ•œ ๋’ค ์นดํ”„์นด ๋งค๋‹ˆ์ €๋ฅผ ์‹คํ–‰ํ•œ๋‹ค.
์ด๋•Œ ํ™˜๊ฒฝ์„ค์ •๊ณผ ์›นUI์—์„œ ์‚ฌ์šฉํ•  port์ •๋ณด๋ฅผ ์ง€์ •ํ•˜์—ฌ ์‹คํ–‰ํ•œ๋‹ค.
Master:9000์œผ๋กœ ์ ‘์†ํ•œ๋‹ค.
9000๋ฒˆ ํฌํŠธ
๋ฐฉํ™”๋ฒฝ ํ•ด์ œ ํ•„์ˆ˜
ํด๋Ÿฌ์Šคํ„ฐ ์ •๋ณด๋ฅผ ์ถ”๊ฐ€ํ•˜๋ฉด
์นดํ”„์นด ๋งค๋‹ˆ์ €์—์„œ
๊ธฐ์กด์˜ ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ๋‚˜ํƒ€๋‚œ๋‹ค.
ํด๋Ÿฌ์Šคํ„ฐ์— ๊ธฐ์กด์˜ ์ƒ์„ฑ ๋˜์–ด์žˆ๋˜ topic๊ณผ ์—ฐ๊ฒฐ๋œ broker๋“ฑ
ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋‹ค์–‘ํ•œ ์ •๋ณด๊ฐ€ ํ•œ๋ˆˆ์— ๋ณด๊ธฐ ์‰ฝ๊ณ  ํŽธ๋ฆฌํ•˜๊ฒŒ ๋‚˜ํƒ€๋‚จ.
Topic์ด๋‚˜ broker๋“ฑ ์ƒ์„ธํ•œ ์ปจํŠธ๋กค๋„ ์‰ฝ๊ฒŒ ๊ฐ€๋Šฅํ•ด์ง„๋‹ค.
zookeeper ์—ญํ• 
Kafka ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋ชจ๋“  ๋ถ„์‚ฐ ์ฒ˜๋ฆฌ๋Š” Zookeeper๋ฅผ ๋งค๊ฐœ๋กœ ์ด๋ฃจ์–ด ์ง„๋‹ค. ์ฆ‰, zookeeper์˜ ์—ญํ• ์€ ๋ถ„์‚ฐ ํ™˜๊ฒฝ์—์„œ
์„œ๋ฒ„๋“ค ๊ฐ„์— ์ƒํ˜ธ ์กฐ์ •์ด ํ•„์š”ํ•œ ๋‹ค์–‘ํ•œ ์„œ๋น„์Šค๋ฅผ ์ œ๊ณตํ•˜๋Š” ์‹œ์Šคํ…œ์ž…๋‹ˆ๋‹ค.
1, ํ•˜๋‚˜์˜ ์„œ๋ฒ„์—๋งŒ ์„œ๋น„์Šค๊ฐ€ ์ง‘์ค‘๋˜์ง€ ์•Š๋„๋ก, ์„œ๋น„์Šค๋ฅผ ์•Œ๋งž๊ฒŒ ๋ถ„์‚ฐํ•˜์—ฌ ๋™์‹œ์— ์ฒ˜๋ฆฌํ•˜๊ฒŒ ํ•ด์ค๋‹ˆ๋‹ค.
2, ํ•˜๋‚˜์˜ ์„œ๋ฒ„์—์„œ ์ฒ˜๋ฆฌํ•œ ๊ฒฐ๊ณผ๋ฅผ ๋‹ค๋ฅธ ์„œ๋ฒ„๋“ค๊ณผ ๋™๊ธฐํ™”ํ•˜์—ฌ ๋ฐ์ดํ„ฐ์˜ ์•ˆ์ •์„ฑ์„ ๋ณด์žฅํ•ด์ค๋‹ˆ๋‹ค.
3, ์šด์˜(active) ์„œ๋ฒ„๊ฐ€ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•ด์„œ ์„œ๋น„์Šค๋ฅผ ์ œ๊ณตํ•  ์ˆ˜ ์—†์„ ๊ฒฝ์šฐ,
๋‹ค๋ฅธ ๋Œ€๊ธฐ ์ค‘์ธ ์„œ๋ฒ„๋ฅผ ์šด์˜์„œ๋ฒ„๋กœ ๋ฐ”๊ฟ”์„œ ์„œ๋น„์Šค๊ฐ€ ์ค‘์ง€ ์—†์ด ์ œ๊ณต๋˜๊ฒŒ ํ•ด์ค๋‹ˆ๋‹ค.
4, ๋ถ„์‚ฐ ํ™˜๊ฒฝ์„ ๊ตฌ์„ฑํ•˜๋Š” ์„œ๋ฒ„๋“ค์˜ ํ™˜๊ฒฝ์„ค์ •์„ ํ†ตํ•ฉ์ ์œผ๋กœ ๊ด€๋ฆฌํ•ด์ค๋‹ˆ๋‹ค.
์ฐธ๊ณ . Zookeeper์˜ ์—ญํ• 
Broker
1.์ƒํƒœ๊ด€๋ฆฌ
Kafka Broker๊ฐ€ ์‚ด์•„ ์žˆ๋‹ค๋ฉด ํ•ญ์ƒ ์ •๊ธฐ์ ์œผ๋กœ ํ•˜ํŠธ ๋น„ํŠธ ์š”์ฒญ์„ ๋ณด๋‚ด ZooKeeper๊ฐ€ ์ƒํƒœ๋ฅผ ๊ฒฐ์ •
2.ํ• ๋‹น๋Ÿ‰
๋‹ค๋ฅธ ์ƒ์‚ฐ ๋ฐ ์†Œ๋น„ ํ• ๋‹น๋Ÿ‰์„ ๊ฐ€์ง€๊ธฐ ์œ„ํ•ด Kafka Broker๋Š” ์ผ๋ถ€ consume๋ฅผ ํ—ˆ์šฉ
3.Replicas
๊ฐ ํ† ํ”ฝ์— ๋Œ€ํ•ด Kafka์˜ Zookeeper๋Š” ์ผ๋ จ์˜ ISR๋ฅผ ์œ ์ง€ํ•˜๊ณ  ์ด์ „์— ์„ ํƒ๋œ ๋ฆฌ๋” ๋…ธ๋“œ๊ฐ€ ๋‹ค์šด๋˜๋ฉด
ํ˜„์žฌ ์‚ด์•„์žˆ๋Š” ๋…ธ๋“œ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ZooKeeper๊ฐ€ ์ƒˆ๋กœ์šด ๋ฆฌ๋”๋ฅผ ์„ ์ถœํ•จ.
4.Node์™€ topic ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ
ZooKeeper๋Š” broker์˜ ๋…ธ๋“œ์™€ ํ† ํ”ฝ์„ ๋“ฑ๋กํ•˜๊ณ  ์ €์žฅํ•œ๋‹ค.
์ฐธ๊ณ . Zookeeper์˜ ์—ญํ• 
consumer
1.Offsets
ZooKeeper๋Š” Kafka์˜ 0.9.1 ๋ฆด๋ฆฌ์Šค์—์„œ ์†Œ๋น„์ž ์˜คํ”„์…‹์„ ์œ„ํ•œ ๊ธฐ๋ณธ ์ €์žฅ ์—”์ง„
Consumer๊ฐ€ ์–ผ๋งˆ๋‚˜ ๋งŽ์€ ๋ฉ”์‹œ์ง€๋ฅผ ๊ฐ ์†Œ๋น„์ž๊ฐ€ ์†Œ๋น„ํ•˜๋Š”์ง€์— ๋Œ€ํ•œ ๋ชจ๋“  ์ •๋ณด๋Š” ZooKeeper์— ์ €์žฅ๋ฉ๋‹ˆ๋‹ค.
2. Registry
๊ณต๊ฐœ ๋ถ„์‚ฐํ˜• ๊ตฌ์„ฑ ์„œ๋น„์Šค, ๋™๊ธฐ ์„œ๋น„์Šค ๋ฐ ๋Œ€์šฉ๋Ÿ‰ ๋ถ„์‚ฐ ์‹œ์Šคํ…œ์„ ์œ„ํ•œ ๋„ค์ด๋ฐ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ๋ฅผ ์ œ๊ณตํ•œ๋‹ค.
ํ•˜์ง€๋งŒ ์ผ์‹œ์ ์ธ zNode์ด๊ณ , consumer๊ฐ€ ๋‹ค์šด๋˜๊ณ  ์ƒˆ ํ”„๋กœ์„ธ์Šค๋ฅผ ๋“ฑ๋กํ•˜๋ฉด ์‚ญ์ œ๋œ๋‹ค.
์ฐธ๊ณ . Zookeeper์˜ ์—ญํ• 
JAVA
1.๋ฉ”์ด๋ธํ”„๋กœ์ ํŠธ๋ฅผ ์„ค์น˜ํ•˜๊ธฐ ์œ„ํ•ด ์•„๋ž˜๋กœ ์ ‘์†
http://maven.apache.org/download.cgi
2. apache-mave-3.6.0-bin.zip ํŒŒ์ผ ๋‹ค์šด๋กœ๋“œ
3. ๋‹ค์šด๋ฐ›์€ ํŒŒ์ผ์„ ์›ํ•˜๋Š” ๊ฒฝ๋กœ์— ํ‘ผ๋‹ค.
C:apache-maven-3.6.0
ํ™˜๊ฒฝ๋ณ€์ˆ˜ ๋“ฑ๋ก
1. ์ปดํ“จํ„ฐ โ€“ ์†์„ฑ โ€“ ๊ณ ๊ธ‰์‹œ์Šคํ…œ์„ค์ • โ€“ ๊ณ ๊ธ‰ํƒญ โ€“ํ™˜๊ฒฝ๋ณ€์ˆ˜ ์ฐฝ์œผ๋กœ ์ด๋™
2. ์‹œ์Šคํ…œ ๋ณ€์ˆ˜ โ€“ ์ƒˆ๋กœ ๋งŒ๋“ค๊ธฐ ํด๋ฆญํ•˜์—ฌ ์œ„์˜ ๋ณ€์ˆ˜๋ฅผ ์ž…๋ ฅ
๋ณ€์ˆ˜ ์ด๋ฆ„ : MAVEN_HOME
๋ณ€์ˆ˜ ๊ฐ’ : Capache-maven-3.6.0
3. ์‹œ์Šคํ…œ ๋ณ€์ˆ˜ Path๋ฅผ ์„ ํƒํ•˜์—ฌ ๋ฉ”์ด๋ธ ๊ฒฝ๋กœ๋ฅผ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋“ฑ๋กํ•œ๋‹ค.
๋ณ€์ˆ˜ ๊ฐ’ : %MAVEN_HOME%bin;
CMD ์ฐฝ์—์„œ mvn โ€“version ๋ช…๋ น์–ด๋ฅผ ์น˜๋ฉด ์œ„์™€ ๊ฐ™์ด ๊ฒฐ๊ณผ๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค.
๋ฉ”์ด๋ธ ํ”„๋กœ์ ํŠธ ์ƒ์„ฑ
1.์•„๋ž˜ํŽ˜์ด์ง€๋กœ ์ ‘์†ํ•ด ํ•ด๋‹น ์นดํ”„์นด ๋ฒ„์ „์˜ ๋””ํŽœ๋˜์‹œ๋ฅผ ๋ณต์‚ฌํ•œ๋‹ค.
https://mvnrepository.com/artifact/org.apache.kafka
2. Apache kafka 2.11-2.0.0์˜ ๋””ํŽœ๋˜์‹œ๋ฅผ ๋ณต์‚ฌํ•˜๊ณ 
์ดํด๋ฆฝ์Šค pom.xml์— ์ถ”๊ฐ€ํ•œ๋‹ค.
Pom.xml์— ๋””ํŽœ๋˜์‹œ ์ถ”๊ฐ€
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.6.4</version>
</dependency>
Producer code
consumer code
producer์—ญํ• ์„ java์—์„œ ๊ตฌํ˜„
Consumer์—ญํ• ์„ java์—์„œ ๊ตฌํ˜„
Real Time
Application(Twitter)
Kafka-python ํ™œ์šฉ
Prerequisites
๏ต Python
๏ต Kafka
๏ต Zookeeper
๏ต Twitter API credentials
Procedure
๏ต apps.twitter.com ์ ‘์† ํ›„ ๊ฐ€์ž… ๋ฐ APP ์ƒ์„ฑ
๏ต Twitter API credential code ์ž๋™ ๋ฐœ๊ธ‰
Linux(CentOS)์—์„œ Anaconda ์„ค์น˜ํ•˜๊ธฐ
Anaconda : ๋ฐ์ดํ„ฐ๋ถ„์„์— ํ•„์š”ํ•œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ๋ชจ์•„๋†“์€ ํŒŒ์ด์ฌ ๊ธฐ๋ฐ˜์˜ ํ”Œ๋žซํผ
1. Anaconda ์„ค์น˜ํŒŒ์ผ(.sh) ๋‹ค์šด๋กœ๋“œ
2. Anaconda ์„ค์น˜ํ•˜๊ธฐ
3. License ์Šน์ธํ•˜๊ธฐ
4. ์„ค์น˜ํ•  ์ฃผ์†Œ ์„ค์ •ํ•˜๊ธฐ
5. PATH ์ง€์ •ํ•˜๊ธฐ
6. ์„ค์น˜์™„๋ฃŒ
1. Anaconda ์„ค์น˜ํŒŒ์ผ(.sh) ๋‹ค์šด๋กœ๋“œ
wget https:// repo.continuum.io/archive/Anaconda3-5.2.0-Linux-x86_64.sh
2. Anaconda ์„ค์น˜ํ•˜๊ธฐ
bash Anaconda3-5.2.0-Linux-x86_64.sh
์œ„์˜ ๋ช…๋ น์–ด๋ฅผ ์ž…๋ ฅํ•˜๋ฉด ์•„๋ž˜์™€ ๊ฐ™์ด Anaconda License๊ฐ€ ๋‚˜ํƒ€๋‚œ๋‹ค. Enter ํ‚ค๋ฅผ ๋ˆŒ๋Ÿฌ์„œ ์ญ‰์ญ‰ ๋„˜๊ธธ ์ˆ˜ ์žˆ๋Š”๋ฐ,
๋ฌด์ž‘์ • ๋ˆŒ๋Ÿฌ๋Œ€๋ฉด ์„ค์น˜๊ฐ€ ์ทจ์†Œ๋˜๋‹ˆ ํ•œ ์ค„์”ฉ ์กฐ์‹ฌํžˆ ๋„˜๊ธฐ๋„๋ก ํ•œ๋‹ค.
3. ๋ผ์ด์„ ์Šค ์Šน์ธํ•˜๊ธฐ
yes๋ฅผ ์ž…๋ ฅํ•˜๊ณ  Enter ํ‚ค๋ฅผ ๋ˆ„๋ฅธ๋‹ค.
4. ์„ค์น˜ํ•  ์ฃผ์†Œ ์„ค์ •ํ•˜๊ธฐ
๊ธฐ๋ณธ๊ฐ’ ์ฃผ์†Œ(/root/anaconda3)์— ์„ค์น˜๋ฅผ ํ•  ๊ฒƒ์ด๋ผ๋ฉด Enter ํ‚ค๋ฅผ ๋ˆ„๋ฅด๊ณ , ๋‹ค๋ฅธ ๊ณณ์— ์ €์žฅ์„ ํ•˜๊ณ ์‹ถ๋‹ค๋ฉด ํ•ด๋‹น
์ฃผ์†Œ๋ฅผ ์ž…๋ ฅํ•˜๊ณ  Enter ํ‚ค๋ฅผ ๋ˆ„๋ฅธ๋‹ค.( ๋˜๋„๋ก์ด๋ฉด root ํด๋”์—๋Š” ์„ค์น˜ํ•˜์ง€ ๋ง ๊ฒƒ )
yes๋ฅผ ์ž…๋ ฅํ•˜๊ณ  ๋‚˜๋ฉด, ์–ด๋””์— ์„ค์น˜ํ•  ๊ฒƒ์ธ์ง€ ๋ฌป๋Š” ๋ฉ”์‹œ์ง€๊ฐ€ ๋‚˜ํƒ€๋‚œ๋‹ค.
5. PATH ์ง€์ •ํ•˜๊ธฐ
์„ค์น˜๊ฐ€ ๋งˆ๋ฌด๋ฆฌ๋˜๋ฉด ์‚ฌ์šฉ์ž์—๊ฒŒ PATH๋ฅผ ์ถ”๊ฐ€ํ•  ๊ฒƒ์ธ์ง€ yes ๋˜๋Š” no๋ฅผ ๋ฌป๋Š” ์ฐฝ์ด ๋‚˜ํƒ€๋‚œ๋‹ค. ํŠน๋ณ„ํ•œ ๊ฒฝ์šฐ๋ฅผ ์ œ
์™ธํ•˜๊ณ ๋Š” yes๋ฅผ ์ž…๋ ฅํ•ด์ฃผ๋ฉด ๋œ๋‹ค.
PATH ์„ค์ •
/etc์— ์žˆ๋Š” profile์— PATH ์ถ”๊ฐ€
vi profile
export PATH=/usr/anaconda3/bin:$PATH
ํŽธ์ง‘ ์™„๋ฃŒ ํ›„ source /etc/profile ์‹คํ–‰ ํ›„ ์„ค์ •์„ ์™„๋ฃŒํ•œ๋‹ค.
์ดํ›„ conda install โ€“update --all
pip
Python ์„ค์น˜ ํ›„ terminal ์ฐฝ์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด ์ž…๋ ฅ
๏ต pip install kafka-python
๏ต pip install python-twitter
๏ต pip install tweepy
*pip ์„ค์น˜๊ฐ€ ์•ˆ๋  ์‹œ: yum --enablerepo=extras install epel-release ๋ช…๋ น์–ด๋กœ EPEL repo๋ฅผ ๋จผ์ € enable ์‹œ
ํ‚จ๋‹ค
Kafka ์‹คํ–‰
๏ต cd /home/zookeeper/bin ์ด๋™
๏ต zkServer.sh start (master->slave1->slave2 ์ˆœ์œผ๋กœ ์‹คํ–‰)
๏ต cd /home/kafka/bin ์ด๋™
๏ต ./kafka-server-start.sh /home/kafka/config/server.properties
Kafka topic ์ƒ์„ฑ
๏ต cd /home/kafka/bin
๏ต ./kafka-topics.sh --create --zookeeper
master:2181,slave1:2181,slave2:2181/kopo-kafka --replication-factor 1 --
partitions 1 --topic trump
*trump ๋ž€ ํ† ํ”ฝ์„ ์ƒ์„ฑํ•ด Trump๋กœ ํŠธ์œ—๋˜๋Š” ๋ฉ”์‹œ์ง€๋ฅผ produceํ•  ์˜ˆ์ •
Python์œผ๋กœ kafka-producer ์ƒ์„ฑ
1.ํ•„์š” libraries ํ˜ธ์ถœ
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
from kafka import SimpleProducer, KafkaClient
Twitter API credential
2.๋ฐœ๊ธ‰๋ฐ›์€ ์ธ์ฆํ‚ค ์ž…๋ ฅ
access_token = ""
access_token_secret = ""
consumer_key = ""
consumer_secret = โ€œโ€
Python์œผ๋กœ kafka-producer ์ƒ์„ฑ
3. Python class ์ƒ์„ฑ
class StdOutListener(StreamListener):
def on_data(self, data):
producer.send_messages("trump", data.encode('utf-8'))
print (data)
return True
def on_error(self, status):
print (status)
Python์œผ๋กœ kafka-producer ์ƒ์„ฑ
4. ์„ค์ •๊ฐ’ ์ž‘์„ฑ
kafka = KafkaClient("localhost:9092")
producer = SimpleProducer(kafka)
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
stream.filter(track="trump")
Output
์ฝ”๋“œ ์‹คํ–‰ ์‹œ
JSON ํ˜•ํƒœ์˜ ๋ฉ”์‹œ์ง€๊ฐ€ ์ถœ๋ ฅ๋˜์–ด
Kafka ์— publish ํ•˜๊ฒŒ๋จ
Kafka consumer ์ฝ˜์†”์—์„œ๋„ ๋™์ผํ•œ ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธ ๊ฐ€๋Šฅ

Kafka slideshare

  • 1.
  • 2.
    Kafka? ๏ต ๏ต image source :https://mesosphere.com/blog/kafka-dcos-tutorial/
  • 3.
  • 4.
    Kafka Major Concepts ๋Œ€์šฉ๋Ÿ‰์‹ค์‹œ๊ฐ„ ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ + ๋ถ„์‚ฐ๋œ ๋ฐ์ดํ„ฐ ํŒŒ์ดํ”„๋ผ์ธ ํ‘œ์ค€ํ™” ๋†’์€ ์ฒ˜๋ฆฌ๋Ÿ‰, ๋น ๋ฅธ ๋ฉ”์‹œ์ง€ ์ „์†ก, ์šด์˜ ํšจ์œจํ™”๋ฅผ ์œ„ํ•ด ๋ถ„์‚ฐ์‹œ์Šคํ…œ, ํŽ˜์ด์ง€์บ์‰ฌ, ๋ฐฐ์น˜์ „์†ก์ฒ˜๋ฆฌ
  • 5.
    Kafka Major Concepts imagesource : https://www.tutorialspoint.com/apache_kafka/apache_kafka_cluster_architecture.htm โ€ข broker : ์นดํ”„์นด ์„œ๋ฒ„๋ฅผ ๊ฐ€๋ฆฌํ‚ด โ€ข zookeeper : ์นดํ”„์นด ์„œ๋ฒ„ (+ํด๋Ÿฌ์Šคํ„ฐ) ์ƒํƒœ๋ฅผ ๊ด€๋ฆฌํ•˜๊ณ  โ€ข cluster : ๋ธŒ๋กœ์ปค๋“ค์˜ ๋ฌถ์Œ
  • 6.
    Kafka Major Concepts imagesource : https://kafka.apache.org/documentation/ โ€ข topic : ๋ฉ”์‹œ์ง€ ์ข…๋ฅ˜ โ€ข partitions : topic ์ด ๋‚˜๋ˆ ์ง€๋Š” ๋‹จ์œ„ โ€ข Log : 1๊ฐœ์˜ ๋ฉ”์„ธ์ง€ โ€ข offset : ํŒŒํ‹ฐ์…˜ ๋‚ด์—์„œ ๊ฐ ๋ฉ”์‹œ์ง€๊ฐ€ ๊ฐ€์ง€๋Š” unique id
  • 7.
    Kafka working mechanism imagesource : https://fr.wikipedia.org/wiki/Fichier:Overview_of_Apache_Kafka.svg โ€ข ์ •ํ•ด์ง„ topic์— producer๊ฐ€ ๋ฉ”์‹œ์ง€๋ฅผ ๋ฐœํ–‰ํ•ด ๋†“์œผ๋ฉด consumer๊ฐ€ ํ•„์š”ํ•  ๋•Œ ํ•ด๋‹น ๋ฉ”์‹œ์ง€๋ฅผ ๊ฐ€์ ธ๊ฐ„๋‹ค.
  • 8.
    Kafka working mechanism imagesource : https://www.tutorialspoint.com/apache_kafka/apache_kafka_cluster_architecture.htm zookeeper ๊ฐ€ kafka ์˜ ๋ถ„์‚ฐ ๋ฉ”์‹œ์ง€ ํ์˜ ์ •๋ณด๋ฅผ ๊ด€๋ฆฌํ•ด์ค€๋‹ค.
  • 9.
    Kafka working mechanism ์—ฌ๋Ÿฌ๊ฐœ์˜ ์นดํ”„์นด ์„œ๋ฒ„๋กœ ๊ตฌ์„ฑํ•ด ๋™์ผํ•œ ์—ญํ• ์„ ํ•˜๋Š” ์„œ๋ฒ„๋ฅผ ์ถ”๊ฐ€ํ•ด ๋ถ€ํ•˜๋ฅผ ๋ถ„์‚ฐํ•  ์ˆ˜ ์žˆ๋Š” ๊ฒƒ ํ•˜์ง€๋งŒ ์„œ๋ฒ„๋งŒ ์ถ”๊ฐ€ํ•˜๋‹ค ๋ณด๋ฉด ๋ถˆํ•„์š”ํ•œ ๋น„์šฉ์ด ์ฆ๊ฐ€ LinkedIn์—์„œ ๊ฐ€์žฅ ์‚ฌ์šฉ๋Ÿ‰ ๋†’์€ ํด๋Ÿฌ์Šคํ„ฐ๋Š” 60๋Œ€์˜ ๋ธŒ๋กœ์ปค๋ฅผ ์‚ฌ์šฉํ•จ image source : http://blog.mmlac.com/log-transport-with-apache-kafka/
  • 10.
    ํ•œ ๋ฒˆ ์ฝ์€ํŒŒ์ผ์˜ ๋‚ด์šฉ์„ ์ด ํŽ˜์ด์ง€ ์บ์‹œ ์˜์—ญ์— ์ €์žฅํ•˜๊ณ , ๊ฐ™์€ ํŒŒ์ผ์˜ ์ ‘๊ทผ์ด ์ผ์–ด๋‚˜๋ฉด ๋””์Šคํฌ์—์„œ ์ฝ์–ด์˜ค๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ ํŽ˜์ด์ง€ ์บ์‹œ์—์„œ ์ฝ๋Š”๋‹ค. ์นดํ”„์นด๋Š” ํŽ˜์ด์ง€ ์บ์‰ฌ๋ฅผ ์ด์šฉํ•ด ๋น ๋ฅธ ์•ก์„ธ์Šค๊ฐ€ ๊ฐ€๋Šฅํ•จ. ์ €๋ ดํ•œ SATA๋””์Šคํฌ๋ฅผ ์‚ฌ์šฉํ•ด๋„ ๋ฌด๋ฐฉ Kafka working mechanism source : https://www.slideshare.net/DmitryTolpeko/apache-kafka-messagingsystem/14
  • 11.
    Kafka working mechanism source: https://www.slideshare.net/DmitryTolpeko/apache-kafka-messagingsystem/14
  • 12.
    source : https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600 ๋ฐ์ดํ„ฐ๋ฅผ์ฃผ๊ณ  ๋ฐ›๋Š” ๊ณผ์ •์—์„œ ์ž‘์€ I/O ๋นˆ๋ฒˆํ•˜๊ฒŒ ์ผ์–ด๋‚˜๋ฉด ์†๋„ ์ €ํ•˜ ๋จ ์นดํ”„์นด๋Š” ์ž‘์€ I/O๋ฅผ ๋ฌถ์–ด์„œ ์ฒ˜๋ฆฌํ•˜์—ฌ ๋„คํŠธ์›Œํฌ ์˜ค๋ฒ„ํ—ค๋“œ๋ฅผ ์ค„์ด๊ณ  ๋ฐฐ์น˜ ์ „์†ก์ฒ˜๋ฆฌ ์†๋„ ํ–ฅ์ƒ Kafka working mechanism
  • 13.
    Kafka Use cases ๋ฐ์ดํ„ฐ๋ถ„์„ ๊ฒฐ๊ณผ๋กœ ์„œ๋น„์Šค ํŽ˜์ด์ง€์— ์žˆ๋Š” ์ปจํ…์ธ ๊ฐ€ ์‚ฌ์šฉ์ž์—๊ฒŒ ์–ผ๋งˆ๋‚˜ ์˜ํ–ฅ์„ ์ฃผ๋Š”์ง€ ์‹ค์‹œ๊ฐ„์œผ๋กœ ๊ฐ์ง€ ์•ž์˜ ์ˆซ์ž๋Š” ๊ธฐ์‚ฌ์˜ ์ˆœ์œ„ ๋’ค์˜ ์ˆซ์ž๋Š” ๊ธฐ์‚ฌ์˜ ์†Œ๋น„์ง€์ˆ˜ source : โ€œkafka exerciseโ€ presented by Dongkeun Lee, Kangyong kim, student at dept. of smart finance, Korea Polytechnics in 2018
  • 14.
    Kafka Use cases ์นด์นด์˜คํ†ก์•ฑ์—์„œ 10๋ถ„ ๊ฐ„ ์†Œ๋น„ํ•œ ๋„ค์ด๋ฒ„ ์—ฐ์•  ๊ธฐ์‚ฌ ์†Œ๋น„ ์ง€์ˆ˜ ๋ถ„์„ source : โ€œkafka exerciseโ€ presented by Dongkeun Lee, Kangyong kim, student at dept. of smart finance, Korea Polytechnics in 2018
  • 15.
    Kafka Installation VM ์„ค์น˜ Java8.0, ZooKeeper, Kafka ์„ค์น˜
  • 16.
    master slave1 slave2 ZooKeeperensemble master slave1 slave2 Kafka Cluster quorum ์‹ค์Šตํ™˜๊ฒฝ ๊ฐœ์š”
  • 17.
    ๊ฐ€์ƒ ๋จธ์‹ (VM) โ€ข VirtualBox์ด์šฉ โ€ข ์ค€๋น„ ๋ฐ ์ ๊ฒ€ ์‚ฌํ•ญ - VirtualBox Latest Ver. - CentOS 7 (DVD Ver.) - JDK 1.8 - ๋ชจ๋“  ์„ค์น˜ ๋ฐ ํ™˜๊ฒฝ์„ค์ •์€ root๊ถŒํ•œ์œผ๋กœ ์„ค์น˜ํ•จ source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 18.
    ๊ฐ€์ƒ ๋จธ์‹ (VM) ์„ค์น˜๋ฐ ์„ค์ • 1. ๊ฐ€์ƒ๋จธ์‹  ์ด๋ฆ„ ์„ค์ • 2~6. ๊ฐ€์ƒ๋จธ์‹  ๋ฉ”๋ชจ๋ฆฌ / ํ•˜๋“œ ์„ค์ • 2 4 3 5 6 1 5 6 source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 19.
    ๊ฐ€์ƒ ๋จธ์‹ (VM) ์„ค์น˜๋ฐ ์„ค์ • 1. ์ƒ์„ฑ๋œ master ๋จธ์‹ ์˜ ๋„คํŠธ์›Œํฌ ์„ค์ •์œผ๋กœ ๋“ค์–ด๊ฐ„๋‹ค 2. ์–ดํƒญํ„ฐ 2์—๋Š” ์–ด๋Œ‘ํ„ฐ์˜ ๋ธŒ๋ฆฌ์ง€๋ฅผ ์„ ํƒ 3. ์–ดํƒญํ„ฐ 3์—๋Š” ํ˜ธ์ŠคํŠธ ์ „์šฉ ์–ด๋Œ‘ํ„ฐ ์„ ํƒ 1 2 source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 20.
    Cent OS 7์„ค์น˜ 1. GNOME ํ…Œ์Šคํฌํƒ‘ ์„ ํƒ 2. ๋กœ์ปฌ ํ‘œ์ค€ ๋””์Šคํฌ ์„ ํƒ 1 2 source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 21.
    Cent OS 7์„ค์น˜ 1. Root ๋น„๋ฐ€๋ฒˆํ˜ธ ์„ค์ • 2. ์‚ฌ์šฉ์ž ID, PW์„ค์ • 1 2 source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 22.
    ๋„คํŠธ์›Œํฌ ์„ค์ • ๏ต Network๊ณ ์ • IP ์„ค์ •(Static) * vi /etc/sysconfig/network-scripts/ifcfg-enp0s8 BOOTPROTO=static ์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€ ONBOOT=yes NETMASK=255.255.255.0 IPADDR=192.168.0.30 GATEWAY=10.0.0.2 ๋„คํŠธ์›Œํฌ ๋Œ€์—ญ๋Œ€๋Š” ๋ณ€๊ฒฝ ๊ฐ€๋Šฅํ•จ ์„ค์ • ์™„๋ฃŒ ํ›„ systemctl restart network source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 23.
    ๋„คํŠธ์›Œํฌ ์„ค์ • ๏ต Host์„ค์ • vi /etc/hosts master 192.168.0.30 master.kopo slave1 192.168.0.31 slave1.kopo slave2 192.168.0.32 slave2.kopo ์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€ source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics vi /etc/hosts
  • 24.
    ๋„คํŠธ์›Œํฌ ์„ค์ • ๏ต Hostname ์„ค์ • vi /etc/sysconfig/network NETWORKING=yes HOSTNAME=master.kopo ์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€ --๋‚˜๋จธ์ง€ VM๋„ ๋ณ€๊ฒฝ-- ์˜ˆ์‹œ) NETWORKING=yes HOSTNAME=slave1.kopo [root@localhost ~]# hostnamectl set-hostname master.kopo source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 25.
    ๋„คํŠธ์›Œํฌ ์„ค์ • ๏ต ๋ฐฉํ™”๋ฒฝํ•ด์ œ - ๋ฐฉํ™”๋ฒฝ ๋„๊ธฐ systemctl stop firewalld - ํ”„๋กœ๊ทธ๋žจ ์ž๋™ ์‹œ์ž‘ ํ•ด์ œ systemctl disable firewalld -๋ฐฉํ™”๋ฒฝ ํ•ด์ฒด ํ›„ ping test ping โ€“c 3 8.8.8.8 - CentOS์˜ ๊ฒฝ์šฐ OS๋ฅผ ์ฒ˜์Œ ์„ค์น˜ํ–ˆ์„ ๋•Œ ๊ธฐ๋ณธ ํฌํŠธ๋ฅผ ์ œ์™ธํ•œ ๋ชจ๋“  ํฌํŠธ๋ฅผ ๋ฐฉํ™”๋ฒฝ์—์„œ ๋ง‰๊ณ  ์žˆ์–ด ์‚ฌ์šฉํ•  ํฌํŠธ๋ฅผ ๊ฐœ๋ฐฉํ•ด ์ฃผ์–ด์•ผํ•จ. - ํฌํŠธ ์ถ”๊ฐ€ firewall-cmd --permanent --zone=public --add-port=ํฌํŠธ๋ฒˆํ˜ธ/tcp - ๋ฐฉํ™”๋ฒฝ ์žฌ์‹œ์ž‘ firewall-cmd --reload - ์™„์ „ํžˆ ๋ฐฉํ™”๋ฒฝ์„ ํ•ด์ฒดํ•˜๊ณ  ์‹ถ์€ ๊ฒฝ์šฐ - ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ๋ชจ๋“  ์„œ๋น„์Šค/ํฌํŠธ ๋ชฉ๋ก์„ ์กฐํšŒ firewall-cmd --list-all - ํ—ˆ์šฉํ•œ ํฌํŠธ ๋ชฉ๋ก firewall-cmd --list-ports - ๋ฐฉํ™”๋ฒฝ ์ƒํƒœ ํ™•์ธ firewall-cmd โ€“state # ์ฃผํ‚คํผ ํด๋ผ์ด์–ธํŠธ ์ ‘์†์šฉ ํฌํŠธ firewall-cmd --permanent --zone=public --add-port=2181/tcp firewall-cmd --reload firewall-cmd --list-ports nc -z master 2181 #์ฃผํ‚คํผ ์•™์ƒ๋ธ” ํฌํŠธ firewall-cmd --permanent --zone=public --add-port=2888/tcp firewall-cmd --permanent --zone=public --add-port=3888/tcp firewall-cmd --reload firewall-cmd --list-ports #์นดํ”„์นด ์ ‘์† ํฌํŠธ firewall-cmd --permanent --zone=public --add-port=9092/tcp firewall-cmd --reload firewall-cmd --list-ports yum install nc-1.84-24.el6.x86_64.rpm source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 26.
    Java ์„ค์น˜ ๏ต Java์„ค์น˜ 1.๋‹ค์šด๋กœ๋“œ Java jdk 1.8์„ ๋ฆฌ๋ˆ…์Šค ์„œ๋ฒ„์— ๋‹ค์šด๋กœ๋“œ ํ›„ ํŠน์ • ํด๋”์— ์ €์žฅ 2. ์••์ถ•ํ•ด์ฒด tar xzf jdk-8u181-linux-x64.tar.gz ๋ช…๋ น์–ด๋กœ ์••์ถ• ํ•ด์ฒด 3. ํด๋” ์ด๋™ mv jdk1.8.0_181 /usr/local ๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด ํด๋” ์ด๋™ 4. ๋งํฌ ์„ค์ • -local ํด๋”๋กœ ์ด๋™ cd /usr/local -๋งํฌ ์„ค์ • ln โ€“s jdk1.8.0_181 java ๋˜๋Š” yum โ€“y install java-1.8.0-openjdk source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 27.
    ์ฃผํ‚คํผ ์„ค์ • cd /usr/local wgethttp://apache.mirror.cdnetworks.com/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz tar zxf zookeeper-3.4.13.tar.gz ln -s zookeeper-3.4.13 zookeeper mkdir -p /data cd /data echo 1 > /data/myid zookeeper node
  • 28.
    [root@master conf]# ll total16 -rw-r--r--. 1 501 games 535 Jun 29 2018 configuration.xsl -rw-r--r--. 1 501 games 2161 Jun 29 2018 log4j.properties -rw-r--r--. 1 root root 167 Nov 8 23:38 zoo.cfg -rw-r--r--. 1 501 games 922 Jun 29 2018 zoo_sample.cfg [root@master conf]# pwd /usr/local/zookeeper/conf [root@master conf]# more zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data clientPort=2181 server.1=master:2888:3888 server.2=slave1:2888:3888 server.3=slave2:2888:3888 maxClientCnxns=200 ์ฃผํ‚คํผ ์„ค์ • vi zoo.cfg๋กœ ์‹คํ–‰ํ•˜์—ฌ ์ขŒ์ธก๊ณผ ๊ฐ™์ด ์ˆ˜์ • ๋ฐ ์ถ”๊ฐ€ํ•œ๋‹ค. tickTime : ์ฃผํ‚คํผ๊ฐ€ ์‚ฌ์šฉํ•˜๋Š” ์‹œ๊ฐ„์— ๋Œ€ํ•œ ๊ธฐ๋ณธ ์ธก์ • ๋‹จ์œ„(๋ฐ€๋ฆฌ์ดˆ) initLimit : Follower๊ฐ€ Leader์™€ ์ดˆ๊ธฐ์— ์—ฐ๊ฒฐํ•˜๋Š” ์‹œ๊ฐ„์— ๋Œ€ํ•œ TimeOut tick ์ˆ˜ syncLimit : Follower๊ฐ€ Leader์™€ ๋™๊ธฐํ™” ํ•˜๋Š” ์‹œ๊ฐ„์— ๋Œ€ํ•œ TimeOut tick ์ˆ˜ dataDir : ์ฃผํ‚คํผ์˜ ํŠธ๋žœ์žญ์…˜ ๋กœ๊ทธ์™€ ์Šค๋ƒ…์ƒท์ด ์ €์žฅ๋˜๋Š” ๊ฒฝ๋กœ clientPort : ์ฃผํ‚คํผ ์‚ฌ์šฉ TCP ํฌํŠธ server.* : ์ฃผํ‚คํผ ์•™์ƒ๋ธ” ๊ตฌ์„ฑ์„ ์œ„ํ•œ ์„œ๋ฒ„ ์„ค์ •, server.myid ํ˜•์‹์œผ๋กœ ์‚ฌ์šฉ
  • 29.
    Java ์„ค์น˜ ๏ต JAVAPath ์„ค์ • (๏ƒ  Optional for Kafka) vi /etc/profile Shift + g ์ œ์ผ ๋ฐ‘์œผ๋กœ ์ด๋™ - PATH ์ถ”๊ฐ€ - export JAVA_HOME=/usr/local/java export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=โ€œ.โ€ Esc ํ›„ shift + : wq ์„ค์ • ์ €์žฅ - ํ™˜๊ฒฝ๋ณ€์ˆ˜ ๋ฐ˜์˜ source /etc/profile ๏ต java -versionjava ์„ค์น˜ ํ™•์ธ source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 30.
    VM ๋ณต์ œ ๋ฐ์„ค์ • - 1๊ฐœ์˜ VM(Master)์— ๋ชจ๋“  ์ž‘์—…์„ ํ•œ ํ›„ VM ๋ณต์ œ๋ฅผ ํ†ตํ•ด datanode๋ฅผ ๊ตฌ์ถ•ํ•œ๋‹ค. slave1 slave1, slave2 2๊ฐœ์˜ VM๋ณต์ œ๊ฐ€ ํ•„์š”ํ•จ source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 31.
    VM ๋ณต์ œ ๋ฐ์„ค์ • Slave1 Salve2 ๋ชจ๋‘ ์ž‘์—… - ๋ณต์ œ๋œ VM์€ ๋„คํŠธ์›Œํฌ ์ดˆ๊ธฐํ™”๋ฅผ ์•ˆ ํ–ˆ๊ธฐ ๋•Œ๋ฌธ์— ์›๋ณธ VM๊ณผ ๋™์ผํ•œ MAC ์ฃผ์†Œ๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ๋‹ค. - ๋์ž๋ฆฌ๋ฅผ ๋ณ€๊ฒฝํ•ด์„œ MAC ์ฃผ์†Œ๋ฅผ ๋‹ค๋ฅด๊ฒŒ ํ•ด ์ค€๋‹ค. source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 32.
    VM ๋ณต์ œ ๋ฐ์„ค์ • Slave1 Salve2 ๋ชจ๋‘ ์ž‘์—… - ๋ณต์ œ๋œ VM network ๋ณ€๊ฒฝ vi/etc/sysconfig/network-scripts/ifcfg-enp0s8 - IPADDR=192.168.0.31 ~ 32 ๋ณ€๊ฒฝ - systemctl restart network ์‹คํ–‰ source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 33.
    VM ๋ณต์ œ ๋ฐ์„ค์ • Slave1 Salve2 ๋ชจ๋‘ ์ž‘์—… - network ๋ณ€๊ฒฝ vi/etc/sysconfig/network - NETWORKING=yes HOSTNAME=slave1~2.kopo ์œผ๋กœ ๋ณ€๊ฒฝ [root@localhost ~]# hostnamectl set-hostname master.kopo vi /etc/sysconfig/network source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 34.
    ๋งˆ์Šคํ„ฐ ๋ฐ ์Šฌ๋ ˆ์ด๋ธŒSSH ์„ค์ • - SSH๋ž€ (Secure Shell)๋Š” ์›๊ฒฉ ๋จธ์‹ ์— ๋กœ๊ทธ์ธํ•˜์—ฌ ๋ช…๋ น๋“ค์„ ์‹คํ–‰ํ•˜๋Š” ํ”„๋กœ๊ทธ๋žจ. - openssh๊ฐ€ ๊ธฐ๋ณธ ์„ค์น˜๋˜์–ด ์žˆ์–ด openssh clients๋งŒ ์ถ”๊ฐ€๋กœ ์•„๋ž˜์™€ ๊ฐ™์ด ์„ค์น˜ํ•œ๋‹ค. ๋งŒ์•ฝ ์‹คํ–‰ ์‹œ package openssh-clinets.. already installed and latest version ์ด๋ผ๋Š” ๋ฌธ๊ตฌ๊ฐ€ ์ถœ๋ ฅ๋˜๋ฉด ํ˜„์žฌ๋‹จ๊ณ„๋Š” ๋„˜์–ด๊ฐ„๋‹ค. yum install openssh-clients source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 35.
    ๋งˆ์Šคํ„ฐ ๋ฐ ์Šฌ๋ ˆ์ด๋ธŒSSH ์„ค์ • - key-gen ์ƒ์„ฑ Master์—์„œ๋งŒ ssh-keygen ๋ช…๋ น์–ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ RSA ํ‚ค ํŽ˜์–ด๋ฅผ ์ƒ์„ฑํ•œ๋‹ค. cd ssh-keygen ์ž…๋ ฅ ํ›„ ์—”ํ„ฐ 3๋ฒˆ ssh-copy-id โ€“i ~/.ssh/id_rsa.pub root@slave1.kopo ssh-copy-id โ€“i ~/.ssh/id_rsa.pub root@slave1.kopo ์ž…๋ ฅ ํ›„ Are you sure you want continue connection ? yes ์ž…๋ ฅ ํ›„ ๊ฐ slave ์„œ๋ฒ„ root ๋น„๋ฐ€๋ฒˆํ˜ธ ์ž…๋ ฅ source : โ€œHadoop ecosystemโ€ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
  • 36.
    [root@slave1 data]# ll total4 -rw-r--r--. 1 root root 2 Oct 30 03:08 myid drwxr-xr-x. 2 root root 131 Nov 27 02:57 version-2 [root@slave1 data]# pwd /data [root@slave1 data]# more myid 2 [root@slave1 data]# ๋ณต์ œ ๋…ธ๋“œ(slave1, slave2)์ฃผํ‚คํผ ์„ค์ • vi myid๋กœ id๋ฅผ ์ˆ˜์ •ํ•œ๋‹ค. slave1 : 2 slave2 : 3
  • 37.
    ์ฃผํ‚คํผ ์‹œ์ž‘๊ณผ ์ข…๋ฃŒ,์ƒํƒœํ™•์ธ [root@slave2 ~]# sh /usr/local/zookeeper/bin/zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@slave2 ~]# sh /usr/local/zookeeper/bin/zkServer.sh stop ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED [root@slave2 ~]# [root@master conf]# sh /usr/local/zookeeper/bin/zkCli.sh
  • 38.
  • 39.
    wget http://apache.mirror.cdnetworks.com/kafka/1.0.0/kafka_2.11-1.0.0.tgz tar zxfkafka_2.11-1.0.0.tgz ln -s kafka_2.11-1.0.0 kafka scp kafka_2.11-1.0.0.tgz root@slave1:/usr/local scp kafka_2.11-1.0.0.tgz root@slave2:/usr/local vi /usr/local/kafka/config/server.properties ์นดํ”„์นด ์„ค์น˜ # The id of the broker. This must be set to a unique integer for each broker. broker.id=1 # A comma seperated list of directories under which to store log files log.dirs=/kdata1,/kdata2 ############################# Zookeeper ############################# # Zookeeper connection string (see zookeeper docs for details). # This is a comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append an optional chroot string to the urls to specify the # root directory for all kafka znodes. zookeeper.connect=master:2181,slave1:2181,slave2:2181/kopo-kafka slave1 : 2 slave2 : 3 mkdir /kdata1 mkdir /kdata2 ์‹ค ์šด์˜์‹œ ๋ณ„๋„ ํŒŒํ‹ฐ์…˜์œผ๋กœ ์„ค์ • for more information: https://kafka.apacheaorg/documentation/#brokerconfigs * ์ฃผํ‚คํผ๋…ธ๋“œ๋ช… ๋ฏธ์ž‘์„ฑๆ™‚ ์ฃผํ‚คํผ ๋ฃจํŠธ๋…ธ๋“œ์— ์ €์žฅ ์ด ์˜ˆ์ œ์—์„  kopo-kafka๋กœ ์„ค์ •ํ•˜์˜€์Œ
  • 40.
    ์นดํ”„์นด ์‹คํ–‰ /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties /usr/local/kafka/bin/kafka-server-stop.sh #...์ค‘๋žตโ€ฆ(๋‹ค๋Ÿ‰์˜๋ฉ”์‹œ์ง€ ํŒ์—… ํ›„) #[kafkaServer id=1] started (kafka.server.KafkaServer) ๊ฐ€ ๋œจ๋ฉด ์„ฑ๊ณต
  • 41.
    #์ฃผํ‚คํผ ํฌํŠธ ํ™•์ธ โ€ขnetstat โ€“ntlp | grep 2181 #์นดํ”„์นด ํฌํŠธ ํ™•์ธ โ€ข netstat โ€“ntlp | grep 9092 ์นดํ”„์นด ์‹คํ–‰
  • 42.
    #์ฃผํ‚คํผ ์ง€๋…ธ๋“œ๋ฅผ ์ด์šฉํ•˜์—ฌ์นดํ”„์นด ์ •๋ณด ํ™•์ธ โ€ข cd /usr/local/zookeeper/bin โ€ข ./zkCli.sh #์ฃผํ‚คํผ ์ถ”๊ฐ€ ๋…ธ๋“œ ํ™•์ธ โ€ข ls / #ํด๋Ÿฌ์Šคํ„ฐ ๋…ธ๋“œ ์—ฐ๊ฒฐ ํ™•์ธ โ€ข ls /kopo-kafka/brokers/ids *ํ™•์ธ ํ›„ quit ์ž…๋ ฅ ์นดํ”„์นด ์‹คํ–‰ ์ฐธ๊ณ > ์นดํ”„์นด ์„œ๋ฒ„ ๋กœ๊ทธ ํ™•์ธ cat /usr/local/kafka/logs/server.log
  • 43.
    ์นดํ”„์นด ํ…Œ์ŠคํŠธ ์นดํ”„์นด ํ† ํ”ฝ์ƒ์„ฑ โ€ข Kafka-topics.sh ์ด์šฉํ•˜์—ฌ kopo-topic ์ƒ์„ฑ โ€ข cd /home/kafka/bin โ€ข ./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --replication-factor 1 --partitions 1 --topic kopo-topic --create [root@master ~]# /usr/local/kafka/bin/kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --replication-factor 1 --partitions 1 --topic ds-topic โ€“create Created topic "ds-topic".
  • 44.
    *ํ”„๋กœ๋“€์„œ ์ ‘์† ์‹œ> ํ”„๋กฌํ”„ํŠธ blink ํ™•์ธ ํ›„ ๋ฉ”์‹œ์ง€(Hello World!) ์ž…๋ ฅ ์นดํ”„์นด ํ…Œ์ŠคํŠธ ์นดํ”„์นด ํ”„๋กœ๋“€์„œ โ€ข Kafka-console-producer.sh ์ด์šฉํ•˜์—ฌ ํ”„๋กœ๋“€์„œ ์ ‘์† โ€ข ./kafka-console-producer.sh --broker-list master:9092,slave1:9092,slave2:9092 --topic kopo-topic
  • 45.
    ์นดํ”„์นด ํ…Œ์ŠคํŠธ ์นดํ”„์นด ์ปจ์Šˆ๋จธ์ƒ์„ฑ โ€ข Kafka-console-consumer.sh ์ด์šฉํ•˜์—ฌ kopo-topic ๋ฉ”์‹œ์ง€ ํ™•์ธ ๊ฐ€๋Šฅ โ€ข ./kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --topic kopo-topic --from- beginning
  • 46.
  • 47.
    Kafka Topic source :https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85
  • 48.
    image source :https://kafka.apache.org/documentation/ โ€ข topic : ๋ฉ”์„ธ์ง€ ์ข…๋ฅ˜ โ€ข partitions : topic ์ด ๋‚˜๋ˆ ์ง€๋Š” ๋‹จ์œ„ โ€ข Log : 1๊ฐœ์˜ ๋ฉ”์„ธ์ง€ โ€ข offset : ํŒŒํ‹ฐ์…˜ ๋‚ด์—์„œ ๊ฐ ๋ฉ”์‹œ์ง€๊ฐ€ ๊ฐ€์ง€๋Š” unique id Topic ์นดํ”„์นด ํด๋Ÿฌ์Šคํ„ฐ๋Š” ํ† ํ”ฝ์—๋‹ค๊ฐ€ ๋ฐ์ดํ„ฐ๋ฅผ ์ €์žฅํ•จ. ์นดํ”„์นด์—์„œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ตฌ๋ถ„ํ•˜๊ธฐ ์œ„ํ•œ ๋‹จ์œ„ Partition ํ•˜๋‚˜์˜ ํ† ํ”ฝ์„ ๋ถ„ํ•  ํ•œ ๊ฒƒ (๋ณ‘๋ ฌ์ฒ˜๋ฆฌ!) ํ•˜๋‚˜์˜ ํŒŒํ‹ฐ์…˜ VS ๋ณต์ˆ˜์˜ ํŒŒํ‹ฐ์…˜ Kafka Topic Offset Unique, Sequential Number in the Partition ์˜คํ”„์…‹์„ ์ด์šฉํ•ด ๋ฉ”์‹œ์ง€์˜ ์ˆœ์„œ๋ฅผ ๋ณด์žฅํ•จ
  • 49.
    Kafka Partition ํŒŒํ‹ฐ์…˜ ์ˆ˜์˜์ฆ๊ฐ€ 1. ํŒŒ์ผ ํ•ธ๋“ค๋Ÿฌ์˜ ๋‚ญ๋น„ ๊ฐ ํŒŒํ‹ฐ์…˜์€ ๋ธŒ๋กœ์ปค์˜ ๋””๋ ‰ํ† ๋ฆฌ์™€ ๋งคํ•‘ ๋˜๊ณ , ์ €์žฅ๋˜๋Š” ๋ฐ์ดํ„ฐ์— ๋Œ€ํ•œ ํŒŒ์ผ ํ•ธ๋“ค ์ˆ˜๊ฐ€ ๋งŽ์•„์ ธ ๋ฆฌ์†Œ์Šค ์ฆ๊ฐ€ 2. ์žฅ์•  ๋ณต๊ตฌ ์‹œ๊ฐ„ ์ฆ๊ฐ€ ํ† ํ”ฝ์€ ์—ฌ๋Ÿฌ ๊ฐœ์˜ ํŒŒํ‹ฐ์…˜์œผ๋กœ ๋‚˜๋ˆ ์ง€๊ณ  ๋ธŒ๋กœ์ปค์—๋Š” ์—ฌ๋Ÿฌ ๊ฐœ์˜ ํŒŒํ‹ฐ์…˜์กด์žฌ ๏ƒ  ํ”„๋กœ๋“€์„œ, ์ปจ์Šˆ๋จธ์˜ ๋ชฉํ‘œ ์ฒ˜๋ฆฌ๋Ÿ‰๊ณผ ์ฒ˜๋ฆฌ ์†๋„๋ฅผ ๊ธฐ์ค€์œผ๋กœ ์ ์ ˆํ•œ ์ˆ˜์˜ ํŒŒํ‹ฐ์…˜์ˆ˜๋ฅผ ์„ค์ •ํ•ด์•ผํ•จ โ€ป ํŒŒํ‹ฐ์…˜์ˆ˜ ์ฆ๊ฐ€๋Š” ๊ฐ€๋Šฅํ•˜๋‚˜ ์ค„์ด๋Š” ๊ฒƒ์€ ๋ถˆ๊ฐ€ํ•จ
  • 50.
    Consumer Consumer Consumer ์„œ๋ฒ„ ์ˆ˜ ์„œ๋ฒ„๋‹น๋ฉ”์‹œ์ง€ ์ „์†ก ์ˆ˜ ํ•ฉ๊ณ„ ํ•„์š” ํŒŒํ‹ฐ์…˜ ์ˆ˜ Producer 3 10 msg /sec 3*10 msg /sec 3 Consumer 6 5 msg /sec 6*5 msg /sec ? partition Producer Producer Consumer Consumer <Broker> Producer partition partition Consumer Kafka Partition ์ฒ˜๋ฆฌ๋Ÿ‰ 10 msg /sec ์ฒ˜๋ฆฌ๋Ÿ‰ 10 msg /sec ์ฒ˜๋ฆฌ๋Ÿ‰ 10 msg /sec
  • 51.
    Consumer Consumer Consumer ์„œ๋ฒ„ ์ˆ˜ ์„œ๋ฒ„๋‹น๋ฉ”์‹œ์ง€ ์ „์†ก ์ˆ˜ ํ•ฉ๊ณ„ ํ•„์š” ํŒŒํ‹ฐ์…˜ ์ˆ˜ Producer 3 10 msg /sec 3*10 msg /sec 3 Consumer 6 5 msg /sec 6*5 msg /sec 6 partition Producer Producer Consumer Consumer <Broker> Producer partition partition Consumer partition partition partition Kafka Partition ํ•˜๋‚˜์˜ consumer๊ฐ€ ์ˆ˜์‹ ํ•  ์ˆ˜ ์žˆ๋Š” ์ฒ˜๋ฆฌ๋Ÿ‰์ด 5 msg / sec ์ด๋ฏ€๋กœ partition ์ˆซ์ž๋ฅผ 2๋ฐฐ๋กœ ๋Š˜๋ ค์ค˜์•ผ ํ•จ
  • 52.
    Kafka Partition &Topic News-Topic Partition01 News-Topic Partition01 News-Topic Partition01 News-Topic Partition02 News-Topic Partition02 News-Topic Partition02 Sports-Topic Partition01 Sports-Topic Partition01 Sports-Topic Partition01 Replication-factor : 3
  • 53.
    Kafka Replication ๏ต ์žฅ์• ๋Œ€์‘ ๊ธฐ๋Šฅ ๏ต ํŒŒํ‹ฐ์…˜์„ ๋ณต์ œํ•˜๋Š” ๊ธฐ๋Šฅ News-Topic Partition01 News-Topic Partition01 News-Topic Partition02 News-Topic Partition02 Sports-Topic Partition01 Sports-Topic Partition01 Replication-factor : 2
  • 54.
    Kafka Replication ๏ต kafka-topics.sh-โ€“zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka -โ€“topic kopo-topic -โ€“describe ๏ต Leader ๋ฒˆํ˜ธ๊ฐ€ ํŒŒํ‹ฐ์…˜์˜ ๋ฆฌ๋” ๋ธŒ๋กœ์ปค ๋ฒˆํ˜ธ ๏ต Replicas ๋ฒˆํ˜ธ : ํŒŒํ‹ฐ์…˜์— ์ฐธ์—ฌํ•œ ๋ธŒ๋กœ์ปค ๋ฒˆํ˜ธ Replication-factor ํ† ํ”ฝ์„ ์ƒ์„ฑํ•  ๋•Œ ์ ์šฉ ์šด์˜ ์ค‘์—๋„ ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ํŒฉํ„ฐ ๊ฐ’์€ ๋ณ€๊ฒฝ๊ฐ€๋Šฅ ํ† ํ”ฝ์ด ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ ํŒฉํ„ฐ 1๋กœ ์„ค์ •๋˜์–ด ์žˆ์œผ๋ฉด ํ•˜๋‚˜์˜ ๋ธŒ๋กœ์ปค์—๋งŒ ๋ฉ”์‹œ์ง€๋ฅผ ์ €์žฅํ•˜๋Š” ์ƒํƒœ Kopo-topic(๋ฆฌ๋”) , kopo-topic(ํŒ”๋กœ์›Œ) ์ฝ๊ธฐ, ์“ฐ๊ธฐ๋Š” ๋ฆฌ๋”๋ฅผ ํ†ตํ•ด์„œ๋งŒ ์ผ์–ด๋‚จ
  • 55.
    Kafka ISR ๏ต ISR: In Sync Replica ๏ต ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋˜๊ณ  ์žˆ๋Š” ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜ ๊ทธ๋ฃน์„ ์˜๋ฏธํ•จ ๏ต ISR๊ทธ๋ฃน์˜ ๊ตฌ์„ฑ์›๋งŒ์ด ๋ฆฌ๋”์˜ ์ž๊ฒฉ์„ ๊ฐ–์Œ ๏ต ๋ฆฌ๋”(Leader)๋งŒ ์ฝ๊ณ  ์“ฐ๊ธฐ, Follower๋Š” Leader์˜ ๋ฐ์ดํ„ฐ๋ฅผ ์ฝ์–ด์„œ ๋™๊ธฐํ™”(Sync) ๏ต ๋ฆฌ๋”์™€ ๋ฐ์ดํ„ฐ ๋™๊ธฐํ™”์ž‘์—…์„ ํ•˜๊ณ  ์ด๊ฒƒ์„ ์œ ์ง€ํ•˜์—ฌ ๋ฆฌํ”Œ๋ฆฌ์ผ€์ด์…˜์˜ ์‹ ๋ขฐ์„ฑ์„ ๋†’์ž„.
  • 56.
    News-Topic Partition01 News-Topic Partition01 News-Topic Partition01 News-Topic Partition02 News-Topic Partition02 News-Topic Partition02 Kafka ISR ISR LeaderFollower Follower ISR Leader FollowerFollower ์žฅ์•  ๋ฐœ์ƒ์‹œ ํ•ด๋‹น ISR์—์„œ ์ถ”๋ฐฉ๋จ ์žฅ์•  ๋ฐœ์ƒ์‹œ ํ•ด๋‹น ISR์—์„œ ์ถ”๋ฐฉ๋˜๊ณ  ๋‚˜๋จธ์ง€ Follower ์ค‘์—์„œ Leader๋ฅผ ์ž๋™ ์„ ์ถœํ•จ Leader๋Š” Follower๋“ค์ด ์ฃผ๊ธฐ์ ์œผ๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ์š”์ฒญํ•˜๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธ ๋งŒ์•ฝ, ์„ค์ •๋œ ์ผ์ •์ฃผ๊ธฐ(replica.lag.time.max.ms)๋งŒํผ ํ™•์ธ ์š”์ฒญ์ด ์˜ค์ง€์•Š๋Š”๋‹ค๋ฉด, Leader ๋Š” ํ•ด๋‹น Follower์˜ ์ด์ƒ์„ ๊ฐ์ง€ํ•˜๊ณ , ISR์—์„œ ์ถ”๋ฐฉ
  • 57.
    Kafka ISR ๏ต ํด๋Ÿฌ์Šคํ„ฐ๋‚ด์˜๋ชจ๋“  ๋ธŒ๋กœ์ปค๊ฐ€ ๋‹ค์šด์ด ๋œ๋‹ค๋ฉด? 1. ๋งˆ์ง€๋ง‰ ๋ฆฌ๋”๊ฐ€ ์‚ด์•„๋‚˜๊ธฐ๋ฅผ ๊ธฐ๋‹ค๋ฆฐ๋‹ค. ๏ƒ  ๋ฉ”์‹œ์ง€ ์†์‹ค์—†์ด ํ”„๋กœ๋“€์„œ์˜ ์š”์ฒญ์„ ์ฒ˜๋ฆฌํ•˜๋ฉด์„œ ์„œ๋น„์Šค ์ง€์†๊ฐ€๋Šฅ 2. ๋จผ์ € ์‚ด์•„๋‚˜๋Š” Broker๊ฐ€ ์ž๋™์œผ๋กœ ๋ฆฌ๋”๊ฐ€ ๋œ๋‹ค. ๏ƒ  ๋ฉ”์‹œ์ง€ ์ผ๋ถ€ ์†์‹ค๋  ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์ง€๋งŒ ์„œ๋น„์Šค๋Š” ์กฐ์†ํžˆ ์ •์ƒํ™”ํ•  ์ˆ˜ ์žˆ์Œ(downtime ์ตœ์†Œํ™”) ๏ต ๋ฐ์ดํ„ฐ์˜ ์™„์ „ํ•จ์„ ์ค‘์‹œํ•˜๋А๋ƒ, ์ง€์†์ ์ด ์„œ๋น„์Šค๋ฅผ ์ค‘์‹œ ํ•˜๋А๋ƒ์— ๋”ฐ๋ผ ์„ค์ •ํ•  ์ˆ˜ ์žˆ์Œ. Unclean.leader.election.enable = false == > 1๋ฒˆ ์ผ๊ด€์„ฑ Unclean.leader.election.enable = true == > 2๋ฒˆ ๊ฐ€์šฉ์„ฑ
  • 58.
    Kafka Producer ์นดํ”„์นด ํ† ํ”ฝ์ƒ์„ฑ โ€ข Kafka-topics.sh ์ด์šฉํ•˜์—ฌ kopo-topic ์ƒ์„ฑ โ€ข cd /home/kafka/bin โ€ข ./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --replication-factor 1 --partitions 1 --topic kopo-topic --create
  • 59.
    *ํ”„๋กœ๋“€์„œ ์ ‘์† ์‹œ> ํ”„๋กฌํ”„ํŠธ blink ํ™•์ธ ํ›„ ๋ฉ”์‹œ์ง€(Hello World!) ์ž…๋ ฅ ์นดํ”„์นด ํ”„๋กœ๋“€์„œ โ€ข Kafka-console-producer.sh ์ด์šฉํ•˜์—ฌ ํ”„๋กœ๋“€์„œ ์ ‘์† โ€ข ./kafka-console-producer.sh --broker-list master:9092,slave1:9092,slave2:9092 --topic kopo-topic -โ€“request-required-acks 1 Kafka Producer ํ”„๋กœ๋“€์„œ ์ฃผ์š” ์˜ต์…˜ Broker-list ์นดํ”„์นด ํด๋Ÿฌ์Šคํ„ฐ์— ์ฒ˜์Œ์—ฐ๊ฒฐ ํ•˜๊ธฐ ์œ„ํ•œ ํ˜ธ์ŠคํŠธ์™€ ํฌํŠธ ์ •๋ณด๋กœ ๊ตฌ์„ฑ๋œ ๋ฆฌ์ŠคํŠธ ์ •๋ณด ๋‚˜ํƒ€๋ƒ„ ํ˜ธ์ŠคํŠธ ํ•˜๋‚˜๋งŒ ์“ธ ์ˆ˜ ๋„ ์žˆ์ง€๋งŒ ์žฅ์• ๊ฐ€ ์ƒ๊ธธ ์ˆ˜ ์žˆ๊ธฐ๋•Œ๋ฌธ์— ๋ฆฌ์ŠคํŠธ ์ „์ฒด๋ฅผ ์ž…๋ ฅํ•˜๋Š” ๊ฑธ ๊ถŒ์žฅ Acks ํ”„๋กœ๋“€์„œ๊ฐ€ ์นดํ”„์นด ํ† ํ”ฝ์˜ ๋ฆฌ๋”์—๊ฒŒ ๋ฉ”์‹œ์ง€๋ฅผ ๋ณด๋‚ธ ํ›„ ์š”์ฒญ์„ ์™„๋ฃŒํ•˜๊ธฐ ์ „ ์Šน์ธ์˜ ์ˆ˜ ํ•ด๋‹น ์˜ต์…˜์˜ ์ˆ˜๊ฐ€ ์ž‘์œผ๋ฉด ์„ฑ๋Šฅ์ด ์ข‹์ง€๋งŒ, ๋ฉ”์‹œ์ง€ ์†์‹ค ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์Œ
  • 60.
    Producer ๋ฉ”์‹œ์ง€๋ฅผ ์ƒ์‚ฐํ•ด์„œ ์นดํ”„์นด์˜ํ† ํ”ฝ์œผ๋กœ ๋ฉ”์‹œ์ง€๋ฅผ ๋ณด๋‚ด๋Š” ์—ญํ• ์„ ํ•˜๋Š” ์„œ๋ฒ„, ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ๋“ฑ ์ฃผ์š”๊ธฐ๋Šฅ 1. ๊ฐ๊ฐ์˜ ๋ฉ”์‹œ์ง€๋ฅผ Topic์˜ Partition์— ๋งคํ•‘ํ•˜๊ณ  Partition์˜ ๋ฆฌ๋”์— ์š”์ฒญ์„ ๋ณด๋‚ด๋Š” ๊ฒƒ 2. key ๊ฐ’์„ ์ •ํ•ด ํ•ด๋‹น key ๋ฅผ ๊ฐ€์ง„ ๋ชจ๋“  ๋ฉ”์‹œ์ง€๋ฅผ ๋™์ผํ•œ ํŒŒํ‹ฐ์…˜์œผ๋กœ ์ „์†ก ๋งŒ์•ฝ key ๊ฐ’์„ ์ž…๋ ฅํ•˜์ง€์•Š์œผ๋ฉด Partition ๋ผ์šด๋“œ๋กœ๋นˆ ๋ฐฉ์‹์œผ๋กœ ํŒŒํ‹ฐ์…˜์— ๊ท ๋“ฑํ•˜๊ฒŒ ๋ถ„๋ฐฐ Kafka Producer from kafka import KafkaProducer producer = kafkaProducer(acks=1, compresstion_type=โ€œgzaipโ€, bootstrap_server=โ€˜master:9092,slave1:9092,slave2:9092โ€™) for i range(1,10): if I % 2 == 1: producer.send(โ€˜kopo-topicโ€™,key=โ€˜1โ€™, value=โ€˜Kopo Testโ€™) else : producer.send(โ€˜kopo-topicโ€™,key=โ€˜2โ€™, value=โ€˜Korea Testโ€™)
  • 61.
    [Producer] Kafka-python https://github.com/dpkp/kafka-python โ˜…pip install kafka-python ์œผ๋กœ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์„ค์น˜ (์‚ฌ์ „ ์กฐ๊ฑด : python 3.xx ์„ค์น˜) from kafka import KafkaProducer producer = KafkaProducer(bootstrap_servers='master:9092,slave1:9092,slave2:9092') future = producer.send('kopo-topic','HWY TEST') result = future.get(timeout=60) print(result) print("END") [root@slave1 pyKafkaTest]# python producer-option.py RecordMetadata(topic='kopo-topic', partition=0, topic_partition=TopicPartition(topic='kopo-topic', partition=0), offset=7, timestamp=1542334214067, checksum=None, serialized_key_size=-1, serialized_value_size=8) END ์ด๋ฅผ ํƒ€ ์„œ๋ฒ„(slave2)์—์„œ consumer๋กœ ํ™•์ธํ•œ ๊ฒฐ๊ณผ๋Š” ์•„๋ž˜์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค. [root@master ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --topic kopo-topic --from-beginning HWY TEST
  • 62.
    Kafka Consumer ์นดํ”„์นด ์ปจ์Šˆ๋จธ์ƒ์„ฑ โ€ข Kafka-console-consumer.sh ์ด์šฉํ•˜์—ฌ kopo-topic ๋ฉ”์‹œ์ง€ ํ™•์ธ ๊ฐ€๋Šฅ โ€ข ./kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --topic kopo-topic --from- beginning
  • 63.
    from kafka importKafkaConsumer consumer = KafkaConsumer('kopo-topic',group_id='kopo-consumer',bootstrap_servers='master:9092,slave1:9092,slave2:9092',enable_auto_commit=True,auto_offset_reset='latest') for message in consumer: print "Topic : %s, Partition: %d, Offset: %d, Key: %s, Value: %s" % (message.topic, message.partition, message.offset, message.key, message.value.decode('utf-8')) https://github.com/dpkp/kafka-python [Consumer] Kafka-python โ˜… pip install kafka-python ์œผ๋กœ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์„ค์น˜ (์‚ฌ์ „ ์กฐ๊ฑด : python 3.xx ์„ค์น˜) late response
  • 64.
    [Consumer] Kafka-python https://github.com/dpkp/kafka-python fromkafka import KafkaConsumer consumer = KafkaConsumer('kopo-topic',group_id='kopo-consumer',bootstrap_servers='master:9092,slave1:9092,slave2:9092โ€™, enable_auto_commit=True,auto_offset_reset='latest',fetch_min_bytes=1) while True: message = consumer.poll(1.0) #print (message.keys()) #print (message.values()) #print "Offset: %d, Key: %s, Value: %s" % (message.offset, message.key, message.value.decode('utf-8')) for tp, mg in message.items(): for m in mg: print ("%s : %d %d : key=%s value=%s" % ( tp.topic, tp.partition, m.offset, m.key, m.value)) consumer ์ฃผ์š” ์˜ต์…˜: https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html# https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html
  • 65.
    Partition and Message News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 kafka-topics.shโ€“zookeeper master:2181,slave1:2181,slave2:2181/news-kafka โ€“topic news-topic โ€“partitions 3 โ€“replication-factor 1 --create kafka-console-producer.sh โ€“broker-list master:9092,slave1:9092,slave2:9092 โ€“topic news-topic 1 2 3 4 5 1 4 2 5 3 kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 โ€“topic news-topic โ€“from-beginning 1 4 2 5 3 kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 โ€“topic news-topic โ€“partition 0 โ€“from-beginning kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 โ€“topic news-topic โ€“partition 1 โ€“from-beginning kafka-console-consumer.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 โ€“topic news-topic โ€“partition 2 โ€“from-beginning consumer๋Š” producer๊ฐ€ ์–ด๋–ค ์ˆœ์„œ๋กœ ๋ฉ”์‹œ์ง€๋ฅผ ๋ณด๋ƒˆ๋Š”์ง€ ๋ชจ๋ฆ„ Topic์˜ Partition์ด ๋ณต์ˆ˜์ธ ๊ฒฝ์šฐ, ๋ฉ”์‹œ์ง€์˜ ์ˆœ์„œ๋ฅผ ๋ณด์žฅํ•˜์ง€ ์•Š์Œ (Partition๋‚ด์˜ ์ˆœ์„œ๋Š” ๋ณด์žฅ๋˜๋‚˜, Partition๊ฐ„ ์ˆœ์„œ๋Š” ๋ณด์žฅํ•˜์ง€ ์•Š์Œ)
  • 66.
    News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 Consumer Group <kopo-consumer> Consumer Consumer Group Kafka๋Š” ํ•˜๋‚˜์˜Topic์— ๋Œ€ํ•˜์—ฌ ์—ฌ๋Ÿฌ Consumer Group์ด ๋™์‹œ์— ์ ‘์†ํ•˜์—ฌ ๋ฉ”์‹œ์ง€๋ฅผ ์ˆ˜์‹ ํ•  ์ˆ˜ ์žˆ๋‹ค. ์ฒ˜๋ฆฌ๋Ÿ‰ โ†‘, ๋ถ€ํ•˜ โ†‘ consumer๋งŒ ์ถ”๊ฐ€ํ•  ๊ฒฝ์šฐ, : ๊ธฐ์กด์˜ consumer์™€ offset์ •๋ณด๊ฐ€ ์—‰์ผœ์„œ ๋ฉ”์‹œ์ง€ ์ค‘๋ณต์ฒ˜๋ฆฌ ๋ฐœ์ƒ
  • 67.
  • 68.
    Consumer Group News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 Consumer Group <kopo-consumer> Consumer Consumer Consumer Consumer sendsHeartBeat to broker - consumer polls - message offset commit * heartbeat.interval.ms = 3000 by default * session.timeout.ms = 10000 by default Consumer๊ฐ€ heartbeat.interval.ms ๋™์•ˆ heartbeat๋ฅผ ๋ณด๋‚ด์ง€ ์•Š๋Š”๋‹ค๋ฉด? session.timeout.ms ์ดํ›„์— ๋ฐ”๋กœ rebalance ์‹œ์ž‘ (์ฆ‰, ํ•ด๋‹น consumer๋Š” out๋จ) Dead
  • 69.
  • 70.
    ์ปจ์Šˆ๋จธ ๋ฆฌ์ŠคํŠธ ๋ช…๋ น์–ด --bootstrap: ๋ธŒ๋กœ์ปค๋ฆฌ์ŠคํŠธ ์„ค์ • --list : ์ปจ์Šˆ๋จธ ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ณด์—ฌ์คŒ. ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:092,slave2:9092 --list Kafka Consumer [root@master bin]# ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:092,slave2:9092 --list OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads= Note: This will not show information about old Zookeeper-based consumers. [2019-05-15 23:03:11,612] WARN Connection to node -2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) kopo-consumer
  • 71.
    Kafka Consumer ์ปจ์Šˆ๋จธ์˜ ์ƒ์„ธ์ •๋ณด๋ฅผ ์กฐํšŒํ•จ --bootstrap-server : ๋ธŒ๋กœ์ปค ๋ฆฌ์ŠคํŠธ ์ž…๋ ฅ --group : ์ปจ์Šˆ๋จธ ๊ทธ๋ฃน ์ด๋ฆ„ ์„ค์ • --describe : ์ƒ์„ธ์ •๋ณด ๋ณด๊ธฐ *LAG : ํ˜„์žฌํ† ํ”ฝ์˜ ์ €์žฅ๋œ ๋ฉ”์‹œ์ง€์™€ ์ปจ์Šˆ๋จธ๊ฐ€ ๊ฐ€์ ธ๊ฐ„ ๋ฉ”์‹œ์ง€์˜ ์ฐจ์ด ํ”„๋กœ๋“€์„œ์™€ ์ปจ์Šˆ๋จธ์˜ ์ฒ˜๋ฆฌ ์†๋„์ฐจ์ด๋ฅผ ์•Œ ์ˆ˜ ์žˆ์Œ ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --group kopo-consumer --describe [root@master bin]# ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --group kopo-consumer --describe OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Note: This will not show information about old Zookeeper-based consumers. TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID test-topic 0 21 21 0 kafka-python-1.4.3-3d11d79a-04f0-4da5-a1b9-808c34f99865/192.168.0.31 kafka-python-1.4.3 kopo-topic 0 425 426 1 - - - [root@master bin]#
  • 72.
    Consumer Commit News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 Consumer Group <kopo-consumer> Consumer 1 Consumer2 Consumer 3 Consumer๋Š” Matching Partition์— ๋Œ€ํ•ด ์ˆ˜์‹ ํ•œ ๋ฉ”์‹œ์ง€์˜ offset ์ •๋ณด๋ฅผ ๊ธฐ๋กํ•˜๊ณ  ์žˆ์Œ ์ด๋Ÿฌํ•œ ๋™์ž‘์„ Commit์ด๋ผ๊ณ  ํ•จ โ€ข enable.auto.commit=true โ€ข auto.commit.interval.ms= 5 (sec) by default 1 2 3 4 5 6 7 8 9 10Partition 03 Consumer 3 1 2 3 4 5 sec 5 sec 5 5๊นŒ์ง€ ์ฒ˜๋ฆฌํ–ˆ์œผ๋‚˜, Commit์€ 4๊นŒ์ง€ ํ•œ ์ƒํ™ฉ
  • 73.
    Consumer Commit News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 Consumer Group <kopo-consumer> Consumer 1 Consumer2 1 2 3 4 5 6 7 8 9 10Partition 03 Consumer 3 1 2 3 4 5 sec 5 sec 5 5๊นŒ์ง€ ์ฒ˜๋ฆฌํ–ˆ์œผ๋‚˜, Commit์€ 4๊นŒ์ง€ ํ•œ ์ƒํ™ฉ Consumer 2 5 6 Consumer 2๋Š” 5 ๋ถ€ํ„ฐ ์ฝ์–ด๋“ค์ž„
  • 74.
    Consumer Commit Manual Commit ๋ฉ”์‹œ์ง€๊ฐ€์ฒ˜๋ฆฌ๊ฐ€ ์™„๋ฃŒ๋  ๋•Œ๊นŒ์ง€ Commit์„ ๋ณด๋ฅ˜ โ€ข enable.auto.commit : false โ€ข commitSync ํ•จ์ˆ˜ ํ˜ธ์ถœ [์ฐธ๊ณ ] ํŠน์ • ํŒŒํ‹ฐ์…˜์ด๋‚˜ ํŠน์ • ์˜คํ”„์…‹๋ถ€ํ„ฐ ๊ฐ€์ ธ์˜ค๋Š” ๊ธฐ๋Šฅ๋„ ์žˆ์Œ News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 News-Topic Partition01 News-Topic Partition02 News-Topic Partition03 Consumer Group <kopo-consumer> Consumer 1 Consumer 2
  • 75.
  • 76.
    How to findZookeeper Leader in zookeeper ensemble? To identify a Zookeeper leader/follower, there are few possible options. Mentioning 2 for keeping this document simple. 1. Check the zookeeper log file on each node, and grep as below: # grep LEAD /var/log/zookeeper/zookeeper-zookeeper-server-xyz.out 2016-02-29 22:33:47,113 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer@829] - LEADING 2016-02-29 22:33:47,114 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:Leader@358] - LEADING - LEADER ELECTION TOOK - 9066 2. Use "nc" command to listen TCP communication on port 2181 and determine if the ZooKeeper server is a leader or a follower. Bellow is a way to check the same: # echo stat | nc localhost 2181 Zookeeper version: 3.4.6-3485--1, built on 12/16/2015 02:35 GMT Clients: /172.25.16.243:37619[1](queued=0,recved=54,sent=54) /172.25.16.253:58745[1](queued=0,recved=53,sent=53) Latency min/avg/max: 0/1/362 Received: 627 Sent: 626 Connections: 9 Outstanding: 0 Zxid: 0x400000003 Mode: leader Node count: 182 Here, we see that the Mode is shown as "leader". If it is a follower, the Mode will be seen as "follower". For example: # echo stat | nc localhost 2181 | grep Mode Mode: leader # echo stat | nc localhost 2181 | grep Mode Mode: follower source : https://community.hortonworks.com/content/supportkb/49435/how-to-find-zookeeper-leader-in-zookeeper-ensemble.html
  • 77.
    Why Kafka isso fast? - page cache source : https://www.slideshare.net/baniuyao/kafka-24299168
  • 78.
  • 79.
    ํ† ํ”ฝ ์ƒ์„ฑ ๋ช…๋ น์–ด --zookeeper: ์ฃผํ‚คํผ ์ •๋ณด, --replication-factor : ํ† ํ”ฝ ๋ณต์ œ ์„ค์ •, --partitions : ํ† ํ”ฝ ๋‚ด ํŒŒํ‹ฐ์…˜์ˆ˜ ์„ค์ • --topic : ํ† ํ”ฝ๋„ค์ž„ ์„ค์ •, --create : ํ† ํ”ฝ์ƒ์„ฑ 1.ํ† ํ”ฝ ์ƒ์„ฑ ๋ช…๋ น์–ด ./kafka-topics.sh โ€“zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka -โ€“replication-factor 1 --partitions 1 -โ€“topic kopo-test --create
  • 80.
    ํ† ํ”ฝ ๋ฆฌ์ŠคํŠธ ๋ช…๋ น์–ด --zookeeper: ์ฃผํ‚คํผ ์ •๋ณด --list : ํ•ด๋‹น ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ๊ฐ€์ง€๊ณ  ์žˆ๋Š” ํ† ํ”ฝ๋ฆฌ์ŠคํŠธ๋ฅผ ๋ณด์—ฌ์คŒ 2.ํ† ํ”ฝ ๋ฆฌ์ŠคํŠธ ๋ช…๋ น์–ด ./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --list
  • 81.
    ํ† ํ”ฝ ์ƒ์„ธ๋ณด๊ธฐ ๋ช…๋ น์–ด --zookeeper: ์ฃผํ‚คํผ ์ •๋ณด, --topic : ์ƒ์„ธ์ •๋ณด๋ฅผ ์•Œ๊ณ  ์‹ถ์€ ํ† ํ”ฝ์ด๋ฆ„ ์„ค์ • --describe : ํ•ด๋‹น ํ† ํ”ฝ์˜ ์ƒ์„ธ ์ •๋ณด๋ฅผ ํ™•์ธ 3.ํ† ํ”ฝ ์ƒ์„ธ๋ณด๊ธฐ ๋ช…๋ น์–ด ./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --topic test-topic --describe Topic:test-topic PartitionCount:1 ReplicationFactor:1 Configs: Topic: test-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
  • 82.
    ๋””์Šคํฌ ๊ณต๊ฐ„ ํ™•๋ณดํ•˜๋Š”๊ฐ€์žฅ ์ข‹์€ ๋ฐฉ๋ฒ•์€ ๊ณต๊ฐ„์„ ๋งŽ์ด ์ฐจ์ง€ํ•˜๋Š” ํ† ํ”ฝ์˜ ๋ณด๊ด€์ฃผ๊ธฐ๋ฅผ ์ค„์ด๋Š” ๊ฒƒ ๋ธŒ๋กœ์ปค์˜ ๊ธฐ๋ณธ ์„ค์ • ๊ฐ’์€ ๋ณด๊ด€์ฃผ๊ธฐ 7์ผ๋กœ ์„ค์ • ๋˜์–ด์žˆ์Œ. --zookeeper : ์ฃผํ‚คํผ ์ •๋ณด, --alter : ๋ณ€๊ฒฝ ์„ค์ •, --entity-type : topics๋ฅผ ์ถ”๊ฐ€, --entity-name : ๋ณ€๊ฒฝํ•˜๊ณ ์ž ํ•˜๋Š” ํ† ํ”ฝ์ด๋ฆ„ ์„ค์ • --add-config : ๋ณด๊ด€์ฃผ๊ธฐ๋ฅผ 1์‹œ๊ฐ„์„ ์˜๋ฏธํ•˜๋Š” retention.ms=360000๋ฅผ ์ž…๋ ฅ *์ถ”๊ฐ€๋œ ์˜ต์…˜์„ ์‚ญ์ œํ•˜๊ณ ์ž ํ• ๋• -โ€“delete-config retention.ms๋ฅผ ์ž…๋ ฅํ•˜๋ฉด ๋Œ 4.ํ† ํ”ฝ์˜ ๋ฉ”์‹œ์ง€ ๋ณด๊ด€์ฃผ๊ธฐ ์„ค์ • ๋ณ€๊ฒฝ ./kafka-configs.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --alter --entity-type topics --entity-name test-topic --add-config retention.ms=3600000 ./kafka-configs.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --alter --entity-type topics -entity-name test-topic --delete-config retention.ms ํ† ํ”ฝ ๋ฉ”์‹œ์ง€ ๋ณด๊ด€์ฃผ๊ธฐ ๋ณ€๊ฒฝ ํ† ํ”ฝ ๋ฉ”์‹œ์ง€ ๋ณด๊ด€์ฃผ๊ธฐ ์˜ต์…˜ ์‚ญ์ œ
  • 83.
    ํ† ํ”ฝ์˜ ํŒŒํ‹ฐ์…˜ ์ˆ˜๋ณ€๊ฒฝ ํ† ํ”ฝ์˜ ํŒŒํ‹ฐ์…˜์€ ์ฆ๊ฐ€๋Š” ๊ฐ€๋Šฅํ•˜์ง€๋งŒ, ๊ฐ์†Œ๋Š” ๋ถˆ๊ฐ€๋Šฅํ•จ ์ฒ˜์Œ ํ† ํ”ฝ์„ ์ƒ์„ฑํ•  ๋•Œ ํŒŒํ‹ฐ์…˜ ์ˆ˜๋ฅผ ์ฒ˜๋ฆฌ๋Ÿ‰์— ๋Œ€ํ•ด ๋ฏธ๋ฆฌ ํ™•์ธํ•˜๊ณ  ์„ค์ •! --zookeeper : ์ฃผํ‚คํผ ์ •๋ณด --alter : ๋ณ€๊ฒฝ ์˜ต์…˜ ์„ ํƒ --topic : ํ•ด๋‹น ํ† ํ”ฝ ์„ค์ • --partitions : ํŒŒํ‹ฐ์…˜ ์ˆ˜ ์„ค์ • ./kafka-topics.sh -zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka -alter -topic test-topic -partitions 2 5.ํ† ํ”ฝ์˜ ํŒŒํ‹ฐ์…˜ ์ˆ˜ ๋ณ€๊ฒฝ [root@master bin]# ./kafka-topics.sh --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --topic test-topic --describe Topic:test-topic PartitionCount:2 ReplicationFactor:1 Configs:retention.ms=3600000 Topic: test-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Topic: test-topic Partition: 1 Leader: 2 Replicas: 2 Isr: 2
  • 84.
  • 85.
    ์นดํ”„์นด ๋งค๋‹ˆ์ € ์„ค์น˜ ์นดํ”„์นด๋งค๋‹ˆ์ €๋ฅผ wget์œผ๋กœ ๋‹ค์šด์„ ๋ฐ›๋Š”๋‹ค. wget https://github.com/yahoo/kafka-manager/archive/1.3.3.17.zip
  • 86.
  • 87.
    Cd kafka-manager-1.3.3.17 ๋กœ์ด๋™ ํ•œ ๋’ค Zipํ˜•ํƒœ์˜ ๋ฐฐํฌ ํŒŒ์ผ์„ ์ƒ์„ฑํ•œ๋‹ค. ./sbt clean dist [์ฐธ๊ณ ] ๋งŒ์•ฝ javac ๊ฐ€ ์—†์–ด์„œ ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•œ๋‹ค๋ฉด? ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์„ค์น˜๊ฐ€๋Šฅํ•œ jdk๋ฅผ ํ™•์ธํ•œ๋‹ค. # yum list java*jdk-devel ์„ค์น˜๊ฐ€๋Šฅํ•œ jdk ๋ฆฌ์ŠคํŠธ ์ค‘ jdk 1.8 ๋ฒ„์ „์„ ์„ค์น˜ํ•œ๋‹ค.(์•„๋ž˜ ์˜ˆ์‹œ) # yum install java-1.8.0-openjdk-devel.x86_64 ์„ค์น˜ ํ›„ java compiler ๋ฒ„์ „์„ ํ™•์ธํ•œ๋‹ค. # javac โ€“version ๋‹ค์‹œ ./sbt clean dist ๋ฅผ ์‹คํ–‰ํ•œ๋‹ค.
  • 88.
    ์ถœ๋ ฅ๋ฌธ์˜ ๋งˆ์ง€๋ง‰ ๊ฒฝ๋กœ์˜ํŒŒ์ผ์„ /usr/local๋กœ ๋ณต์‚ฌํ•œ๋‹ค. ๊ฒฝ๋กœํ™•์ธ
  • 89.
    Cd /usr/local/ ๊ฒฝ๋กœ๋กœ์ด๋™ํ•œ ๋’ค ๋ณต์‚ฌํ•œ zipํŒŒ์ผ์˜ ์••์ถ•์„ ํ‘ผ๋‹ค.
  • 90.
    ์••์ถ• ํ’€๊ณ  ๋‚œ๋’ค ์ƒ๊ธด ๋””๋ ‰ํ† ๋ฆฌ์—์„œ vi conf/application.conf ์ฃผํ‚คํผ ์ฃผ์†Œ๋ฅผ ์„ค์ •ํ•ด์ค€๋‹ค.
  • 91.
    ์„ค์ •์„ ์™„๋ฃŒ ํ•œ๋’ค ์นดํ”„์นด ๋งค๋‹ˆ์ €๋ฅผ ์‹คํ–‰ํ•œ๋‹ค. ์ด๋•Œ ํ™˜๊ฒฝ์„ค์ •๊ณผ ์›นUI์—์„œ ์‚ฌ์šฉํ•  port์ •๋ณด๋ฅผ ์ง€์ •ํ•˜์—ฌ ์‹คํ–‰ํ•œ๋‹ค. Master:9000์œผ๋กœ ์ ‘์†ํ•œ๋‹ค. 9000๋ฒˆ ํฌํŠธ ๋ฐฉํ™”๋ฒฝ ํ•ด์ œ ํ•„์ˆ˜
  • 92.
    ํด๋Ÿฌ์Šคํ„ฐ ์ •๋ณด๋ฅผ ์ถ”๊ฐ€ํ•˜๋ฉด ์นดํ”„์นด๋งค๋‹ˆ์ €์—์„œ ๊ธฐ์กด์˜ ํด๋Ÿฌ์Šคํ„ฐ๊ฐ€ ๋‚˜ํƒ€๋‚œ๋‹ค.
  • 93.
    ํด๋Ÿฌ์Šคํ„ฐ์— ๊ธฐ์กด์˜ ์ƒ์„ฑ๋˜์–ด์žˆ๋˜ topic๊ณผ ์—ฐ๊ฒฐ๋œ broker๋“ฑ ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋‹ค์–‘ํ•œ ์ •๋ณด๊ฐ€ ํ•œ๋ˆˆ์— ๋ณด๊ธฐ ์‰ฝ๊ณ  ํŽธ๋ฆฌํ•˜๊ฒŒ ๋‚˜ํƒ€๋‚จ.
  • 94.
    Topic์ด๋‚˜ broker๋“ฑ ์ƒ์„ธํ•œ์ปจํŠธ๋กค๋„ ์‰ฝ๊ฒŒ ๊ฐ€๋Šฅํ•ด์ง„๋‹ค.
  • 95.
  • 96.
    Kafka ํด๋Ÿฌ์Šคํ„ฐ์˜ ๋ชจ๋“ ๋ถ„์‚ฐ ์ฒ˜๋ฆฌ๋Š” Zookeeper๋ฅผ ๋งค๊ฐœ๋กœ ์ด๋ฃจ์–ด ์ง„๋‹ค. ์ฆ‰, zookeeper์˜ ์—ญํ• ์€ ๋ถ„์‚ฐ ํ™˜๊ฒฝ์—์„œ ์„œ๋ฒ„๋“ค ๊ฐ„์— ์ƒํ˜ธ ์กฐ์ •์ด ํ•„์š”ํ•œ ๋‹ค์–‘ํ•œ ์„œ๋น„์Šค๋ฅผ ์ œ๊ณตํ•˜๋Š” ์‹œ์Šคํ…œ์ž…๋‹ˆ๋‹ค. 1, ํ•˜๋‚˜์˜ ์„œ๋ฒ„์—๋งŒ ์„œ๋น„์Šค๊ฐ€ ์ง‘์ค‘๋˜์ง€ ์•Š๋„๋ก, ์„œ๋น„์Šค๋ฅผ ์•Œ๋งž๊ฒŒ ๋ถ„์‚ฐํ•˜์—ฌ ๋™์‹œ์— ์ฒ˜๋ฆฌํ•˜๊ฒŒ ํ•ด์ค๋‹ˆ๋‹ค. 2, ํ•˜๋‚˜์˜ ์„œ๋ฒ„์—์„œ ์ฒ˜๋ฆฌํ•œ ๊ฒฐ๊ณผ๋ฅผ ๋‹ค๋ฅธ ์„œ๋ฒ„๋“ค๊ณผ ๋™๊ธฐํ™”ํ•˜์—ฌ ๋ฐ์ดํ„ฐ์˜ ์•ˆ์ •์„ฑ์„ ๋ณด์žฅํ•ด์ค๋‹ˆ๋‹ค. 3, ์šด์˜(active) ์„œ๋ฒ„๊ฐ€ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•ด์„œ ์„œ๋น„์Šค๋ฅผ ์ œ๊ณตํ•  ์ˆ˜ ์—†์„ ๊ฒฝ์šฐ, ๋‹ค๋ฅธ ๋Œ€๊ธฐ ์ค‘์ธ ์„œ๋ฒ„๋ฅผ ์šด์˜์„œ๋ฒ„๋กœ ๋ฐ”๊ฟ”์„œ ์„œ๋น„์Šค๊ฐ€ ์ค‘์ง€ ์—†์ด ์ œ๊ณต๋˜๊ฒŒ ํ•ด์ค๋‹ˆ๋‹ค. 4, ๋ถ„์‚ฐ ํ™˜๊ฒฝ์„ ๊ตฌ์„ฑํ•˜๋Š” ์„œ๋ฒ„๋“ค์˜ ํ™˜๊ฒฝ์„ค์ •์„ ํ†ตํ•ฉ์ ์œผ๋กœ ๊ด€๋ฆฌํ•ด์ค๋‹ˆ๋‹ค. ์ฐธ๊ณ . Zookeeper์˜ ์—ญํ• 
  • 97.
    Broker 1.์ƒํƒœ๊ด€๋ฆฌ Kafka Broker๊ฐ€ ์‚ด์•„์žˆ๋‹ค๋ฉด ํ•ญ์ƒ ์ •๊ธฐ์ ์œผ๋กœ ํ•˜ํŠธ ๋น„ํŠธ ์š”์ฒญ์„ ๋ณด๋‚ด ZooKeeper๊ฐ€ ์ƒํƒœ๋ฅผ ๊ฒฐ์ • 2.ํ• ๋‹น๋Ÿ‰ ๋‹ค๋ฅธ ์ƒ์‚ฐ ๋ฐ ์†Œ๋น„ ํ• ๋‹น๋Ÿ‰์„ ๊ฐ€์ง€๊ธฐ ์œ„ํ•ด Kafka Broker๋Š” ์ผ๋ถ€ consume๋ฅผ ํ—ˆ์šฉ 3.Replicas ๊ฐ ํ† ํ”ฝ์— ๋Œ€ํ•ด Kafka์˜ Zookeeper๋Š” ์ผ๋ จ์˜ ISR๋ฅผ ์œ ์ง€ํ•˜๊ณ  ์ด์ „์— ์„ ํƒ๋œ ๋ฆฌ๋” ๋…ธ๋“œ๊ฐ€ ๋‹ค์šด๋˜๋ฉด ํ˜„์žฌ ์‚ด์•„์žˆ๋Š” ๋…ธ๋“œ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ZooKeeper๊ฐ€ ์ƒˆ๋กœ์šด ๋ฆฌ๋”๋ฅผ ์„ ์ถœํ•จ. 4.Node์™€ topic ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ ZooKeeper๋Š” broker์˜ ๋…ธ๋“œ์™€ ํ† ํ”ฝ์„ ๋“ฑ๋กํ•˜๊ณ  ์ €์žฅํ•œ๋‹ค. ์ฐธ๊ณ . Zookeeper์˜ ์—ญํ• 
  • 101.
    consumer 1.Offsets ZooKeeper๋Š” Kafka์˜ 0.9.1๋ฆด๋ฆฌ์Šค์—์„œ ์†Œ๋น„์ž ์˜คํ”„์…‹์„ ์œ„ํ•œ ๊ธฐ๋ณธ ์ €์žฅ ์—”์ง„ Consumer๊ฐ€ ์–ผ๋งˆ๋‚˜ ๋งŽ์€ ๋ฉ”์‹œ์ง€๋ฅผ ๊ฐ ์†Œ๋น„์ž๊ฐ€ ์†Œ๋น„ํ•˜๋Š”์ง€์— ๋Œ€ํ•œ ๋ชจ๋“  ์ •๋ณด๋Š” ZooKeeper์— ์ €์žฅ๋ฉ๋‹ˆ๋‹ค. 2. Registry ๊ณต๊ฐœ ๋ถ„์‚ฐํ˜• ๊ตฌ์„ฑ ์„œ๋น„์Šค, ๋™๊ธฐ ์„œ๋น„์Šค ๋ฐ ๋Œ€์šฉ๋Ÿ‰ ๋ถ„์‚ฐ ์‹œ์Šคํ…œ์„ ์œ„ํ•œ ๋„ค์ด๋ฐ ๋ ˆ์ง€์ŠคํŠธ๋ฆฌ๋ฅผ ์ œ๊ณตํ•œ๋‹ค. ํ•˜์ง€๋งŒ ์ผ์‹œ์ ์ธ zNode์ด๊ณ , consumer๊ฐ€ ๋‹ค์šด๋˜๊ณ  ์ƒˆ ํ”„๋กœ์„ธ์Šค๋ฅผ ๋“ฑ๋กํ•˜๋ฉด ์‚ญ์ œ๋œ๋‹ค. ์ฐธ๊ณ . Zookeeper์˜ ์—ญํ• 
  • 103.
  • 104.
    1.๋ฉ”์ด๋ธํ”„๋กœ์ ํŠธ๋ฅผ ์„ค์น˜ํ•˜๊ธฐ ์œ„ํ•ด์•„๋ž˜๋กœ ์ ‘์† http://maven.apache.org/download.cgi 2. apache-mave-3.6.0-bin.zip ํŒŒ์ผ ๋‹ค์šด๋กœ๋“œ 3. ๋‹ค์šด๋ฐ›์€ ํŒŒ์ผ์„ ์›ํ•˜๋Š” ๊ฒฝ๋กœ์— ํ‘ผ๋‹ค. C:apache-maven-3.6.0
  • 105.
    ํ™˜๊ฒฝ๋ณ€์ˆ˜ ๋“ฑ๋ก 1. ์ปดํ“จํ„ฐโ€“ ์†์„ฑ โ€“ ๊ณ ๊ธ‰์‹œ์Šคํ…œ์„ค์ • โ€“ ๊ณ ๊ธ‰ํƒญ โ€“ํ™˜๊ฒฝ๋ณ€์ˆ˜ ์ฐฝ์œผ๋กœ ์ด๋™ 2. ์‹œ์Šคํ…œ ๋ณ€์ˆ˜ โ€“ ์ƒˆ๋กœ ๋งŒ๋“ค๊ธฐ ํด๋ฆญํ•˜์—ฌ ์œ„์˜ ๋ณ€์ˆ˜๋ฅผ ์ž…๋ ฅ ๋ณ€์ˆ˜ ์ด๋ฆ„ : MAVEN_HOME ๋ณ€์ˆ˜ ๊ฐ’ : Capache-maven-3.6.0 3. ์‹œ์Šคํ…œ ๋ณ€์ˆ˜ Path๋ฅผ ์„ ํƒํ•˜์—ฌ ๋ฉ”์ด๋ธ ๊ฒฝ๋กœ๋ฅผ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋“ฑ๋กํ•œ๋‹ค. ๋ณ€์ˆ˜ ๊ฐ’ : %MAVEN_HOME%bin;
  • 106.
    CMD ์ฐฝ์—์„œ mvnโ€“version ๋ช…๋ น์–ด๋ฅผ ์น˜๋ฉด ์œ„์™€ ๊ฐ™์ด ๊ฒฐ๊ณผ๋ฅผ ์ถœ๋ ฅํ•œ๋‹ค.
  • 107.
  • 108.
    1.์•„๋ž˜ํŽ˜์ด์ง€๋กœ ์ ‘์†ํ•ด ํ•ด๋‹น์นดํ”„์นด ๋ฒ„์ „์˜ ๋””ํŽœ๋˜์‹œ๋ฅผ ๋ณต์‚ฌํ•œ๋‹ค. https://mvnrepository.com/artifact/org.apache.kafka 2. Apache kafka 2.11-2.0.0์˜ ๋””ํŽœ๋˜์‹œ๋ฅผ ๋ณต์‚ฌํ•˜๊ณ  ์ดํด๋ฆฝ์Šค pom.xml์— ์ถ”๊ฐ€ํ•œ๋‹ค.
  • 109.
  • 110.
  • 111.
  • 112.
  • 113.
  • 114.
  • 115.
    Prerequisites ๏ต Python ๏ต Kafka ๏ตZookeeper ๏ต Twitter API credentials
  • 116.
    Procedure ๏ต apps.twitter.com ์ ‘์†ํ›„ ๊ฐ€์ž… ๋ฐ APP ์ƒ์„ฑ ๏ต Twitter API credential code ์ž๋™ ๋ฐœ๊ธ‰
  • 117.
    Linux(CentOS)์—์„œ Anaconda ์„ค์น˜ํ•˜๊ธฐ Anaconda: ๋ฐ์ดํ„ฐ๋ถ„์„์— ํ•„์š”ํ•œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ๋ชจ์•„๋†“์€ ํŒŒ์ด์ฌ ๊ธฐ๋ฐ˜์˜ ํ”Œ๋žซํผ 1. Anaconda ์„ค์น˜ํŒŒ์ผ(.sh) ๋‹ค์šด๋กœ๋“œ 2. Anaconda ์„ค์น˜ํ•˜๊ธฐ 3. License ์Šน์ธํ•˜๊ธฐ 4. ์„ค์น˜ํ•  ์ฃผ์†Œ ์„ค์ •ํ•˜๊ธฐ 5. PATH ์ง€์ •ํ•˜๊ธฐ 6. ์„ค์น˜์™„๋ฃŒ
  • 118.
    1. Anaconda ์„ค์น˜ํŒŒ์ผ(.sh)๋‹ค์šด๋กœ๋“œ wget https:// repo.continuum.io/archive/Anaconda3-5.2.0-Linux-x86_64.sh 2. Anaconda ์„ค์น˜ํ•˜๊ธฐ bash Anaconda3-5.2.0-Linux-x86_64.sh ์œ„์˜ ๋ช…๋ น์–ด๋ฅผ ์ž…๋ ฅํ•˜๋ฉด ์•„๋ž˜์™€ ๊ฐ™์ด Anaconda License๊ฐ€ ๋‚˜ํƒ€๋‚œ๋‹ค. Enter ํ‚ค๋ฅผ ๋ˆŒ๋Ÿฌ์„œ ์ญ‰์ญ‰ ๋„˜๊ธธ ์ˆ˜ ์žˆ๋Š”๋ฐ, ๋ฌด์ž‘์ • ๋ˆŒ๋Ÿฌ๋Œ€๋ฉด ์„ค์น˜๊ฐ€ ์ทจ์†Œ๋˜๋‹ˆ ํ•œ ์ค„์”ฉ ์กฐ์‹ฌํžˆ ๋„˜๊ธฐ๋„๋ก ํ•œ๋‹ค. 3. ๋ผ์ด์„ ์Šค ์Šน์ธํ•˜๊ธฐ yes๋ฅผ ์ž…๋ ฅํ•˜๊ณ  Enter ํ‚ค๋ฅผ ๋ˆ„๋ฅธ๋‹ค.
  • 119.
    4. ์„ค์น˜ํ•  ์ฃผ์†Œ์„ค์ •ํ•˜๊ธฐ ๊ธฐ๋ณธ๊ฐ’ ์ฃผ์†Œ(/root/anaconda3)์— ์„ค์น˜๋ฅผ ํ•  ๊ฒƒ์ด๋ผ๋ฉด Enter ํ‚ค๋ฅผ ๋ˆ„๋ฅด๊ณ , ๋‹ค๋ฅธ ๊ณณ์— ์ €์žฅ์„ ํ•˜๊ณ ์‹ถ๋‹ค๋ฉด ํ•ด๋‹น ์ฃผ์†Œ๋ฅผ ์ž…๋ ฅํ•˜๊ณ  Enter ํ‚ค๋ฅผ ๋ˆ„๋ฅธ๋‹ค.( ๋˜๋„๋ก์ด๋ฉด root ํด๋”์—๋Š” ์„ค์น˜ํ•˜์ง€ ๋ง ๊ฒƒ ) yes๋ฅผ ์ž…๋ ฅํ•˜๊ณ  ๋‚˜๋ฉด, ์–ด๋””์— ์„ค์น˜ํ•  ๊ฒƒ์ธ์ง€ ๋ฌป๋Š” ๋ฉ”์‹œ์ง€๊ฐ€ ๋‚˜ํƒ€๋‚œ๋‹ค. 5. PATH ์ง€์ •ํ•˜๊ธฐ ์„ค์น˜๊ฐ€ ๋งˆ๋ฌด๋ฆฌ๋˜๋ฉด ์‚ฌ์šฉ์ž์—๊ฒŒ PATH๋ฅผ ์ถ”๊ฐ€ํ•  ๊ฒƒ์ธ์ง€ yes ๋˜๋Š” no๋ฅผ ๋ฌป๋Š” ์ฐฝ์ด ๋‚˜ํƒ€๋‚œ๋‹ค. ํŠน๋ณ„ํ•œ ๊ฒฝ์šฐ๋ฅผ ์ œ ์™ธํ•˜๊ณ ๋Š” yes๋ฅผ ์ž…๋ ฅํ•ด์ฃผ๋ฉด ๋œ๋‹ค.
  • 120.
    PATH ์„ค์ • /etc์— ์žˆ๋Š”profile์— PATH ์ถ”๊ฐ€ vi profile export PATH=/usr/anaconda3/bin:$PATH ํŽธ์ง‘ ์™„๋ฃŒ ํ›„ source /etc/profile ์‹คํ–‰ ํ›„ ์„ค์ •์„ ์™„๋ฃŒํ•œ๋‹ค. ์ดํ›„ conda install โ€“update --all
  • 121.
    pip Python ์„ค์น˜ ํ›„terminal ์ฐฝ์—์„œ ๋‹ค์Œ ๋ช…๋ น์–ด ์ž…๋ ฅ ๏ต pip install kafka-python ๏ต pip install python-twitter ๏ต pip install tweepy *pip ์„ค์น˜๊ฐ€ ์•ˆ๋  ์‹œ: yum --enablerepo=extras install epel-release ๋ช…๋ น์–ด๋กœ EPEL repo๋ฅผ ๋จผ์ € enable ์‹œ ํ‚จ๋‹ค
  • 122.
    Kafka ์‹คํ–‰ ๏ต cd/home/zookeeper/bin ์ด๋™ ๏ต zkServer.sh start (master->slave1->slave2 ์ˆœ์œผ๋กœ ์‹คํ–‰) ๏ต cd /home/kafka/bin ์ด๋™ ๏ต ./kafka-server-start.sh /home/kafka/config/server.properties
  • 123.
    Kafka topic ์ƒ์„ฑ ๏ตcd /home/kafka/bin ๏ต ./kafka-topics.sh --create --zookeeper master:2181,slave1:2181,slave2:2181/kopo-kafka --replication-factor 1 -- partitions 1 --topic trump *trump ๋ž€ ํ† ํ”ฝ์„ ์ƒ์„ฑํ•ด Trump๋กœ ํŠธ์œ—๋˜๋Š” ๋ฉ”์‹œ์ง€๋ฅผ produceํ•  ์˜ˆ์ •
  • 124.
    Python์œผ๋กœ kafka-producer ์ƒ์„ฑ 1.ํ•„์š”libraries ํ˜ธ์ถœ from tweepy.streaming import StreamListener from tweepy import OAuthHandler from tweepy import Stream from kafka import SimpleProducer, KafkaClient
  • 125.
    Twitter API credential 2.๋ฐœ๊ธ‰๋ฐ›์€์ธ์ฆํ‚ค ์ž…๋ ฅ access_token = "" access_token_secret = "" consumer_key = "" consumer_secret = โ€œโ€
  • 126.
    Python์œผ๋กœ kafka-producer ์ƒ์„ฑ 3.Python class ์ƒ์„ฑ class StdOutListener(StreamListener): def on_data(self, data): producer.send_messages("trump", data.encode('utf-8')) print (data) return True def on_error(self, status): print (status)
  • 127.
    Python์œผ๋กœ kafka-producer ์ƒ์„ฑ 4.์„ค์ •๊ฐ’ ์ž‘์„ฑ kafka = KafkaClient("localhost:9092") producer = SimpleProducer(kafka) l = StdOutListener() auth = OAuthHandler(consumer_key, consumer_secret) auth.set_access_token(access_token, access_token_secret) stream = Stream(auth, l) stream.filter(track="trump")
  • 128.
    Output ์ฝ”๋“œ ์‹คํ–‰ ์‹œ JSONํ˜•ํƒœ์˜ ๋ฉ”์‹œ์ง€๊ฐ€ ์ถœ๋ ฅ๋˜์–ด Kafka ์— publish ํ•˜๊ฒŒ๋จ Kafka consumer ์ฝ˜์†”์—์„œ๋„ ๋™์ผํ•œ ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธ ๊ฐ€๋Šฅ