Kafka Use cases
๋ฐ์ดํฐ๋ถ์ ๊ฒฐ๊ณผ๋ก ์๋น์ค ํ์ด์ง์ ์๋ ์ปจํ ์ธ ๊ฐ ์ฌ์ฉ์์๊ฒ
์ผ๋ง๋ ์ํฅ์ ์ฃผ๋์ง ์ค์๊ฐ์ผ๋ก ๊ฐ์ง
์์ ์ซ์๋ ๊ธฐ์ฌ์ ์์
๋ค์ ์ซ์๋ ๊ธฐ์ฌ์ ์๋น์ง์
source : โkafka exerciseโ presented by Dongkeun Lee, Kangyong kim, student at dept. of smart finance, Korea Polytechnics in 2018
14.
Kafka Use cases
์นด์นด์คํก์ฑ์์ 10๋ถ ๊ฐ ์๋นํ ๋ค์ด๋ฒ ์ฐ์ ๊ธฐ์ฌ ์๋น ์ง์ ๋ถ์
source : โkafka exerciseโ presented by Dongkeun Lee, Kangyong kim, student at dept. of smart finance, Korea Polytechnics in 2018
๊ฐ์ ๋จธ์ (VM)
โข VirtualBox์ด์ฉ
โข ์ค๋น ๋ฐ ์ ๊ฒ ์ฌํญ
- VirtualBox Latest Ver.
- CentOS 7 (DVD Ver.)
- JDK 1.8
- ๋ชจ๋ ์ค์น ๋ฐ
ํ๊ฒฝ์ค์ ์ root๊ถํ์ผ๋ก
์ค์นํจ
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
18.
๊ฐ์ ๋จธ์ (VM) ์ค์น๋ฐ ์ค์
1. ๊ฐ์๋จธ์ ์ด๋ฆ ์ค์ 2~6. ๊ฐ์๋จธ์ ๋ฉ๋ชจ๋ฆฌ / ํ๋ ์ค์
2
4
3
5 6
1
5 6
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
19.
๊ฐ์ ๋จธ์ (VM) ์ค์น๋ฐ ์ค์
1. ์์ฑ๋ master ๋จธ์ ์ ๋คํธ์ํฌ ์ค์ ์ผ๋ก ๋ค์ด๊ฐ๋ค
2. ์ดํญํฐ 2์๋ ์ด๋ํฐ์ ๋ธ๋ฆฌ์ง๋ฅผ ์ ํ
3. ์ดํญํฐ 3์๋ ํธ์คํธ ์ ์ฉ ์ด๋ํฐ ์ ํ
1 2
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
20.
Cent OS 7์ค์น
1. GNOME ํ ์คํฌํ ์ ํ
2. ๋ก์ปฌ ํ์ค ๋์คํฌ ์ ํ
1 2
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
21.
Cent OS 7์ค์น
1. Root ๋น๋ฐ๋ฒํธ ์ค์
2. ์ฌ์ฉ์ ID, PW์ค์
1
2
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
22.
๋คํธ์ํฌ ์ค์
๏ต Network๊ณ ์ IP ์ค์ (Static)
* vi /etc/sysconfig/network-scripts/ifcfg-enp0s8
BOOTPROTO=static
์์ ๋ฐ ์ถ๊ฐ
ONBOOT=yes
NETMASK=255.255.255.0
IPADDR=192.168.0.30
GATEWAY=10.0.0.2
๋คํธ์ํฌ ๋์ญ๋๋ ๋ณ๊ฒฝ ๊ฐ๋ฅํจ
์ค์ ์๋ฃ ํ
systemctl restart network
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
23.
๋คํธ์ํฌ ์ค์
๏ต Host์ค์
vi /etc/hosts
master
192.168.0.30 master.kopo
slave1
192.168.0.31 slave1.kopo
slave2
192.168.0.32 slave2.kopo
์์ ๋ฐ ์ถ๊ฐ
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
vi /etc/hosts
24.
๋คํธ์ํฌ ์ค์
๏ต Hostname ์ค์
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=master.kopo
์์ ๋ฐ ์ถ๊ฐ
--๋๋จธ์ง VM๋ ๋ณ๊ฒฝ--
์์)
NETWORKING=yes
HOSTNAME=slave1.kopo
[root@localhost ~]# hostnamectl set-hostname master.kopo
source : โHadoop ecosystemโ presented by prof. HyoKwan Kim at dept. of smart finance, Korea Polytechnics
wget http://apache.mirror.cdnetworks.com/kafka/1.0.0/kafka_2.11-1.0.0.tgz
tar zxfkafka_2.11-1.0.0.tgz
ln -s kafka_2.11-1.0.0 kafka
scp kafka_2.11-1.0.0.tgz root@slave1:/usr/local
scp kafka_2.11-1.0.0.tgz root@slave2:/usr/local
vi /usr/local/kafka/config/server.properties
์นดํ์นด ์ค์น
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# A comma seperated list of directories under which to store log files
log.dirs=/kdata1,/kdata2
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=master:2181,slave1:2181,slave2:2181/kopo-kafka
slave1 : 2
slave2 : 3
mkdir /kdata1
mkdir /kdata2
์ค ์ด์์ ๋ณ๋ ํํฐ์ ์ผ๋ก ์ค์
for more information:
https://kafka.apacheaorg/documentation/#brokerconfigs
* ์ฃผํคํผ๋ ธ๋๋ช ๋ฏธ์์ฑๆ
์ฃผํคํผ ๋ฃจํธ๋ ธ๋์ ์ ์ฅ
์ด ์์ ์์ kopo-kafka๋ก ์ค์ ํ์์
์ปจ์๋จธ ๋ฆฌ์คํธ ๋ช ๋ น์ด
--bootstrap: ๋ธ๋ก์ปค๋ฆฌ์คํธ ์ค์
--list : ์ปจ์๋จธ ๋ฆฌ์คํธ๋ฅผ ๋ณด์ฌ์ค.
./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:092,slave2:9092 --list
Kafka Consumer
[root@master bin]# ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:092,slave2:9092 --list
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=
Note: This will not show information about old Zookeeper-based consumers.
[2019-05-15 23:03:11,612] WARN Connection to node -2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kopo-consumer
71.
Kafka Consumer
์ปจ์๋จธ์ ์์ธ์ ๋ณด๋ฅผ ์กฐํํจ
--bootstrap-server : ๋ธ๋ก์ปค ๋ฆฌ์คํธ ์ ๋ ฅ
--group : ์ปจ์๋จธ ๊ทธ๋ฃน ์ด๋ฆ ์ค์
--describe : ์์ธ์ ๋ณด ๋ณด๊ธฐ
*LAG : ํ์ฌํ ํฝ์ ์ ์ฅ๋ ๋ฉ์์ง์ ์ปจ์๋จธ๊ฐ ๊ฐ์ ธ๊ฐ ๋ฉ์์ง์ ์ฐจ์ด
ํ๋ก๋์์ ์ปจ์๋จธ์ ์ฒ๋ฆฌ ์๋์ฐจ์ด๋ฅผ ์ ์ ์์
./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --group kopo-consumer --describe
[root@master bin]# ./kafka-consumer-groups.sh --bootstrap-server master:9092,slave1:9092,slave2:9092 --group kopo-consumer --describe
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Note: This will not show information about old Zookeeper-based consumers.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test-topic 0 21 21 0 kafka-python-1.4.3-3d11d79a-04f0-4da5-a1b9-808c34f99865/192.168.0.31 kafka-python-1.4.3
kopo-topic 0 425 426 1 - - -
[root@master bin]#
How to findZookeeper Leader in
zookeeper ensemble?
To identify a Zookeeper leader/follower, there are few possible options. Mentioning 2 for keeping this document simple.
1. Check the zookeeper log file on each node, and grep as below:
# grep LEAD /var/log/zookeeper/zookeeper-zookeeper-server-xyz.out
2016-02-29 22:33:47,113 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer@829] - LEADING
2016-02-29 22:33:47,114 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:Leader@358] - LEADING - LEADER ELECTION TOOK - 9066
2. Use "nc" command to listen TCP communication on port 2181 and determine if the ZooKeeper server is a leader or a follower.
Bellow is a way to check the same:
# echo stat | nc localhost 2181
Zookeeper version: 3.4.6-3485--1, built on 12/16/2015 02:35 GMT
Clients:
/172.25.16.243:37619[1](queued=0,recved=54,sent=54)
/172.25.16.253:58745[1](queued=0,recved=53,sent=53)
Latency min/avg/max: 0/1/362
Received: 627
Sent: 626
Connections: 9
Outstanding: 0
Zxid: 0x400000003
Mode: leader
Node count: 182
Here, we see that the Mode is shown as "leader". If it is a follower, the Mode will be seen as "follower".
For example:
# echo stat | nc localhost 2181 | grep Mode
Mode: leader
# echo stat | nc localhost 2181 | grep Mode
Mode: follower
source : https://community.hortonworks.com/content/supportkb/49435/how-to-find-zookeeper-leader-in-zookeeper-ensemble.html
77.
Why Kafka isso fast?
- page cache
source : https://www.slideshare.net/baniuyao/kafka-24299168