KEMBAR78
[Demo session] 관리형 Kafka 서비스 - Oracle Event Hub Service | PDF
1
Managed	Kafka	Service
Oracle	Event	Hub	CS
6th Oracle	Developer	Meetup
2
+ =
Managed	Kafka	Service
Oracle	Event	Hub	CS
3
About:	Me
• 1999	~	2006	Java	(SI,	Developer)
• 2007	~	2013	Middleware	(Oracle,	SC)
– WAS,	Memory	Cache,	CEP,	DB	Replication
• 2013	~	2014	NoSQL,	CouchDB (SE)
• 2015	~	Big	Data,	ML,	Deep	Learning(SC)
민수
수학
Analysis
컨텐츠
4
Compute
Networking
Storage
Load	Balancing
Edge	Services
Container
Container	Pipline
FastConnect
Ravello
Compute	Classic
Networking	Classic
Storage	Classic
Infrastructure	Service Platform	Service
Database
Database	Backup
Big	Data
Big	Data	Cloud
Event	Hub
MySQL
Autonomous	NoSQL	Database	Cloud
Data	Hub
Autonomous	Data	Warehouse	Cloud
Data	Management Application	Development
Java
Application	Container
Mobile	and	Chatbots
Autonomous	Visual	Builder	Cloud
AI	Platform
Blockchain
Developer
Messaging
API	Catalog
Autonomous	Integration	Cloud
Oracle	Integration	Cloud
Data	Integration
Internet	of	Things
API	Management
Self-Service	Integration
Self-Service	Integration
SOA
Integration
Process	Automation
Application	Performance	Monitoring
Infrastructure	Monitoring
Log	Analytics
Orchestration
IT	Analytics
Management
Content and Experience
Content	and	Experience
WebCenter	Portal	Cloud
DIVA	Cloud
Business	Analytics
Autonomous	Analytics	Cloud
Analytics	Cloud
Business	Intelligence
Data	Visualization
Essbase
Security
CASB
Identity
Configuration	and	Compliance
Security	Monitoring	and	Analytics
5
Compute
Networking
Storage
Load	Balancing
Edge	Services
Container
Container	Pipline
FastConnect
Ravello
Compute	Classic
Networking	Classic
Storage	Classic
Infrastructure	Service Platform	Service
Database
Database	Backup
Big	Data
Big	Data	Cloud
Event	Hub
MySQL
Autonomous	NoSQL	Database	Cloud
Data	Hub
Autonomous	Data	Warehouse	Cloud
Data	Management Application	Development
Java
Application	Container
Mobile	and	Chatbots
Autonomous	Visual	Builder	Cloud
AI	Platform
Blockchain
Developer
Messaging
API	Catalog
Autonomous	Integration	Cloud
Oracle	Integration	Cloud
Data	Integration
Internet	of	Things
API	Management
Self-Service	Integration
Self-Service	Integration
SOA
Integration
Process	Automation
Application	Performance	Monitoring
Infrastructure	Monitoring
Log	Analytics
Orchestration
IT	Analytics
Management
Content and Experience
Content	and	Experience
WebCenter	Portal	Cloud
DIVA	Cloud
Business	Analytics
Autonomous	Analytics	Cloud
Analytics	Cloud
Business	Intelligence
Data	Visualization
Essbase
Security
CASB
Identity
Configuration	and	Compliance
Security	Monitoring	and	Analytics
6
Compute
Networking
Storage
Load	Balancing
Edge	Services
Container
Container	Pipline
FastConnect
Ravello
Compute	Classic
Networking	Classic
Storage	Classic
Infrastructure	Service Platform	Service
Database
Database	Backup
Big	Data
Big	Data	Cloud
Event	Hub
MySQL
Autonomous	NoSQL	Database	Cloud
Data	Hub
Autonomous	Data	Warehouse	Cloud
Data	Management Application	Development
Java
Application	Container
Mobile	and	Chatbots
Autonomous	Visual	Builder	Cloud
AI	Platform
Blockchain
Developer
Messaging
API	Catalog
Autonomous	Integration	Cloud
Oracle	Integration	Cloud
Data	Integration
Internet	of	Things
API	Management
Self-Service	Integration
Self-Service	Integration
SOA
Integration
Process	Automation
Application	Performance	Monitoring
Infrastructure	Monitoring
Log	Analytics
Orchestration
IT	Analytics
Management
Content and Experience
Content	and	Experience
WebCenter	Portal	Cloud
DIVA	Cloud
Business	Analytics
Autonomous	Analytics	Cloud
Analytics	Cloud
Business	Intelligence
Data	Visualization
Essbase
Security
CASB
Identity
Configuration	and	Compliance
Security	Monitoring	and	Analytics
7
https://www.youtube.com/watch?v=I32hmY4diFY&t=1125s
ETL	Is	Dead,	Long	Live	Streams:	
real-time	streams	w/	Apache	Kafka
8
Apps Apps Apps
NoSQL
Search
RDBMS
Monitoring
RT
Analysis
DWH
Hadoop
https://www.youtube.com/watch?v=I32hmY4diFY&t=1125s
ETL	Is	Dead,	Long	Live	Streams:	
real-time	streams	w/	Apache	Kafka
9
Apps Apps Apps
NoSQL
Search
RDBMS
Monitoring
RT
Analysis
DWH
Hadoop
https://www.youtube.com/watch?v=I32hmY4diFY&t=1125s
ETL	Is	Dead,	Long	Live	Streams:	
real-time	streams	w/	Apache	Kafka
10
Producers
Consumers
11
Producers
Consumers
Stream	
APIs
12
Producers
Stream	
APIsConnectors
Consumers
13
나의 아저씨
14
나의 카프카!
1
https://youtu.be/I32hmY4diFY?t=500
15
나의 카프카!
Databus (Lambda/Kappa)
Data	Vault	Model	2.0MSA Big	Data
16
In	Oracle	Cloud
Self	Managed Platform	Managed
Apache	Kafka	on	IAAS	Compute Event	Hub	Cloud	Service	(Dedicated) Event	Hub	Cloud	Service	(Topics)
$0.06	OCPU/hour
(69원)
$0.27	OCPU/hour
(290원)
17
Step	01:	클러스터 생성
18
19
20
2018. 5. 17 오전 11시 35분 33초 UTC Activity Submitted
2018. 5. 17 오전 11시 35분 33초 UTC Activity Started
2018. 5. 17 오전 11시 35분 33초 UTC Started operation to create service [demokafka] in identity domain [idcs-c69311285cdb4062b8455d9a1fcb18c8].
2018. 5. 17 오전 11시 42분 32초 UTC Completed creating service [demokafka] resources in domain [idcs-c69311285cdb4062b8455d9a1fcb18c8].
2018. 5. 17 오전 11시 42분 37초 UTC Started operation to establish security on VMs for component [connect].
2018. 5. 17 오전 11시 42분 38초 UTC Started operation to establish security on VMs for component [restprxy].
2018. 5. 17 오전 11시 42분 38초 UTC Completed operation to establish security on VMs for component [restprxy]
2018. 5. 17 오전 11시 42분 47초 UTC Started operation to establish security on VMs for component [kafka].
2018. 5. 17 오전 11시 42분 48초 UTC Started operation to check provisioning status on the VMs for [kafka]
2018. 5. 17 오전 11시 42분 48초 UTC Started operation to check provisioning status on the VMs for [restprxy]
2018. 5. 17 오전 11시 42분 48초 UTC Started operation to check provisioning status on the VMs for [connect]
2018. 5. 17 오전 11시 44분 00초 UTC Provisioning of component [restprxy] succeeded.
2018. 5. 17 오전 11시 44분 00초 UTC Provisioning of component [kafka] succeeded.
2018. 5. 17 오전 11시 44분 00초 UTC Provisioning of component [connect] succeeded.
2018. 5. 17 오전 11시 44분 00초 UTC Started service reachability check operation for [restprxy].
2018. 5. 17 오전 11시 44분 00초 UTC Started service reachability check operation for [kafka].
2018. 5. 17 오전 11시 44분 00초 UTC Started service reachability check operation for [connect].
2018. 5. 17 오전 11시 44분 07초 UTC Successfully reached demokafka-connect-1 at port 8083. Successfully reached demokafka-connect-1 at port 1080.
2018. 5. 17 오전 11시 44분 07초 UTC Successfully completed service reachability check for [connect]
EHCS	인스턴스 생성 로그:
5	node,	11분 소요
21
22
Oracle	Event	Hub	Service	아키텍처
OCI	Classic	
OCI
23
OEHCS	클러스터| REST	Proxy
REST	Proxy
AVRO	Registry
Monitoring	App
REST	Proxy
AVRO Registry
Monitoring	App
REST	Proxy
AVRO	Registry
Monitoring	App
User
1080
Kafka	
brokers
Kafka	
brokersKafka	
brokers
• 포트 외부 오픈
• Zookeeper	2181	&	JMX:	9098
• Kafka	6667	&	JMX:	9097
Zookeeper
Zookeeper
Zookeeper
Nimbula Sec	List	- Kafka
6667
24
25
OEHCS	클러스터| Native	API
Trusted	
source	IP	
with	in	tenancy
6667Kafka	
brokers
Kafka	
brokers
Kafka	
brokers
Zookeeper
ZookeeperZookeeper
OPC		
Trusted	
VM
26
Step	02:	Topic	관리
27
28
29
Step	02-1:	Topic	수정
30
Step	02-1:	Topic	수정
31
Step	02-2:	Topic	확장
32
Step	02-2:	Topic	확장
33
Step	02-2:	Topic	확장
34
Step	02-3:	Topic	모니터링
35
36
Step	03:	Cluster	확장
37
Step	03:	Cluster	확장
38
Oracle	PaaS	Service	Manager
Oracle	Cloud	Infrastructure
REST	
API
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
PaaS
REST	
API
REST	
API
REST	
API
REST	
API
REST	
API
REST	
API
REST	
API
REST	
API
REST	
API
REST	
API
REST	
API
OCI	CLI
Compute Network Storage DNS Load	Balancing Security
Web	UI Chef Terraform Ansible
PaaS	Service	Manager(PSM)
Oracle PaaS Service Manager 소개 및 설치: http://taewan.kim/post/installing_pms/
39
PaaS	Service	Manager
https://docs.oracle.com/en/cloud/paas/java-cloud/pscli/oehcs-create-service.html
40
Oracle	PaaS	Service	Manager
$ sudo -H psm oehpcs help
DESCRIPTION
Oracle Event Hub Cloud Service - Dedicated
SYNOPSIS
psm OEHPCS <command> [parameters]
AVAILABLE COMMANDS
o services o service
o create-serviceo delete-service
o restart-service o scale-out
o stop o start
o restart o add-storage
o scale o available-patches
o applied-patches
o patch o precheck-patch
o rollback
o check-health
o operation-status o activities
o access-rules o create-access-rule
o delete-access-rule o enable-access-rule
o disable-access-rule o add-ssh-public-key
o help
41
Oracle	Cloud	SDK
42
Flume
아파치 플룸(Apache	Flume)	:	http://taewan.kim/post/flume_images/
43
Flafka =	Flume	+	Kafka
44
https://www.confluent.io/product/connectors/
- Scalable	Framework
- Open	Source
- Only	Configuration
45
Step04:	OEHCS의 Kafka	Connector	지원
46
OEHCS의 Kafka	Connector	지원
47
OEHCS의 Kafka	Connector	지원 – OCI	Object	Storage
48
OEHCS의 Kafka	Connector	지원 – JDBC	Sink
49
OEHCS	Roadmap
- Connector
- Sink
- Oracle	DataHub:	Cassandra
- Apache	Hbase
- S3
- Elastic	Search
- Source
- Twitter
- Management
- Auto	Scale-out
- Multi	Datacenter
- Audit
- Disk	Storage	암호화
- PCI	인준
50
Step05:	Extension
51
Step06:	Patch
http://taewan.kim/post/oracloud_patch/
52
Oracle	Integrations
Confidential	– Oracle	 5
Oracle	Application	Container	Cloud	ServiceOracle	Storage	Cloud	Service
Oracle	Database	Cloud	Service Oracle	Java	Cloud	Service
Oracle	Golden	Gate	Cloud	Service
Oracle	Big	Data	Cloud	- CE
Oracle	IOT	Cloud	Service
Oracle	Integration	Cloud	Service
Oracle	Analytics	Cloud	Service
53
Oracle	Cloud	– Data	Lake
Storage	Cloud	Service	(Data	Lake)
Web
Crawler
Spark	
Streaming
Event	
hub
GPU
Deep	
Learning
Identity	Domain
• Upload	CLI
• OSACS	(비동기 복제)
BigData
Connector
SDK
REST	API
Library
IaaS
Computing
Data	Processing
Machine	Learning
ADWC OAC
IaaS
Computing
On-Premise
Server
Oracle	Software	Appliance	Cloud	Service
Visualization
54
Oracle	Trial
http://taewan.kim/post/oracle_cloud_trial/
55
실습 가이드
• Eventhub 소개
– http://taewan.kim/post/kafka_paas_eve
nthub_cs/
• Eventhub
– https://github.com/oracle/learning-
library/blob/master/workshops/journey
2-new-data-lake/LabGuide400.md
• Eventhub +	Spark	Streaming
– https://github.com/oracle/learning-
library/blob/master/workshops/journey
2-new-data-lake/LabGuide500.md
56

[Demo session] 관리형 Kafka 서비스 - Oracle Event Hub Service