KEMBAR78
From CoreOS to Kubernetes and Concourse CI | PDF
From CoreOS
To Kubernetes
Denis Izmaylov
Feb 16, 2017
Denis Izmaylov
1. Over 17 years of Software and Web
development experience
2. The last 7 years focused exclusively on
Front-end and Web: 15+ projects,

including React.js, SPA and high-load
3. Open Source contributor
4. Talks: HighLoad++ 2015, AgileDays 2016,
DevConf 2016, React Amsterdam, etc
5. Regular speaker on MoscowJS
6. Few popular articles and interviews

on Medium and Habrahabr
7. Events: Moscow Node.js Meetup, React
Moscow Meetup and this one
Web and Mobile development services
1. We develop Web, Mobile
and DevOps projects for
our customers
2. Global international
market (EU, APAC, US, HK)
3. Most efficient and
performant technologies
4. More than 20 talents
engineers and designers
5. Remote-friendly
Prerequisite Knowledge
1. Linux
2. Docker basic
3. Kubernetes basic
4. Monolithic architecture
5. Microservices principles
6. YAML
Chapter 2
Watch the previous talk:
Moscow Node.js Meetup 8
at Yandex (2016, July 13)
https://events.yandex.ru/events/yagosti/13-jul-2016/
What problems can you get
with CoreOS?
Why Kubernetes is

not enough?
How to scale fast
and effortless?
11
Customer Goals
1. Production cluster on 3-5 machines
2. Be ready for fast growing (up to 10x)
3. Fault tolerance
4. Faster changes delivery
5. Full control for each service (in-house,
independent, no-SaaS)
6. Blue-Green (Zero Downtime) Deployment
7. Good DX (Developer Experience)
12
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS
Microservices
14
Monolithic Architecture
15
Problems
1. Slow
2. Expensive
3. Hard
4. High risks
5. Vendor lock-in
6. Low quality
Typical Monolith
16
17
What if…?
Each function

with dependencies
18
What if…?
Each function

with dependencies
19
What if…?
20
Microservices Architecture
Microservices Architecture
Single Responsibility Principle
21
22
Problems
1. Slow
2. Expensive
3. Hard
4. High risks
5. Vendor lock-in
6. Low quality
1. Fast
2. Cheap
3. Easy
4. Low risks
5. Vendor-less
6. Cost/quality
Solved
23
Effects
1. Resistance
• Fail-over
• Isolated failures
2. Delivery speed
3. Changes are simple
4. Freedom
5. Security
www.reactivemanifesto.org
25
“Microservices allow engineering teams
to move quickly to grow a product…
assuming they don’t get bogged down
by the complexity of operating a
distributed system”
26
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS
27
Blue-Green Deployment
28
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS
Scale Cube
30
Scale Cube
• X-axis: by cloning Services
• Y-axis: by functional decomposition
• Z-axis: by sharding databases
31
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS
32
33
1. First Container-optimized OS: 3+ years, since July 2013
2. Mini OS: based on ChromeOS, small and fast
3. Security-focused: by using auto-updates (like
browsers)
4. No package manager: each application is running as

a Container on top of OS
5. Built-in support: Containers, Container networking,
Service discovery
6. Services: runs through systemd
7. Configuration: one Cloud Config file
34
Components
1. etcd
2. fleet
3. flannel
* state on March, 2016
35
etcd
Distributed key-value storage:
1. Common

configuration
2. Service

Discovery
3. Resource

locking
36
fleet
Distributed systemd
37
flannel
Overlay network for your containers
38
39
Cloud Config
40
test-app@.service
41
Deploy & Destroy
42
Summary
1. CoreOS and Docker: you don’t think
about OS anymore
2. Containers space
3. Now a server is just a core of your“CPU”
4. Auto-schedule: you don’t have to know
where is your Container is running
5. Configuration-less
6. Amazing expandable
Breaking News
45
Dec 12, 2016
https://coreos.com/blog/tectonic-self-
driving.html#coreos-linux-is-now-container-
linux
Moving from fleet
to Kubernetes
Today
51
Solution
1. Microservices
2. Blue-Green (Zero Downtime)
Deployment
3. Scale Cube
4. CoreOS
52
Customer Goals
Production cluster on 3-5 machines
Be ready for fast growing (up to 10x)
Fault tolerance
Faster changes delivery
Full control for each service (in-house,
independent, no-SaaS)
Blue-Green (Zero Downtime) Deployment
Good DX (Developer Experience)
Wake up!
55
Real Life
1. Launch a new containers
2. Integrate containers between each
other
3. Connect containers with Load Balancers
4. Rolling updates for the launched
containers
5. Use Blue-Green Deployment strategy
56
Simple Steps
1. Create Fleet unit file for each service
2. Each service requires Service Discovery
3. Sidekick model: create a Discovery Unit
4. Rolling updates: small set of commands
5. BGD deployment: small script
Dive Deeper
58
Project Profile
1. Isomorphic React.js Application
2. RESTful API on Node.js
3. Serve Static service (nginx)
4. Golang service
5. C++ service
6. PHP service
7. JSON Configuration service
59
Go Ahead
• Fleet unit for each service
• Discovery Unit for each service
• X-scaling: multiple instances
• Environments (dev, stage, prod)
60
Go Ahead
• Fleet unit for each service
• Discovery Unit for each service
• X-scaling: multiple instances
• Environments (dev, stage, prod)
6
2
6
3
61
Go Ahead
6
2
6
3
x
x
x
62
Go Ahead
6
2
6
3
x
x
x
216 containers
216 containers
Very interesting
I can’t configure CoreOS anymore
66
Real Summary
1. CoreOS is OS only
2. Not enough for microservices
management
3. Good enough as a platform of other
tools (PaaS, SaaS, etc)
4. Configure CoreOS Cluster Step-by-Step:
https://github.com/axept/devops-open
From CoreOS
68
Next Level Requirements
1. Microservices Management
2. Continuous Delivery / Deployment
3. SSL support
4. TLD support
5. Monitoring
6. Alerting and Notifications
May 2016
71
Kubernetes
1. Open-Source Cluster Management
2. Originally developed by Google
3. Open-sourced in June 2014
4. Deploy and manage applications
declaratively
5. Declarative: state your desired results, let
the system actuate
6. Control loops: observe, rectify, repeat
7. No groups: labels is more powerful
Kubernetes is greek for“helmsmen”
– How can we spend more time
building useful services
and less time maintaining the platform?
74
Some Concepts
1. Controller: drives current state to target
state
2. Pod: a small group of tightly-integrated
Containers
3. Label: identifying metadata attached to
other objects (e.g. to Pods)
4. Service: exposes a name, port and IP for a
groups of Pods
5. Ingress: connects Load Balancers with
Services by using URL rules
75
Controller
1. Run on Master

continuously
2. Each K8s object gets

its own Controller
3. Control Loop:
• get state
• calculate diff
• apply changes
4. Pluggable and lightweight
Get state
from etcd
Compare
with
worker’s
state
Actuate it
76
Pods
1. Group of Containers
2. One or multiply containers
3. Has its own IP and localhost
4. Scale together
5. Shared secrets
6. Shared volumes
7. Shared resources (CPU and RAM)
8. Can be tagged with labels
77
Label
1. Identifying metadata
2. Can be attached to any API
object
3. Generally represents identity
4. It could be application name,
role, phase, environment,
version, etc
5. Using by Selectors
6. The grouping mechanism only
78
Example for Labels
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
79
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app = demo
80
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app=demo

role=FE
81
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app=demo

role=BE
82
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app = demo
phase = dev
83
Selectors
Think about it like SQL:“select … where …”
app: demo
phase: dev
role: FE
app: demo
phase: dev
role: BE
app: demo
phase: test
role: FE
app: demo
phase: test
role: BE
app = demo
phase = test
84
Services
1. Serve as a kind of

Load Balancers for Pods
2. Expose name, port and

stable IP for a group of Pods
3. IP can be internal or public
4. Can implement Service Discovery

between applications and features

to support Zero-downtime deployments
85
Ingress
1. Works with Load
Balancers, Cloud LB,
nginx, etc
2. Provides a single root
URL to multiple services
3. Publicly expose private
networks
4. Can use HTTP Headers
5. Supports SSL
And a lot of other things
Explore

Feature Tracking and Backlog:
https://github.com/kubernetes/features
88
In next talk
1. How to configure Concourse CI

with Kubernetes properly?
2. How to implement TLD (Top-Level Domains,
e.g. google.de, yahoo.eu, yandex.ru)?
3. How to configure persistent SSL certificates
and temporary SSL certificates by
Letsencrypt?
4. How to configure monitoring and alerting
using Grafana?
5. How to install the cluster in a one click?
In next talk
90
Summary
1. Kubernetes saved us from“1000 Fleet Units”
hell in CoreOS and a decades of custom
scripts
2. K8s saved us a lot of months
3. Some other interesting features:
• Role-based Access Control
• AWS, GCE, Azure support
• Apache License Version 2.0
91
What We Got
1. Micro-services Management
2. Continuous Delivery / Deployment
3. Easy SSL and TLD management
4. Monitoring
5. Notifications
92
Achievements
1. Production cluster on 3-5 machines
2. Be ready for fast growing (up to 10x)
3. Fault tolerance
4. Faster changes delivery
5. Full control for each service (in-house,
independent, no-SaaS)
6. Blue-Green (Zero Downtime) Deployment
7. Great DX (Developer Experience)
That’s not enough!
We developed
One-Click Installer
95
One-Click Installer
1. Uses Ansible
2. Prepare Configuration directory

with keys, vars, secrets, etc
3. Optional - prepare network interfaces on target
machines
4. Configure servers list and SSH credentials
5. Launch the script
6. Get a new cluster in 3 minutes with

isolated CI and Private Docker Registry
96
Effects
1. CoreOS, Kubernetes and Application became

an isolated node
2. Micro-clusters Architecture
98
Effects
1. CoreOS, Kubernetes and Application became

an isolated node
2. Micro-clusters Architecture
3. Possible to get Cross-cluster Load Balancing and
Service Discovery
4. Automated Installation
5. 24/7 Support friendly
6. No“Black box”inside
7. Efficient Geo-scalability
99
Take aways
1. CoreOS is not about microservices management
2. CoreOS is just Container OS (Container Linux!)
3. CoreOS is good as a tool for SaaS, PaaS, etc
4. Kubernetes is really great for microservices
management and large clusters
5. Kubernetes is the new operation system (OS)
6. Some interesting things will be in next talk
“Microservices allow engineering teams
to move quickly to grow a product…
assuming they don’t get bogged down
by the complexity of operating a
distributed system”
7.
101
Read (1 of 2)
1. Kubernetes The Hard Way

https://github.com/kelseyhightower/kubernetes-the-
hard-way
2. Design Patterns for Container-based Distributed
Systems by Brendan Burns and David Oppenheimer,
Google

https://www.usenix.org/node/196347
3. Site Reliability Engineering: How Google Runs
Production Systems

http://shop.oreilly.com/product/0636920041528.do
102
Read (2 of 2)
4. Kubernetes Cluster Federation (previously "Ubernetes")

https://github.com/kubernetes/kubernetes/blob/
release-1.5/docs/design/federated-services.md
5. From Google to the world: the Kubernetes origin story

https://cloudplatform.googleblog.com/2016/07/from-
Google-to-the-world-the-Kubernetes-origin-story.html
6. Minikube is a tool that makes it easy to run Kubernetes
locally. Minikube runs a single-node Kubernetes cluster
inside a VM on your laptop.

https://kubernetes.io/docs/getting-started-guides/
minikube/
103
Observe
1. https://twitter.com/kubeconio
2. https://twitter.com/kubernetesio
3. https://twitter.com/kelseyhightower
4. https://twitter.com/kubernetesnyc
5. https://twitter.com/kubernetesber
6. https://github.com/kubernetes-incubator
7. https://twitter.com/kubernetesmsk
Welcome to the club
Questions?
@DenisIzmaylov
https://github.com/DenisIzmaylov
denis_izmaylov
Denis Izmaylov
Send“Hello”to:
izmaylov.dm@gmail.com

From CoreOS to Kubernetes and Concourse CI