K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jaeseok Yoo
K8s, Amazon EKS

Time
9:30 - 10:30 Docker & Container Orchestration, k8s
10:30 – 10:45 Beak
10:45 - 12:00 K8s, Amazon EKS
HoL: Launch EKS Cluster
12:00 – 13:00 Launch
13:00 – 13:40 HoL: Launch microservices
13:40 – 14:20 HoL: Helm
15:15 – 16:00 HoL: Monitoring with Prometheus and Grafana

Docker

애플리케이션의 구성
런 타임 엔진 코드
디펜던시 구성

• 다른 애플리케이션 스택
• 다른 하드웨어 배포 환경
• 다른 환경에서 애플리케이션을
실행하는 효율적인 방법은?
• 다른 환경으로 쉽게
마이그레이션하는 방법은?
문제점

솔루션 - 도커
이식성 : 이미지 기반 배포
유연성 : 마이크로 서비스 모듈화
신속성 : 가벼운 도커 이미지
효율성 : OS kernel 공유

VM과 컨테이너 비교
Server (Host)
Host OS
Hypervisor
App 2
Guest OS Guest OS Guest OS
Bins/Libs Bins/Libs Bins/Libs
App 1 App 3
VM
Server (Host)
Host OS
Docker
Bins/Libs Bins/Libs Bins/Libs
App 1 App 2 App 3
Container
Hypervisor
Guest OS

Docker 이미지 구성
bootfs
kernel
Base image
Image
Image
W
ritable
Container
add
nginx
add
nodejs
U
buntu
References
parent
image
Base Image : 템플릿으로 사용되는
읽기 전용 이미지
Base Image에서 시작해서 커스텀
Image 추가하는 방식
Dockerfile 활용하여 손쉽게 배포 관련
구성 설정 및 재배포에 용이함

Docker 엔진 구조 및 Docker CLI 예
• docker build # Build an image from a Dockerfile
• docker info # Display system-wide information
• docker images # List all images on a Docker host
• docker run # Run an image
• docker ps # List all running and stopped instances
• docker stop # Stop a running instances
• docker rm # Remove an instance
• docker rmi # Remove an image
• docker pull # Download an image from registry
• docker push # Upload an image to the registry

Dockerfile은 이미지를 빌드하기 위한
일련의 명령어 모음

Dockerfile
# our base image
FROM alpine:3.5
# Install python and pip
RUN apk add --update py2-pip
# install Python modules needed by the Python app
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r
/usr/src/app/requirements.txt
# copy files required for the app to run
COPY app.py /usr/src/app/
COPY templates/index.html /usr/src/app/templates/
# tell the port number the container should expose
EXPOSE 5000
# run the application
CMD ["python", "/usr/src/app/app.py"]
$ docker build -t <YOUR_USERNAME>/myfirstapp .
Sending build context to Docker daemon 9.728 kB
Step 1 : FROM alpine:latest
---> 0d81fc72e790
Step 2 : RUN apk add --update py-pip
---> 976a232ac4ad
Removing intermediate container 8abd4091b5f5
Step 3 : COPY requirements.txt /usr/src/app/
---> 65b4be05340c
Step 4 : RUN pip install --no-cache-dir -r
/usr/src/app/requirements.txt
---> 8de73b0730c2
Step 5 : COPY app.py /usr/src/app/
…
Dockerfile은 컨테이너 내부 이미지 환경 및 구성 정의

Dockerfile best practice - 딱 필요한 Base 파일 선택
From the stock ubuntu image:
ubuntu latest 2b1dc137b502 52 seconds ago 458 MB
From python:2.7-alpine:
alpine latest d3145c9ba1fa 2 minutes ago 86.8 MB

FROM ubuntu:latest
RUN apt-get update -y && apt-get install -y python-pip python-dev build-essential
LABEL maintainer changsul@amazon.com
COPY . /app
WORKDIR /app
RUN pip install r requirements.txt
EXPOSE 5000
ENTRYPOINT ["python"]
CMD ["application.py"]

FROM python:2.7-alpine
LABEL maintainer changsul@amazon.com
COPY . /app
WORKDIR /app
RUN pip install r requirements.txt
EXPOSE 5000

FROM python:2.7-alpine
LABEL maintainer abbyfull@amazon.com
COPY requirements.txt /app
RUN pip install r /app/requirements.txt
COPY . /app
WORKDIR /app
EXPOSE 5000
Dockerfile best practice - 캐쉬 무효화 최소화

Dockerfile best practice
빌드 이미지 크기 및 레이어 수 최소화
런타임시 필요한 것만 선택
각 빌드별 태깅
Semantic version (i.e. “1.3.2-9”)
Build Number (i.e., “127”)
Build Id (i.e. “511d5e51-b415-4cb2-b229-b3c8a46b7a2f”)
템프 파일 제거
RUN apt-get update && apt-get install -y
bzr
cvs
git
mercurial
subversion
&& rm rf /var/lib/apt/lists/*

고객사례 - Nextdoor
Base OS version
Apt packages:
OpenSSL
libpq
syslog-ng
Datadog
Python runtime
PyPI packages:
Boto
Django
Mapnik
SendGrid
Source code
Static assets
Images
JS
CSS

Layer 별 각기 다른 업데이트 주기
Quarterly
Weekly/
monthly
Continuous

AMI에서 Docker Container로 변경
Base OS layer
System packages
Python packages
Nextdoor source

Docker 이전에는 빌드 20분 소요
chroot
sudo apt-get install
sudo pip install
git clone
make install
dpkg create

Base image , system deps 추가
FROM hub.corp.nextdoor.com/nextdoor/nd_base:precise
ADD app/docker/scripts/apt-fast
app/docker/scripts/system-deps.sh
/deps/
RUN /deps/system-deps.sh

Python virtualenv 설정 업데이트
ADD app/docker/scripts/venv-deps.sh
app/apps/nextdoor/etc/requirements*.txt
app/apps/nextdoor/etc/nextdoor.yml
app/services/scheduler/etc/scheduler.yml
app/services/supervisor/etc/supervisor.yml
/deps/
RUN /deps/venv-deps.sh

App 소스 업데이트
ADD app/static/nextdoorv2/images /app/static/nextdoorv2/images
ADD app/thrift /deps/thrift
ADD app/nd /deps/nd
ADD app /app

빌드 시간 20분 -> 평균 2분
ECS에 최종 배포까지 평균 5분

https://docs.docker.com/
https://en.wikipedia.org/wiki/Docker_(software)
https://en.wikipedia.org/wiki/LXC
https://en.wikipedia.org/wiki/Linux_namespaces
https://en.wikipedia.org/wiki/Cgroups
https://en.wikipedia.org/wiki/Chroot
https://www.slideshare.net/Docker/creating-effective-images-abby-fuller-aws
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
https://github.com/docker/labs/blob/master/beginner/chapters/webapps.md
http://crosbymichael.com/dockerfile-best-practices.html
References

Common Questions
• How do I deploy my containers to hosts?
• How do I do zero downtime or blue green deployments?
• How do I keep my containers alive?
• How can my containers talk to each other?
• Linking? Service Discovery?
• How can I configure my containers at runtime?
• What about secrets?
• How do I best optimize my "pool of compute”?

How do we make this work at scale?

We need to
• start, stop, and monitor lots of containers running on
lots of hosts
• decide when and where to start or stop containers
• control our hosts and monitor their status
• manage rollouts of new code (containers) to our hosts
• manage how traffic flows to containers and how
requests are routed

Container Orchestration
Instance Instance Instance
OS OS OS
Container Runtime Container Runtime Container Runtime
App Service App App Service Service

myJob: {
Cpu: 10
Mem: 256
}
Orchestrator
Schedule
Run “myJob”

Instance/OS Instance/OS Instance/OS
Service Management
Scheduling
Resource Management
OrchestrationService Management
§Availability
§Lifecycle
§Discovery

Service Management
Scheduling
Resource Management
Orchestration
Scheduling
§Placement
§Scaling
§Upgrades
§Rollbacks

Service Management
Scheduling
Resource Management
Orchestration
Resource Management
§ Memory
§ CPU
§ Ports

What are container orchestration tools?

Amazon Container Services Landscape
MANAGEMENT
Deployment, Scheduling,
Scaling & Management of
containerized applications
HOSTING
Where the containers run
Amazon Elastic
Container Service
Amazon Elastic
Container Service
for Kubernetes
Amazon EC2 AWS Fargate
IMAGE REGISTRY
Container Image
Repository
Amazon Elastic
Container Registry

Run a (managed) container on AWS
AMAZON CONTAINER SERVICES
Choose your orchestration tool1
Choose your launch type2
ECS EKS
EC2 Fargate EC2 Fargate

What is Kubernetes?
Open source container
management platform
Helps you run
containers at scale
Gives you primitives
for building
modern applications

Kubernetes(K8s) Components
Control Plane (Controller)
Etcd Lightweight, open source Key-Value store containing the cluster
API Server Serves the APIs required to manage the cluster
Scheduler Determines where (on which nodes) pods will run in the cluster
Controller Manager
The “worker on the controller” that actually manages the cluster
(e.g. replication)
Kubernetes Node
kubelet Runs the node, starts and stops containers
kube-proxy
Acts as a network proxy – routes traffic based upon IP and Port.
Each service is assigned a unique port on the nodes it runs across,
kube-proxy allows that port to be mapped to whatever the service
expects.
cAdvisor Agent that monitors node health and statistics

Kubernetes(K8s) Architecture

Kubernetes(K8s) Objects
• kubectl
• Pods
• Labels
• Deployments
• Replication Controllers
• Services

kubectl
• Command line interface for
running commands against the
k8s API
• Intuitive familiar commands
(get, create, describe, delete,
etc.) that are simple to learn and
easy to use
~/.kube/config
k8s master
kube-api
scheduler

Pods
• A group of one or more
containers
• Shared:
• Data volumes
• cgroup
• Namespace – network, IPC, etc. node
pod1 pod2

Labels
• Key/Value Pairs
• Used to query specific resources
within your cluster
pod1
pod2
dev
prod
app001
app001

ReplicaSets
• Ensure that a specified number
of pod “replicas” exist in the
cluster
23

Deployments
• Declarative updates for Pods
and ReplicaSets
23

Containers on Hosts
Host 1
Host 2
Host 3
A host is a server – e.g. EC2 virtual machine.
We run these hosts together as a cluster.
Web App
To start let’s run a 3 copies of our web
app across our cluster of EC2 hosts.
3x
Our simple example web application is
already containerized.
Cluster

Run n containers
Host 1
Host 2
Host 3
We define a deployment and set the replicas
to 3 for our container.
deploymentkubectl
rep = 3

Scale up!
Host 1
Host 2
Host 3
Need more containers?
Update the replication set!
deploymentkubectl
rep = 5
The new containers are started on the cluster.

Untimely termination
Host 1
Host 2
Host 3
Oh no! Our host has died!
Replication
set
rep = 5
Kubernetes notices only 3 of the 5
containers are running and starts 2
additional containers on the remaining
hosts.

Services
• A Kubernetes Service is an abstraction which defines a logical set
of Pods and a policy by which to access them - sometimes called a
micro-service. The set of Pods targeted by a Service is (usually)
determined by a Label Selector.
• Let’s talk about what are the differences between LoadBalancer,
NodePort and Ingress

Services : ClusterIP
• Exposes the service on a cluster-
internal IP
• Only reachable from within the
cluster
• Access possible via kube-proxy
• Useful for debugging services,
connecting from your laptop or
displaying internal dashboards

Services : NodePort
• Exposes the service on each Node’s IP
at a static port.
• Routes to a ClusterIP service, which is
automatically created.
• from outside the cluster:
<NodeIP>:<NodePort>
• 1 service per port
• Uses ports 30000-32767

Services : LoadBalancer
• Exposes the service externally using a
cloud provider’s load balancer.
• NodePort and ClusterIP services (to
which LB will route) automatically
created.
• Each service exposed with a
LoadBalancer (ELB or NLB) will get its
own IP address
• Exposes L4 (TCP) or L7 (HTTP)
services

Service : LoadBalancer - Sample
apiVersion: v1
kind: Service
metadata:
name: my-nginx-lb
labels:
app: nginx-lb
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx-lb
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx-lb
spec:
replicas: 3
template:
metadata:
labels:
app: nginx-lb
spec:
containers:
- name: nginx-lb
image: nginx:1.7.9
ports:
- containerPort: 80

Services
One of the ways traffic gets to your containers.
• Internal IP addresses are assigned to each container
• Services are connected to containers
and use labels to reference which containers
to route requests to
IP
IP
IP
Service
IP

Deployments
IP
IP
IP
Service
IPReplication set
version = 1
count = 3
Deployment
Services work with deployments to manage
updating or adding new pods.
Let’s say we want to deploy a new version of our
web app as a ‘canary’ and see how it handles
traffic.

Deployments
IP
IP
IP
Service
IPReplication set
version = 1
count = 3
The deployment creates a new replication set
for our new pod version.
Replication set
version = 2
count = 1
IP
Deployment

Deployments – Rolling Update
IP
IP
IP
Service
IPReplication set
version = 1
count = 3
Only after the new pod returns a healthy
status to the service do we add more new
pods and scale down the old.
Replication set
version = 2
count = 1
IP
Deployment
Replication set
version = 1
count = 0
Replication set
version = 2
count = 3

Deployments - Blue/Green
Service
app=nginx
Version=1
IP
Replication set
app=nginx
version=1
count=3
Deployment
Replication set
app=nginx
version=2
count=3
Deployment
Service
app=nginx
version=2

Deployments – Canary
Service
app=nginx
Version=1
IP
Replication set
app=nginx
version=1
count=3
Deployment
Replication set
app=nginx
version=2
count=1
Deployment
Service
app=nginx
Replication set
app=nginx
version=2
count=2
Replication set
app=nginx
version=1
count=2
Replication set
app=nginx
version=1
count=1
Replication set
app=nginx
version=2
count=3

Services : Ingress

Services : Ingress
• exposes HTTP/HTTPS
routes to services within
the cluster
• Many implementations:
ALB, Nginx, F5, HAProxy
etc
• Default Service Type:
ClusterIP

ALB Ingress Controller
AWS Resources
Kubernetes Cluster
Node Node
Kubernetes
API Server ALB Ingress
Controller
Node
HTTP ListenerHTTPS Listener
Rule: /cheesesRule: /charcuterie
TargetGroup:
Green (IP Mode)
TargetGroup:
Blue (Instance
Mode)
NodePort NodePort

ConfigMap and Secret

ConfigMap & Secret
• ConfigMap and Secret allow you to decouple
configuration artifacts from image content to keep
containerized applications portable.
• You can pass the ConfigMap or Secret to the pod by environment
variable or volume mount.
• Secret uses Base64 encoding.

StatefulSet

Statefulset Properties
• Network identifiers
• Persistent Storage
• Ordered graceful deployment and scaling
• Ordered graceful termination
• Ordered rolling updates
• If none of these fit your portfolio, use Deployment or Replicaset

StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request
StorageClass
gp2 io1 sc1 encrypted
io1
st1
3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Network
Identifiers
Ordered
Deployment

StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request
3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Ordered
Scaling
mysql-4

Storage

Lifecycle of a storage volume
Provisioning Binding Using Reclaiming
• Static
• Dynamic*
• Control loop watches
for PVC requests and
satisfies if PV is
available.
• For Dynamic, PVC
will provision PV
• PVC to PV binding is
one-to-one mapping
• Cluster mounts
volume based on
PVC
• Retain (default)
• Recycle
• Delete

Storage Class Persistent Volume Persistent Volume
Claim
Pod

What if I need specific volume type?
StorageClass
gp2 io1 sc1 encrypted
io1
st1
1) Admin pre-provisions
StorageClass based
on workload needs
2) End user requests for
specific volume types
(For ex, encrypted
io1 volume)
3) Control loop watches
PVC request and
allocates volume if
PV exists
MySQL Pods
4) End user creates
stateful workload

57%of Kubernetes workloads
run on AWS today
— Cloud Native Computing Foundation

Containers options on AWS – over time
Docker
Host
AWS Cloud
AWSmanagedCustomermanaged

Amazon ECS
EC2 Container
Instances
Auto Scaling group
2015
ECS API
Docker
Host
AWS Cloud

AWS Fargate
Amazon ECS
EC2 Container
Instances
Auto Scaling group
2017
ECS API
Docker
Host
AWS Cloud

AWS Fargate
Amazon ECS
EC2 Container
Instances
Auto Scaling group
Worker
nodes
Auto Scaling groupDIY K8S
ECS API
K8s API
Docker
Host
AWS Cloud

AWS Fargate
Amazon ECSAmazon EKS
EC2 Container
Instances
Auto Scaling group
Worker
nodes
2018
K8s API ECS API
K8s API
Docker
Host
AWS Cloud

Management of the
Kubernetes control plane
Phase 1

Management of the
Kubernetes control plane
Phase 1
Phase 2
Management of the
Kubernetes data plane

AWS Fargate
EC2 Container
Instances
Auto Scaling group
Managed
Node Groups
Auto Scaling group
Worker
nodes
2019
K8s API ECS API
K8s API
Docker
Host
AWS Cloud

AWS Fargate
EC2 Container
Instances
K8s API ECS API
AWS Cloud
Auto Scaling group
Managed
Node Groups
Auto Scaling group
Worker
nodes
NEW
Docker
Host
K8s API

Kubernetes on AWS
Managed Kubernetes on
AWS
Highly
available
Automated
version
upgrades
Integration
with other
AWS services
Etcd
Master
Managed
Kubernetes
control
plane CloudTrail,
CloudWatch, ELB,
IAM, VPC, PrivateLink

Kubernetes on AWS
3x Kubernetes masters for HA

Availability
Zone 1
Master Master
Availability
Zone 2
Availability
Zone 3
Master
Workers Workers Workers
Customer Account
AWS Managed

EKS Control Plane

EKS Architecture
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Exec, Logs,
Proxy
Internet

What happens when I run ‘kubectl create –f pods.yaml’?

IAM Authentication
Kubectl
3) Authorizes AWS Identity with RBAC
K8s API
1) Passes AWS Identity
2) Verifies AWS Identity
4) K8s action
allowed/denied
AWS Auth

Kubernetes Control Plane
Master Node
Scheduler
Controller
Manager
Cloud
Controller
Manager
API Server
etcd
Kubectl

Cluster Authentication and Authorization
• User or IAM role who creates EKS cluster gains Admin privileges
• This {“super”} user/role can then add additional users or IAM roles
and configure RBAC permissions
• To add, configure aws-auth Configmap
kubectl edit -n kube-system configmap/aws-auth

aws-auth configuration
apiVersion: v1
data:
mapRoles: |
- rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
mapUsers: |
- userarn: arn:aws:iam::555555555555:user/admin
username: admin
groups:
- system:masters
- userarn: arn:aws:iam::555555555555:user/john
username: john
groups:
- pod-admin # k8s RBAC group

EKS Data Plane

Kubernetes Data Plane
Worker Node
kube-dnsKubelet
aws-
node
Container runtime
Control Plane
API
kube-
proxy

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT
ExecStart=/usr/bin/kubelet --cloud-provider aws
--config /etc/kubernetes/kubelet/kubelet-config.json
--allow-privileged=true
--kubeconfig /var/lib/kubelet/kubeconfig
--container-runtime docker
--network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS
Restart=on-failure
RestartForceExitStatus=SIGPIPE
RestartSec=5
KillMode=process
[Install]
WantedBy=multi-user.target

EKS AMI Build Scripts
https://github.com/awslabs/amazon-eks-ami
Source of truth for EKS Optimized AMI
Easily build your own EKS AMI
Build assets for EKS AMI for each supported Kubernetes version

EKS Optimized AMI with GPU Support
Easily run Tensorflow/Kubeflow on Amazon EKS
Includes NVIDIA packages to support Amazon P2 and P3 instances
Available on AWS Marketplace

EKS Managed Node Group
• You can create, update, or terminate nodes for your cluster with a
single operation.
• Nodes run using the latest Amazon EKS-optimized AMIs in your
AWS account while node updates and terminations gracefully
drain nodes to ensure that your applications stay available.
• All managed nodes are provisioned as part of an Amazon EC2
Auto Scaling group that is managed for you by Amazon EKS.

Worker Node Setup – Bootstrapping
/etc/eks/bootstrap.sh <cluster-name> [options]
Uses UserData for configuring System resources and extra Kubelet
config
Reserve compute resources for System Daemons (Kubelet, Container
runtime) and Pod eviction thresholds

EKS Upgrades

Kubernetes Version
Versions supported: 1.12.10, 1.13.12, 1.14.9
EKS will support up to 3 versions of Kubernetes at once
”Deprecation” will prevent new cluster creation on old version

EKS Platform Version
Platform Version revisions represent API server configuration
changes or Kubernetes patches
Platform Versions increment within a Kubernetes version only

EKS Kubernetes Version Updates
New UpdateClusterVersion API –
supports in place updates of Kubernetes
version
Introduces an ”update” EKS API object
ListUpdates and DescribeUpdate APIs to
provide visibility into the status of a
given update

Updating Worker Nodes
Two options:
1) Create new node group with latest EKS AMI >> taint old nodes >>
drain old nodes >> terminate old CFN template
2) Simply update AMI in CFN template; “rolling” replacement policy
terminates nodes
(Downsides: un-graceful termination of applications)

Container Services Roadmap
https://github.com/aws/containers-roadmap

Get Started
https://eksworkshop.com
Modules:
• Health Checks
• Logging with Elasticsearch, Fluentd, and
Kibana (EFK)
• Monitoring using Prometheus and Grafana
• Servicemesh with Istio
• Stateful Containers using StatefulSets

Thank you!

K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트

More Related Content

What's hot

Similar to K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트

More from Amazon Web Services Korea

Recently uploaded

In this document

K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트