Amazon Container Services – 유재석 (AWS 솔루션즈 아키텍트)

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jaeseok Yoo
Container, Container, Container …

Time
13:00 – 13:15 Docker & Container Orchestration
13:15 – 14:00 Kubernetes & Amazon EKS
14:00 – 14:15 HoL: Launch EKS Cluster
14:15 – 14:30 Break
14:30 – 15:45 HoL: Deploy Dashboard, Microservices, Logging
16:00 – 16:45 Amazon ECS & Dedicated Server on Container
16:45 – 17:00 Clean Up

Docker

애플리케이션의 구성
런 타임 엔진 코드
디펜던시 구성

• 다른 애플리케이션 스택
• 다른 하드웨어 배포 환경
• 다른 환경에서 애플리케이션을
실행하는 효율적인 방법은?
• 다른 환경으로 쉽게
마이그레이션하는 방법은?
문제점

솔루션 - 도커
이식성 : 이미지 기반 배포
유연성 : 마이크로 서비스 모듈화
신속성 : 가벼운 도커 이미지
효율성 : OS kernel 공유

VM과 컨테이너 비교
Server (Host)
Host OS
Hypervisor
App 2
Guest OS Guest OS Guest OS
Bins/Libs Bins/Libs Bins/Libs
App 1 App 3
VM
Server (Host)
Host OS
Docker
Bins/Libs Bins/Libs Bins/Libs
App 1 App 2 App 3
Container
Hypervisor
Guest OS

Docker 이미지 구성
bootfs
kernel
Base image
Image
Image
W
ritable
Container
add
nginx
add
nodejs
U
buntu
References
parent
image
Base Image : 템플릿으로 사용되는
읽기 전용 이미지
Base Image에서 시작해서 커스텀
Image 추가하는 방식
Dockerfile 활용하여 손쉽게 배포 관련
구성 설정 및 재배포에 용이함

Dockerfile
# our base image
FROM alpine:3.5
# Install python and pip
RUN apk add --update py2-pip
# install Python modules needed by the Python app
COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r
/usr/src/app/requirements.txt
# copy files required for the app to run
COPY app.py /usr/src/app/
COPY templates/index.html /usr/src/app/templates/
# tell the port number the container should expose
EXPOSE 5000
# run the application
CMD ["python", "/usr/src/app/app.py"]
$ docker build -t <YOUR_USERNAME>/myfirstapp .
Sending build context to Docker daemon 9.728 kB
Step 1 : FROM alpine:latest
---> 0d81fc72e790
Step 2 : RUN apk add --update py-pip
---> 976a232ac4ad
Removing intermediate container 8abd4091b5f5
Step 3 : COPY requirements.txt /usr/src/app/
---> 65b4be05340c
Step 4 : RUN pip install --no-cache-dir -r
/usr/src/app/requirements.txt
---> 8de73b0730c2
Step 5 : COPY app.py /usr/src/app/
…
Dockerfile은 컨테이너 내부 이미지 환경 및 구성 정의

Dockerfile best practice - 딱 필요한 Base 파일 선택
From the stock ubuntu image:
ubuntu latest 2b1dc137b502 52 seconds ago 458 MB
From python:2.7-alpine:
alpine latest d3145c9ba1fa 2 minutes ago 86.8 MB

고객사례 - Nextdoor
Base OS version
Apt packages:
OpenSSL
libpq
syslog-ng
Datadog
Python runtime
PyPI packages:
Boto
Django
Mapnik
SendGrid
Source code
Static assets
Images
JS
CSS

Layer 별 각기 다른 업데이트 주기
Quarterly
Weekly/
monthly
Continuous

AMI에서 Docker Container로 변경
Base OS layer
System packages
Python packages
Nextdoor source

Docker 이전에는 빌드 20분 소요
chroot
sudo apt-get install
sudo pip install
git clone
make install
dpkg create

Base image , system deps 추가
FROM hub.corp.nextdoor.com/nextdoor/nd_base:precise
ADD app/docker/scripts/apt-fast
app/docker/scripts/system-deps.sh
/deps/
RUN /deps/system-deps.sh

Python virtualenv 설정 업데이트
ADD app/docker/scripts/venv-deps.sh
app/apps/nextdoor/etc/requirements*.txt
app/apps/nextdoor/etc/nextdoor.yml
app/services/scheduler/etc/scheduler.yml
app/services/supervisor/etc/supervisor.yml
/deps/
RUN /deps/venv-deps.sh

App 소스 업데이트
ADD app/static/nextdoorv2/images /app/static/nextdoorv2/images
ADD app/thrift /deps/thrift
ADD app/nd /deps/nd
ADD app /app

빌드 시간 20분 -> 평균 2분
ECS에 최종 배포까지 평균 5분

https://docs.docker.com/
https://en.wikipedia.org/wiki/Docker_(software)
https://en.wikipedia.org/wiki/LXC
https://en.wikipedia.org/wiki/Linux_namespaces
https://en.wikipedia.org/wiki/Cgroups
https://en.wikipedia.org/wiki/Chroot
https://www.slideshare.net/Docker/creating-effective-images-abby-fuller-aws
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
https://github.com/docker/labs/blob/master/beginner/chapters/webapps.md
http://crosbymichael.com/dockerfile-best-practices.html
References

Common Questions
• How do I deploy my containers to hosts?
• How do I do zero downtime or blue green deployments?
• How do I keep my containers alive?
• How can my containers talk to each other?
• Linking? Service Discovery?
• How can I configure my containers at runtime?
• What about secrets?
• How do I best optimize my "pool of compute”?

How do we make this work at scale?

We need to
• start, stop, and monitor lots of containers running on
lots of hosts
• decide when and where to start or stop containers
• control our hosts and monitor their status
• manage rollouts of new code (containers) to our hosts
• manage how traffic flows to containers and how
requests are routed

Container Orchestration
Instance Instance Instance
OS OS OS
Container Runtime Container Runtime Container Runtime
App Service App App Service Service

myJob: {
Cpu: 10
Mem: 256
}
Orchestrator
Schedule
Run “myJob”

Instance/OS Instance/OS Instance/OS
Service Management
Scheduling
Resource Management
OrchestrationService Management
§Availability
§Lifecycle
§Discovery

Service Management
Scheduling
Resource Management
Orchestration
Scheduling
§Placement
§Scaling
§Upgrades
§Rollbacks

Service Management
Scheduling
Resource Management
Orchestration
Resource Management
§ Memory
§ CPU
§ Ports

What are container orchestration tools?

Amazon Container Services Landscape
MANAGEMENT
Deployment, Scheduling,
Scaling & Management of
containerized applications
HOSTING
Where the containers run
Amazon Elastic
Container Service
Amazon Elastic
Container Service
for Kubernetes
Amazon EC2 AWS Fargate
IMAGE REGISTRY
Container Image
Repository
GA : June 6, 2018
Seoul : Jan 11, 2019
Amazon Elastic
Container Registry

Run a (managed) container on AWS
AMAZON CONTAINER SERVICES
Choose your orchestration tool1
Choose your launch type2
ECS EKS
EC2 Fargate EC2 Fargate

What is Kubernetes?
Open source container
management platform
Helps you run
containers at scale
Gives you primitives
for building
modern applications

Kubernetes(K8s) Components
Control Plane (Controller)
Etcd Lightweight, open source Key-Value store containing the cluster
API Server Serves the APIs required to manage the cluster
Scheduler Determines where (on which nodes) pods will run in the cluster
Controller Manager
The “worker on the controller” that actually manages the cluster
(e.g. replication)
Kubernetes Node
kubelet Runs the node, starts and stops containers
kube-proxy
Acts as a network proxy – routes traffic based upon IP and Port.
Each service is assigned a unique port on the nodes it runs across,
kube-proxy allows that port to be mapped to whatever the service
expects.
cAdvisor Agent that monitors node health and statistics

Kubernetes(K8s) Architecture

Kubernetes(K8s) Objects
• kubectl
• Pods
• Labels
• Deployments
• Replication Controllers
• Services

kubectl
• Command line interface for
running commands against the
k8s API
• Intuitive familiar commands
(get, create, describe, delete,
etc.) that are simple to learn and
easy to use
~/.kube/config
k8s master
kube-api
scheduler

Pods
• A group of one or more
containers
• Shared:
• Data volumes
• cgroup
• Namespace – network, IPC, etc. node
pod1 pod2

Labels
• Key/Value Pairs
• Used to query specific resources
within your cluster
pod1
pod2
dev
prod
app001
app001

ReplicaSets
• Ensure that a specified number
of pod “replicas” exist in the
cluster
23

Deployments
• Declarative updates for Pods
and ReplicaSets
23

Services
• Abstraction which defines
a logical set of pods and
policy by which to access
them

Services
• Service Discovery:
• Environment variables
• DNS
• Publishing Services:
• LoadBalancer (ELB)
• ClusterIP, NodePort, External Name
(DNS)

kubectl
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-31-24-193.ec2.internal Ready <none> 2m v1.10.3
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-5blrq 1/1 Running 0 3m
kube-system aws-node-btn9b 1/1 Running 0 3m
kube-system aws-node-wvd92 1/1 Running 1 3m
kube-system kube-dns-64b69465b4-gnzpz 3/3 Running 0 1h
kube-system kube-proxy-5prxp 1/1 Running 0 3m
kube-system kube-proxy-86q8k 1/1 Running 0 3m
kube-system kube-proxy-89stl 1/1 Running 0 3m

Dashboard
Deploy the dashboard to your cluster
$ kubectl apply -f
https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kub
ernetes-dashboard.yaml
secret "kubernetes-dashboard-certs" created
serviceaccount "kubernetes-dashboard" created
role.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" created
rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" created
deployment.apps "kubernetes-dashboard" created
service "kubernetes-dashboard" created
Create an eks-admin Account and Cluster Role Binding

Dashboard
$ vi eks-admin-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: eks-admin
namespace: kube-system
$ kubectl apply -f eks-admin-service-
account.yaml
$ eks-admin-cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: eks-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: eks-admin
namespace: kube-system
$ kubectl apply -f eks-admin-cluster-role-
binding.yaml

Dashboard
Retrieve an authentication token
$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin |
awk '{print $1}')
$ kubectl proxy
Starting to serve on 127.0.0.1:8001
Access at http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-
dashboard:/proxy/
copy and paste token for login

Dashboard

Run Nginx
$ kubectl run my-nginx --image nginx --port 80
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
my-nginx 1 1 1 1 13s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
my-nginx-77f56b88c8-dmvtg
1/1 Running 0 33s
$ kubectl describe pod/my-nginx-77f56b88c8-dmvtg
Name: my-nginx-77f56b88c8-dmvtg
Namespace: default
Node: ip-172-31-24-193.ec2.internal/172.31.24.193
Start Time: Fri, 29 Jun 2018 22:04:37 +0900
Labels: pod-template-hash=3391264474
run=my-nginx
Annotations: <none>
Status: Running
IP: 172.31.28.55
Controlled By: ReplicaSet/my-nginx-77f56b88c8

Run Nginx - expose within cluster
$ kubectl expose deployment my-nginx --target-port=80 [--type=LoadBalancer]
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1h
my-nginx ClusterIP 10.100.211.73 <none> 80/TCP 11s
$ kubectl edit svc/my-nginx
apiVersion: v1
kind: Service
…
spec:
clusterIP: 10.100.211.73
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
run: my-nginx
sessionAffinity: None
type: ClusterIP -> LoadBalancer (replace and save)
status:
loadBalancer: {}
add --type=LoadBalancer if you want expose to internet

Run Nginx - expose to internet
$ watch -n 1 “kubectl get services“
my-nginx LoadBalancer 10.100.211.73 <pending> 80:31743/TCP 7m
…
my-nginx LoadBalancer 10.100.211.73 a60e942cbd32d... 80:31743/TCP 7m
$ curl http:// a60e942cbd32d11e7992202c08f5229f-284158314.ap-northeast-
2.elb.amazonaws.com
* clean up
$ kubectl delete svc/my-nginx deployment/my-nginx

Run Nginx w/ YAML
$ vi my-nginx.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80
$ kubectl create -f ./my-nginx.yaml
NAME DESIRED CURRENT UP-TO-
DATE AVAILABLE AGE
my-
nginx 2 2 2 1 6s
$ kubectl delete pod my-nginx

Run Nginx w/ YAML
$ vi my-nginx-app.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
app: nginx
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx
---
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
$ kubectl create -f ./my-nginx-app.yaml

Using Labels
$ kubectl label pods -l app=nginx tier=webserver
pod "my-nginx-431080787-0fqx9" labeled
pod "my-nginx-431080787-d8g3q" labeled
pod "my-nginx-431080787-k2r4m" labeled
$ kubectl get pods -l app=nginx -L tier
NAME READY STATUS RESTARTS AGE TIER
my-nginx-431080787-0fqx9 1/1 Running 0 1m webserver
my-nginx-431080787-d8g3q 1/1 Running 0 1m webserver
my-nginx-431080787-k2r4m 1/1 Running 0 1m webserver

Scaling Application
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
my-nginx 3 3 3 3 4m
$ kubectl get pods -l app=nginx
my-nginx-431080787-0fqx9 1/1 Running 0 4m
my-nginx-431080787-d8g3q 1/1 Running 0 4m
my-nginx-431080787-k2r4m 1/1 Running 0 4m
$ kubectl scale deployment/my-nginx --replicas=2
$ kubectl get pods -l app=nginx
my-nginx-431080787-0fqx9 1/1 Running 0 4m
my-nginx-431080787-d8g3q 1/1 Running 0 4m
$ kubectl delete -f my-nginx-app.yaml

and more …
In-place updates of resources
$ kubectl apply
$ kubectl edit
$ kubectl patch
$ kubectl annotate
…
Disruptive updates
$ kubectl replace
$ kubectl rolling-update
…
$ kubectl autoscale
$ kubectl rolling-update
…
http://kubernetes.io/docs/user-guide/
https://github.com/kubernetes/ku
bernetes/tree/master/examples

Run 2048 w/ YAML
$ vi my-2048.yaml
apiVersion: v1
kind: Service
metadata:
name: my-2048
labels:
app: my-2048
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: my-2048
---
kind: Deployment
metadata:
name: my-2048
spec:
replicas: 1
template:
metadata:
labels:
app: my-2048
spec:
containers:
- name: my-2048
image: sdscello/2048:1
ports:
- containerPort: 80
$ kubectl create -f ./my-2048.yaml
* open a browser and connect to the ELB

Run 2048 w/ CI and CD
Source Code
Github
Jenkins
Registry
Kubernetes Clusterpush
trigger
build run
Enduser
ELB

Storage

Lifecycle of a storage volume
Provisioning Binding Using Reclaiming
• Static
• Dynamic*
• Control loop watches
for PVC requests and
satisfies if PV is
available.
• For Dynamic, PVC
will provision PV
• PVC to PV binding is
one-to-one mapping
• Cluster mounts
volume based on
PVC
• Retain (default)
• Recycle
• Delete

What if I need specific volume type?
StorageClass
gp2 io1 sc1 encrypted
io1
st1
1) Admin pre-provisions
StorageClass based
on workload needs
2) End user requests for
specific volume types
(For ex, encrypted
io1 volume)
3) Control loop watches
PVC request and
allocates volume if
PV exists
MySQL Pods
4) End user creates
stateful workload

Storage : Storage Class
$ vi gp2-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
reclaimPolicy: Retain
mountOptions:
- debug
$ kubectl create -f gp2-storage-class.yaml
$ kubectl get storageclass
Set gp2 as default storage
$ kubectl patch storageclass gp2 -p
'{"metadata":
{"annotations":{"storageclass.kubernetes.io/is-
default-class":"true"}}}’
$ kubectl get storageclass
NAME PROVISIONER AGE
gp2 (default) kubernetes.io/aws-ebs 24s

Storage : Persistent Volume
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM
POLICY STATUS CLAIM STORAGECLASS REASON AGE
* Create 5Gi EBS volume
$ aws ec2 create-volume --size 5 --region ap-northeast-2 --availability-zone ap-northeast-2c --
volume-type gp2
{
"AvailabilityZone": "us-east-1d",
"CreateTime": "2018-07-02T06:29:50.000Z",
"Encrypted": false,
"Size": 5,
"SnapshotId": "",
"State": "creating",
"VolumeId": "vol-0e9bda6cdc69834a7",
"Iops": 100,
"Tags": [],
"VolumeType": "gp2"
}
Replace it to your zone

Storage : Persistent Volume and Claim
$ vi my-aws-pv.yaml
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
name: "pv0001"
spec:
capacity:
storage: "5Gi"
accessModes:
- "ReadWriteOnce"
awsElasticBlockStore:
fsType: "ext4"
volumeID: " vol-0e9bda6cdc69834a7"
$ kubectl create -f my-aws-pv.yaml
$ vi my-aws-pvc.yaml
apiVersion: v1
kind : PersistentVolumeClaim
metadata:
name: pvc0001
spec:
storageClassName: ""
volumeName: pv0001
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5G
$ kubectl create -f my-aws-pvc.yaml

Storage : Persistent Volume and Claim
$ vi my-aws-pvc-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: redis
spec:
containers:
- name: redis
image: redis
volumeMounts:
- name: pvdemo
mountPath: /data
volumes:
- name: pvdemo
persistentVolumeClaim:
claimName: pvc0001
$ kubectl create -f my-aws-pvc-pod.yaml
$ kubectl describe pods redis
Name: redis
Namespace: default
Node: ip-172-31-36-
113.ec2.internal/172.31.36.113
Start Time: Mon, 02 Jul 2018 17:03:26 +0900
Labels: <none>
Annotations: <none>
Status: Running
IP: 172.31.34.41
Containers:
redis:
Mounts:
/data from pvdemo (rw)
/var/run/secrets/kubernetes.io/serviceaccount from
default-token-wtfrw (ro)
Volumes:
pvdemo:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: pvc0001
ReadOnly: false

Storage : Persistent Volume
* log into the worker instance that is running redis pod
$ kubectl get pods
Redis 1/1 Running 0 5s
$ kubectl exec -it redis -- /bin/bash
root@redis:/data# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 20G 2.8G 18G 14% /
tmpfs 998M 0 998M 0% /dev
tmpfs 998M 0 998M 0% /sys/fs/cgroup
/dev/xvdbw 4.8G 20M 4.6G 1% /data
/dev/xvda1 20G 2.8G 18G 14% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 998M 12K 998M 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 998M 0 998M 0% /sys/firmware

Services

Services
• A Kubernetes Service is an abstraction which defines a logical set
of Pods and a policy by which to access them - sometimes called a
micro-service. The set of Pods targeted by a Service is (usually)
determined by a Label Selector.
• Let’s talk about what are the differences between LoadBalancer,
NodePort and Ingress

Services : ClusterIP
• Exposes the service on a cluster-
internal IP
• Only reachable from within the
cluster
• Access possible via kube-proxy
• Useful for debugging services,
connecting from your laptop or
displaying internal dashboards

Services : LoadBalancer
• Exposes the service externally using a
cloud provider’s load balancer.
• NodePort and ClusterIP services (to
which LB will route) automatically
created.
• Each service exposed with a
LoadBalancer (ELB or NLB) will get its
own IP address
• Exposes L4 (TCP) or L7 (HTTP)
services

Service : LoadBalancer - Sample
$ vi my-nginx-lb.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx-lb
labels:
app: nginx-lb
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx-lb
---
kind: Deployment
metadata:
name: my-nginx-lb
spec:
replicas: 3
template:
metadata:
labels:
app: nginx-lb
spec:
containers:
- name: nginx-lb
image: nginx:1.7.9
ports:
- containerPort: 80
$ kubectl create -f ./my-nginx-lb.yaml
$ kubectl get services -o wide
// Find ELB name and connect for test
* clean up
$ kubectl delete –f ./my-nginx-lb.yaml

Services : LoadBalancer - NLB
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer

Services : NodePort
• Exposes the service on each Node’s IP
at a static port.
• Routes to a ClusterIP service, which is
automatically created.
• from outside the cluster:
<NodeIP>:<NodePort>
• 1 service per port
• Uses ports 30000-32767

Service : NodePort - Sample
$ vi my-nginx-np.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx-np
labels:
app: nginx-np
spec:
type: NodePort
ports:
- port: 80
selector:
app: nginx-np
---
kind: Deployment
metadata:
name: my-nginx-np
spec:
replicas: 3
template:
metadata:
labels:
app: nginx-np
spec:
containers:
- name: nginx-np
image: nginx:1.7.9
ports:
- containerPort: 80
$ kubectl create -f ./my-nginx-np.yaml
NAME TYPE CLUSTER-IP EXTERNAL-
IP PORT(S) AGE SELECTOR
my-nginx-
np NodePort 10.100.90.163 <none> 80:31923/
TCP 4s app=nginx-np

$ kubectl describe services my-nginx-np
Name: my-nginx-np
Namespace: default
Labels: app=nginx-np
Annotations: <none>
Selector: app=nginx-np
Type: NodePort
IP: 10.100.90.163
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 31923/TCP
Endpoints: 172.31.31.134:80,172.31.41.219:80,172.31.76.169:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
ClusterIP can be accessible from any Pods are running in the cluster
31923 is the port that listen in the workers. You can access the Pod
from internet if you open a firewall for workers’ security group

$ kubectl run -i --tty --image busybox test --restart=Never --rm /bin/sh
# wget -qO- 10.100.90.163
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
</html>
# exit

* Update security group to allow the access to the workers from outside of internet
* Note Public IP of all the workers and try to connect each of nodes with same port
$ curl 54.89.86.193:31923
<!DOCTYPE html>
<html>
<head>
<style>
body {
width: 35em;
margin: 0 auto;
}
</style>
</head>
<body>
* clean up
$ kubectl delete –f ./my-nginx-np.yaml

Services : Ingress
• Unlike all the above examples, Ingress is actually NOT a type of
service. Instead, it sits in front of multiple services and act as a
“smart router” or entrypoint into your cluster.
• Demo is at the end of the page as it requires helm for ingress
controller

Helm

Helm from DEIS

What is Helm?
• Helm helps you manage Kubernetes applications
• Helm Charts helps you define, install, and upgrade even the
most complex Kubernetes application.
• Charts are easy to create, version, share, and publish
• so start using Helm and stop the copy-and-paste madness.
https://github.com/kubernetes/helm

Preparation - helm
$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --
serviceaccount=kube-system:tiller
$ helm init --service-account tiller
$ kubectl get pods --all-namespaces
kube-system tiller-deploy-f5597467b-z6vrm 1/1 Running 0 7m

Wordpress - helm
$ helm search
NAME VERSION DESCRIPTION
stable/acs-engine-autoscaler 2.1.1 Scales worker nodes within agent pools
stable/aerospike 0.1.5 A Helm chart for Aerospike in Kubernetes
stable/artifactory 6.2.0 Universal Repository Manager supporting all maj...
stable/aws-cluster-autoscaler 0.3.1 Scales worker nodes within autoscaling groups.
stable/buildkite 0.2.0 Agent for Buildkite
stable/centrifugo 2.0.0 Centrifugo is a real-time messaging server.
stable/chaoskube 0.6.0 Chaoskube periodically kills random pods in you...
stable/chronograf 0.3.0 Open-source web application written in Go and R...
stable/cluster-autoscaler 0.2.1 Scales worker nodes within autoscaling groups.
stable/cockroachdb 0.5.1 CockroachDB is a scalable, survivable, strongly...
…
stable/testlink 0.4.15 Web-based test management system that facilitat...
stable/traefik 1.14.2 A Traefik based Kubernetes ingress controller w...
stable/uchiwa 0.2.2 Dashboard for the Sensu monitoring framework
stable/voyager 2.0.0 Voyager by AppsCode - Secure Ingress Controller...
stable/weave-cloud 0.1.2 Weave Cloud is a add-on to Kubernetes which pro...
stable/wordpress 0.7.4 Web publishing platform for building blogs and ...
stable/zeppelin 1.0.0 Web-based notebook that enables data-driven, in...
stable/zetcd 0.1.4 CoreOS zetcd Helm chart for Kubernetes

Wordpress - helm
$ helm install stable/wordpress
RESOURCES:
==> v1/Secret
NAME TYPE DATA AGE
lumpy-mandrill-mariadb Opaque 2 2s
lumpy-mandrill-wordpress Opaque 2 2s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
lumpy-mandrill-mariadb Bound pvc-883cf38a-d348-11e7-9922-02c08f5229fc 8Gi RWO gp2 2s
lumpy-mandrill-wordpress Bound pvc-883da980-d348-11e7-9922-02c08f5229fc 10Gi RWO gp2 2s
==> v1/Service
lumpy-mandrill-mariadb ClusterIP 10.100.235.4 <none> 3306/TCP 2s
lumpy-mandrill-wordpress LoadBalancer 10.100.33.99 a88484869d348... 80:30079/TCP,443:32070/TCP 2s

Wordpress - helm
$ helm install stable/wordpress
NOTES:
1. Get the WordPress URL:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
Watch the status with: 'kubectl get svc --namespace default -w lumpy-mandrill-wordpress'
export SERVICE_IP=$(kubectl get svc --namespace default lumpy-mandrill-wordpress -o
jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo http://$SERVICE_IP/admin
2. Login with the following credentials to see your blog
echo Username: user
echo Password: $(kubectl get secret --namespace default lumpy-mandrill-wordpress -o jsonpath="{.data.wordpress-
password}" | base64 --decode)

Wordpress - helm
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 1d
zooming-frog-mariadb ClusterIP 10.100.160.162 <none> 3306/TCP 8m
zooming-frog-wordpress LoadBalancer 10.100.209.213 a4e9a5ae47c61... 80:32573/TCP,443:32191/TCP 8m
$ kubectl describe service lumpy-mandrill-wordpress
Name: zooming-frog-wordpress
Namespace: default
Labels: app=zooming-frog-wordpress
chart=wordpress-1.0.9
heritage=Tiller
release=zooming-frog
Annotations: <none>
Selector: app=zooming-frog-wordpress
Type: LoadBalancer
IP: 10.100.209.213
LoadBalancer Ingress: a4e9a5ae47c6111e8a86112fe8484ed4-1956022530.us-east-1.elb.amazonaws.com
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 32573/TCP

Wordpress - helm
$ kubectl get secret --namespace default lumpy-mandrill-wordpress –o
jsonpath="{.data.wordpress-password}" | base64 –decode
* Open a browser and connect to the Wordpress Site and Admin Site

Wordpress - helm
$ ls -al ~/.helm/cache/archive
total 64
drwxr-xr-x 4 kimsaeho ANTDomain Users 136 Jun 1 11:58 .
drwxr-xr-x 3 kimsaeho ANTDomain Users 102 May 11 17:36 ..
-rw-r--r-- 1 kimsaeho ANTDomain Users 15532 Jun 30 21:29 wordpress-1.0.9.tgz
$ tar xvfz ~/.helm/cache/archive/wordpress-1.0.6.tgz -C .
$ helm ls
NAME REVISION UPDATED STATUS CHART NAMESPACE
zooming-frog 1 Sat Jun 30 21:30:00 2018 DEPLOYED wordpress-1.0.9 default
* clean up
$ helm delete --purge zooming-frog
look at the some important files (Chart.yaml, values.yaml) that defines how the package is deploying the applications

Services : Ingress

Services : Ingress
• exposes HTTP/HTTPS
routes to services within
the cluster
• Many implementations:
ALB, Nginx, F5, HAProxy
etc
• Default Service Type:
ClusterIP

ALB Ingress Controller
AWS Resources
Kubernetes Cluster
Node Node
Kubernetes
API Server ALB Ingress
Controller
Node
HTTP ListenerHTTPS Listener
Rule: /cheesesRule: /charcuterie
TargetGroup:
Green (IP Mode)
TargetGroup:
Blue (Instance
Mode)
NodePort NodePort

Service : Ingress - Sample - Extended
ELBingress-*.popori.net
Nginx Ingress
ingress-nginx.popori.net
Ingress-tutum.popori.net
Jenkins
Github
Registry
build
push
pull
run

Service : Ingress - Sample
$ helm install stable/nginx-ingress --name=nginx-ingress --namespace=kube-system --set
rbac.create=true
NAME: nginx-ingress
LAST DEPLOYED: Sun Jul 1 00:35:45 2018
NAMESPACE: kube-system
STATUS: DEPLOYED
==> v1/Service
nginx-ingress-controller LoadBalancer 10.100.198.62 <pending> 80:30396/TCP,443:30752/TCP 1s
nginx-ingress-default-backend ClusterIP 10.100.170.212 <none> 80/TCP 1s
==> v1/Pod(related)
nginx-ingress-controller-67b9bf4c56-plhgf 0/1 Running 0 1s
nginx-ingress-default-backend-d676cbb5f-xcbzf 0/1 ContainerCreating 0 1s
NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace kube-system get services -o wide -w nginx-ingress-
controller'

$ vi my-nginx-ingress.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
app: nginx
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: nginx
---
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: sdscello/nginx
ports:
- containerPort: 80
$ kubectl create -f ./my-nginx-ingress.yaml

$ vi my-nginx-ingress-expose.yaml
kind: Ingress
metadata:
name: my-nginx-ingress
spec:
rules:
- host: ingress.popori.net
http:
paths:
- path: /
backend:
serviceName: my-nginx
servicePort: 80
* If you don’t have your own domain, you can
use ELB DNS Name instead
$ kubectl create -f ./my-nginx-ingress-
expose.yaml
$ kubectl describe services my-nginx-ingress
Name: my-nginx-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
ingress.popori.net
/ my-nginx:80 (<none>)
Annotations:
…
* Connect to your domain and make sure you
can see the nginx index page

Let’s run another pod
$ vi my-tutum-ingress.yaml
apiVersion: v1
kind: Service
metadata:
name: my-tutum
labels:
app: tutum
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: tutum
---
kind: Deployment
metadata:
name: my-tutum
spec:
replicas: 3
template:
metadata:
labels:
app: tutum
spec:
containers:
- name: tutum
image: tutum/hello-world
ports:
- containerPort: 80
$ kubectl create -f ./my-tutum-ingress.yaml

$ kubectl edit ingress my-nginx-ingress
kind: Ingress
metadata:
name: my-nginx-ingress
namespace: default
spec:
rules:
- host: ingress.popori.net
http:
paths:
- backend:
serviceName: my-nginx
servicePort: 80
path: /
- backend:
serviceName: my-tutum
servicePort: 80
path: /tutum
status:
loadBalancer:
ingress:
- {}
Add this lines, so it rewrites /tutum requests to the appropriate pod

$ curl http://ingress.popori.net
<!DOCTYPE html>
<html>
<head>
<style>
body {
width: 35em;
margin: 0 auto;
}
</style>
</head>
<body>
<p>If you see this page, the nginx web server is
successfully installed and
working. Further configuration is required.</p>
…
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
$ curl http://ingress.popori.net/tutum
<html>
<head>
<title>Hello world!</title>
<link
href='http://fonts.googleapis.com/css?family=Open+Sans:4
00,700' rel='stylesheet' type='text/css’>
…
</head>
<body>
<img id="logo" src="logo.png" />
<h1>Hello world!</h1>
<h3>My hostname is my-tutum-8479747799-8jqks</h3>
<h3>Links found</h3>
<b>MY_TUTUM</b> listening in 80 available at
tcp://10.100.253.39:80<br />
<b>MY_NGINX</b> listening in 80 available at
tcp://10.100.50.246:80<br />
<b>KUBERNETES</b> listening in 443 available at
tcp://10.100.0.1:443<br />
</body>
</html>

StatefulSet

Statefulset Properties
• Network identifiers
• Persistent Storage
• Ordered graceful deployment and scaling
• Ordered graceful termination
• Ordered rolling updates
• If none of these fit your portfolio, use Deployment or Replicaset

StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request
StorageClass
gp2 io1 sc1 encrypted
io1
st1
3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Network
Identifiers
Ordered
Deployment

StatefulSet
1) Define headless
service, statefulset
and PVC
2) Control loop allocates
PV based on PVC
request
3) Kubernetes creates
statefulset
MySQL Pods
mysql-0 mysql-1 mysql-2 mysql-3
Ordered
Scaling
mysql-4

StatefulSet
• StatefulSets are intended to be used with stateful applications
and distributed systems.
• Like a Deployment, a StatefulSet manages Pods that are based on
an identical container spec. Unlike a Deployment, a StatefulSet
maintains a sticky identity for each of their Pods. These pods are
created from the same spec, but are not interchangeable: each has
a persistent identifier that it maintains across any rescheduling.

StatefulSet - Sample
$ vi my-nginx-ss.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
app: my-nginx
spec:
ports:
- port: 80
clusterIP: None
selector:
app: my-nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-web
spec:
serviceName: "my-nginx"
replicas: 2
selector:
matchLabels:
app: my-nginx
template:
metadata:
labels:
app: my-nginx
spec:
containers:
- name: my-nginx
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- name: my-pv
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: my-pv
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi

$ kubectl get pods -w
my-web-0 0/1 Pending 0 7s
my-web-0 0/1 ContainerCreating 0 15s
my-web-0 1/1 Running 0 24s
* StatefulSet with N replicas, when Pods are being deployed, they are created sequentially, in order
from {0..N-1}.
* Notice that the my-web-1 Pod is not launched until the my-web-0 Pod is Running and Ready

kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 6d
my-nginx ClusterIP None <none> 80/TCP 1m
$ kubectl get statefulset
NAME DESIRED CURRENT AGE
my-web 2 2 1m
$ kubectl get pods
my-web-0 1/1 Running 0 2m
$ kubectl exec -it my-web-0 -- /bin/bash
root@my-web-0:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 20G 3.0G 18G 15% /
tmpfs 998M 0 998M 0% /dev
/dev/xvdbp 976M 2.6M 907M 1% /usr/share/nginx/html

$ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'hostname'; done
my-web-0
my-web-1
$ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'echo $(hostname) >
/usr/share/nginx/html/index.html'; done
$ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'apt-get -qq update; apt-get -y install curl'; done
$ for i in 0 1; do kubectl exec -it my-web-$i -- sh -c 'curl localhost'; done
my-web-0
my-web-1

$ kubectl run -i --tty --image busybox test --restart=Never --rm /bin/sh
# nslookup my-web-0.my-nginx
Server: 10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local
Name: my-web-0.my-nginx
Address 1: 172.31.34.41 my-web-0.my-nginx.default.svc.cluster.local
# wget -qO- my-web-0.my-nginx.default.svc.cluster.local
my-web-0

$ kubectl get pods -w -l app=my-nginx
my-web-0 1/1 Running 0 2d
my-web-1 1/1 Running 0 2d
my-web-0 1/1 Terminating 0 2d
$ kubectl delete pods -l app=my-nginx
pod "my-web-0" deleted
pod "my-web-1" deleted

$ for i in 0 1; do ../../kubectl exec -it my-web-$i -- sh -c 'df -h | grep html'; done
/dev/xvdbp 976M 2.6M 907M 1% /usr/share/nginx/html
/dev/xvdcv 976M 2.6M 907M 1% /usr/share/nginx/html
$ for i in 0 1; do ../../kubectl exec -it my-web-$i -- sh -c 'curl localhost'; done
sh: 1: curl: not found
command terminated with exit code 127
sh: 1: curl: not found
command terminated with exit code 127
$ for i in 0 1; do kubectl exec my-web-$i -- sh -c 'apt-get -qq update; apt-get -y install curl'; done
$ for i in 0 1; do kubectl exec -it my-web-$i -- sh -c 'curl localhost'; done
my-web-0
my-web-1
Because new pods has launched, manually installed curl is no longer exists in the pods.
But, the contents (index.html) that stored in EBS volume is still available.

57%of Kubernetes workloads
run on AWS today
— Cloud Native Computing Foundation

What is Amazon EKS?
• Amazon Elastic Container Service for Kubernetes (Amazon EKS) is
a managed service that makes it easy for you to run Kubernetes
on AWS without needing to stand up or maintain your own
Kubernetes control plane. Kubernetes is an open-source system
for automating the deployment, scaling, and management of
containerized applications.

Kubernetes on AWS
Managed Kubernetes on
AWS
Highly
available
Automated
version
upgrades
Integration
with other
AWS services
Etcd
Master
Managed
Kubernetes
control
plane CloudTrail,
CloudWatch, ELB,
IAM, VPC, PrivateLink

Kubernetes on AWS
3x Kubernetes masters for HA

Availability
Zone 1
Master Master
Availability
Zone 2
Availability
Zone 3
Master
Workers Workers Workers
Customer Account
AWS Managed

EKS Control Plane

EKS Architecture
mycluster.eks.amazonaws.com
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
Kubectl

EKS Architecture
EKS VPCCustomer VPC
Worker Nodes
EKS-Owned
ENI
Kubernetes
API calls
Exec, Logs,
Proxy
Internet

EKS Architecture

CloudWatch Integration

Kubernetes Control Plane
Highly available and single tenant
infrastructure
All “native AWS” components
Fronted by an NLB
VPC
API Server ASG
Etcd ASG
NLB
AZ-1 AZ-2 AZ-3
ELB
Instances
Instances

Master Node
Scheduler
Controller
Manager
Cloud Controller
Manager
API Server
etcd

What happens when I run ‘kubectl create –f pods.yaml’?

IAM Authentication
Kubectl
3) Authorizes AWS Identity with RBAC
K8s API
1) Passes AWS Identity
2) Verifies AWS Identity
4) K8s action
allowed/denied
AWS Auth

Master Node
Scheduler
Controller
Manager
Cloud
Controller
Manager
API Server
etcd
Kubectl

API Server
Kubectl
Authorization
Webhook RBACaws-iam-
authenticator
Authentication Admission Controllers
Mutating
Webhook
Validation
Webhook

kubectl configuration
# [...]
users:
- name: aws
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "CLUSTER_ID"
- "-r"
- "ROLE_ARN"
# no client certificate/key needed here!

Cluster Authentication and Authorization
• User or IAM role who creates EKS cluster gains Admin privileges
• This {“super”} user/role can then add additional users or IAM roles
and configure RBAC permissions
• To add, configure aws-auth Configmap
kubectl edit -n kube-system configmap/aws-auth

aws-auth configuration
apiVersion: v1
data:
mapRoles: |
- rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
mapUsers: |
- userarn: arn:aws:iam::555555555555:user/admin
username: admin
groups:
- system:masters
- userarn: arn:aws:iam::555555555555:user/john
username: john
groups:
- pod-admin # k8s RBAC group

EKS Data Plane

Kubernetes Data Plane
Worker Node
kube-dnsKubelet
aws-
node
Container runtime
Control Plane
API
kube-
proxy

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT
ExecStart=/usr/bin/kubelet --cloud-provider aws
--config /etc/kubernetes/kubelet/kubelet-config.json
--allow-privileged=true
--kubeconfig /var/lib/kubelet/kubeconfig
--container-runtime docker
--network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS
Restart=on-failure
RestartForceExitStatus=SIGPIPE
RestartSec=5
KillMode=process
[Install]
WantedBy=multi-user.target

EKS AMI Build Scripts
https://github.com/awslabs/amazon-eks-ami
Source of truth for EKS Optimized AMI
Easily build your own EKS AMI
Build assets for EKS AMI for each supported Kubernetes version

EKS Optimized AMI with GPU Support
Easily run Tensorflow/Kubeflow on Amazon EKS
Includes NVIDIA packages to support Amazon P2 and P3 instances
Available on AWS Marketplace

Worker Node Setup – Bootstrapping
/etc/eks/bootstrap.sh <cluster-name> [options]
Uses UserData for configuring System resources and extra Kubelet
config
Reserve compute resources for System Daemons (Kubelet, Container
runtime) and Pod eviction thresholds

EKS Upgrades

Kubernetes Version
Versions supported: 1.10.13, 1.11.8, 1.12.6, 1.13.7
EKS will support up to 3 versions of Kubernetes at once
”Deprecation” will prevent new cluster creation on old version

Container Services Roadmap
https://github.com/aws/containers-roadmap

EKS Platform Version
Platform Version revisions represent API server configuration
changes or Kubernetes patches
Platform Versions increment within a Kubernetes version only

EKS Platform Version

EKS Kubernetes Version Updates
New UpdateClusterVersion API –
supports in place updates of Kubernetes
version
Introduces an ”update” EKS API object
ListUpdates and DescribeUpdate APIs to
provide visibility into the status of a
given update

Updating Worker Nodes
Two options:
1) Create new node group with latest EKS AMI >> taint old nodes >>
drain old nodes >> terminate old CFN template
2) Simply update AMI in CFN template; “rolling” replacement policy
terminates nodes
(Downsides: un-graceful termination of applications)

EKS Networking & Load Balancing

AWS VPC CNI Plugin
ENI
Secondary IPs:
10.0.0.1
10.0.0.2
10.0.0.1
10.0.0.2
ENI
10.0.0.20
10.0.0.22
Secondary IPs:
10.0.0.20
10.0.0.22
ec2.associateaddress()
VPC Subnet –
10.0.0.0/24
Instance 1 Instance 2
VPC

AWS VPC CNI plugin – understanding IP
allocation
Primary CIDR range è RFC 1918 addresses è 10/8, 172.16/12, 192.168/16
Used in EKS for:
• Pods
• X-account ENIs for (masters à workers) communication (exec, logs,
proxy etc.)
• Internal Kubernetes services network (10.100/16 or 172.20/16 –
chosen based on your VPC range)
Setup:
• EKS cluster creation è provide list of subnets (in at least 2 AZs!) è
tagging

AWS VPC CNI plugin – understanding IP
allocation
Secondary CIDR ranges (new!) è non-RFC 1918 address blocks
(100.64.0.0/10 and 198.19.0.0/16)
Used in EKS for:
• Pods only
How?
• EKS custom network config è enable è create ENIConfig CRD è
annotate nodes
CNI
1.2.1+

Load Balancing
All three AWS Elastic Load Balancing products are supported
NLB and CLB supported by Kubernetes Service type=LoadBalancer
Internal and External Load Balancer support

Load Balancing
Want to use an Internal Load Balancer? Use annotation:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
Want to use an NLB? Use annotation:
service.beta.kubernetes.io/aws-load-balancer-type: nlb

Production-Ready 1.0 Release
Supported by Amazon EKS Team
Open Source Development: https://github.com/kubernetes-
sigs/aws-alb-ingress-controller
Customers are using it in production today!

AWS Resources
Kubernetes Cluster
Node Node
Kubernetes
API Server ALB Ingress
Controller
Node
HTTP ListenerHTTPS Listener
Rule: /cheesesRule: /charcuterie
TargetGroup:
Green (IP Mode)
TargetGroup:
Blue (Instance
Mode)
NodePort NodePort
Ingress
Resource
Creation via
Kubectl or API

Windows Support

CSI Drivers for EFS and FSx Lustre

AWS App Mesh GA

Get Started
https://eksworkshop.com
Modules:
• Health Checks
• Logging with Elasticsearch, Fluentd, and
Kibana (EFK)
• Monitoring using Prometheus and Grafana
• Servicemesh with Istio
• Stateful Containers using StatefulSets

Amazon ECS
ALB ALB
AZ 1 AZ 2
user/scheduler
Scheduler
Cluster State Service
Placement Engine
Event Stream
Internet
ECS agent ECS agent ECS agent
EC2 instance EC2 instance EC2 instance

Amazon ECS
EC2 INSTANCES
LOAD
BALANCER
Internet
ECS
AGENT
TASK
Container
TASK
Container
ECS
AGENT
TASK
Container
TASK
Container
AGENT COMMUNICATION
SERVICE
Amazon ECS
API
CLUSTER MANAGEMENT
ENGINE
KEY/VALUE STORE
ECS
AGENT
TASK
Container
TASK
Container
LOAD
BALANCER

Amazon ECS : Cluster
EC2 INSTANCES
LOAD
BALANCER
Internet
ECS
AGENT
TASK
Container
TASK
Container
ECS
AGENT
TASK
Container
TASK
Container
AGENT COMMUNICATION
SERVICE
Amazon ECS
API
CLUSTER MANAGEMENT
ENGINE
KEY/VALUE STORE
ECS
AGENT
TASK
Container
TASK
Container
LOAD
BALANCER

Amazon ECS : Task
EC2 INSTANCES
LOAD
BALANCER
Internet
ECS
AGENT
TASK
Container
TASK
Container
ECS
AGENT
TASK
Container
TASK
Container
AGENT COMMUNICATION
SERVICE
Amazon ECS
API
CLUSTER MANAGEMENT
ENGINE
KEY/VALUE STORE
ECS
AGENT
TASK
Container
TASK
Container
LOAD
BALANCER

Tasks are defined via Task Definitions
{
"containerDefinitions": [
{
"name": "simple-app",
"image": "httpd:2.4",
"cpu": 10,
"memory": 300,
"portMappings": [
{
"hostPort": 80,
"containerPort": 80,
"protocol": "tcp"
}
],
"essential": true,
"mountPoints": [
{
"containerPath": "/usr/local/apache2/htdocs",
"sourceVolume": "my-vol"
}
]
},
{
"name": "busybox",
"image": "busybox",
"cpu": 10,
"memory": 200,
"volumesFrom": [
{
"sourceContainer": "simple-app"
}
],
"command": [
"/bin/sh -c "...""
],
"essential": false
}
],
"volumes": [
{
"name": “my-vol"
}
]
}

{
{
"name": "simple-app",
"image": "httpd:2.4",
"cpu": 10,
"memory": 300,
"portMappings": [
{
"hostPort": 80,
"containerPort": 80,
"protocol": "tcp"
}
],
"essential": true,
"mountPoints": [
{
"containerPath": "/usr/local/apache2/htdocs",
"sourceVolume": "my-vol"
}
]
},
10 CPU units (1024 is a full CPU)
300 MB of memory
Expose port 80 in container
to port 80 on host
Create and mount volumes
Essential to our task

{
"name": "busybox",
"image": "busybox",
"cpu": 10,
"memory": 200,
"volumesFrom": [
{
"sourceContainer": "simple-app"
}
],
"command": [
"/bin/sh -c "...""
],
"essential": false
}
],
"volumes": [
{
"name": “my-vol"
}
]
}
From Docker Hub
Mount volume from other container
Command to exec
Volumes

Task log to CloudWatch Logs
CloudWatch Logs Amazon S3
Amazon Kinesis
AWS Lambda
Amazon ElasticSearch
Amazon ECS Store
Stream
Process
Search
CloudWatch Logs
CloudWatch Logs
CloudWatch Logs

IAM Task Role
AWS IAM
Amazon
DynamoDB
S3
AWS IAM
DynamoDBRole
S3Role
Amazon
ECS
IAM Task
Role
Identity
Access
Management
(IAM)
ECS Task

Task Placement Constraints
Name Example
AMI ID
attribute:ecs.ami-id == ami-
eca289fb
Availability
Zone
attribute:ecs.availability-
zone == us-east-1a
Instance
Type
attribute:ecs.instance-type
== t2.small
Distinct
Instances
type=“distinctInstance”
Custom attribute:stack == prod
Cluster
Constraints
Custom
Constraints
Placement
Strategies
Apply Filter
CPU, memory, port requirements
AZ, EC2 type, AMI, or custom
constraints
Spread or Binpack
placement strategy
Select final instances for
task deployment

Task Placement Strategies
Binpacking Spread Affinity Distinct Instance

Example : Instance type and Zone
g2.2xlarge t2.small t2.micro t2.medium
t2.medium t2.small g2.2xlarge
t2.small
t2.small t2.medium
us-east-1aus-east-1d

Amazon ECS : Service
EC2 INSTANCES
LOAD
BALANCER
Internet
ECS
AGENT
TASK
Container
TASK
Container
ECS
AGENT
TASK
Container
TASK
Container
AGENT COMMUNICATION
SERVICE
Amazon ECS
API
CLUSTER MANAGEMENT
ENGINE
KEY/VALUE STORE
ECS
AGENT
TASK
Container
TASK
Container
LOAD
BALANCER

Task and Service
• Split an application that only runs with
necessary bin/libs
• IAM task role must be set
• Restricted use of privileged users within a
container
• Configure LogDriver (awslogs, fluentd, gelf,
journald, splunk, syslog .. )
S e r v i c e sT a s k s
• Configure task placement to distribute equally
across multiple availability zones
• Service Auto Scaling
• Application Load Balancer

CloudWatch ECS Metric
2 Dimensions
• ClusterName
• ServiceName
4 metrics
• CPUReservation
• MemoryReservation
• CPUUtilization
• MemoryUtilization
Container
Instance
…
Cluster
Task
definition
Task
Service
CloudWatch
ECS Metrics
CloudWatch
EC2 Metrics
Container
Instance
Container
Instance

ECS Cluster (EC2 Instance) Auto Scale out
Event: Per cluster CPU, memory
reservation, or usage
New services
ECS
ECS cluster
CloudWatch
Developers
CloudWatch event

ECS Cluster (EC2 Instance) Auto Scale in
Draining
ECS
ECS cluster
CloudWatch
Event: Per cluster CPU, memory
reservation, or usage

Service Auto Scaling
Amazon EC2
Service
Resource
buffer
(+~15%)

Auto Scaling Target Tracking
Only need to set the target value for the
metric
(ex: CPU utilization 50%)
Auto Scaling automatically adjusts the Task
DesiredCount in Service
CloudWatch metric
ECSServiceAverageCPUUtilization
ECSServiceAverageMemoryUtilization
ALBRequestCountPerTarget
CPUTraffic
DesiredCount
Time
100%
0%
50%
10%
20%
30%
40%
60%
70%
80%
90%
5
30
10
15
20
25
Target CPU Utilization DesiredCount

Get Started
https://ecsworkshop.com
Modules:
• Introduction
• Platform
• Frontend Rails App
• Node.js Backend API
• Crystal Backend API

AWS Fargate : Only focus on tasks!
Simple, Easy, efficient
Serverless
Container!
=No EC2 Instances
to provision, scale
or manage
ECS
Native API ,
Integrated with
VPC, ELB, IAM,
CloudWatch
and more
Pay for CPU,
Memory Usage

AWS Fargate
AWS VPC
networking mode
Advanced task
placement
Deep integration
with AWS platform
ECS CLI…{ }
Global footprint (in 2018)
Powerful scheduling
engines
Auto scaling
CloudWatch metrics
Load balancers

AWS Fargate
Scheduling and Orchestration
Cluster Manager Placement Engine
ECS
AMI
Docker
agent
ECS
agent
EC2 Instance
ECS
AMI
Docker
agent
ECS
agent
EC2 Instance
ECS
AMI
Docker
agent
ECS
agent
EC2 Instance

Amazon EC2 and AWS Fargate Hybrid cluster
ECS Instance ECS Instance ECS Instance ECS InstanceECS Instance ECS Instance
EC2
FARGATE
Notifications
Amazon ECS CLUSTER
Availability Zone #1 Availability Zone #2 Availability Zone #3
Subnet 2
172.31.2.0/24
Subnet 1
172.31.1.0/24
Subnet 3
172.31.3.0/24
Web
Shopping
Cart

Cluster level isolation
PROD Cluster Infrastructure
DEV Cluster Infrastructure
BETA Cluster Infrastructure
QA Cluster Infrastructure
Web Web
Shopping
Cart
Shopping
Cart
Notifications NotificationsWeb
Shopping
Cart NotificationsWeb
Shopping
Cart
Shopping
Cart
Notifications NotificationsWeb Web
PROD CLUSTER BETA CLUSTER
DEV CLUSTER QA CLUSTER

Fargate
Define application containers: Image
URL, CPU & Memory requirements, etc.
register
Task Definition
create
Cluster
• Infrastructure Isolation
boundary
• IAM Permissions boundary
run
Task
• A running instantiation of
a task definition
• Use Fargate launch type
create
Service
Elastic Load
Balancing

CPU & Memory specification
Task Level Resources:
• Total CPU/memory across all containers
• Required fields
• Billing dimensions
Units
• CPU: cpu-units. 1 vCPU = 1024 cpu-units
• Memory: MB
Container Level Resources:
• Defines sharing of task resources among
containers
• Optional fields
{
"family": "scorekeep",
"cpu": "1 vCpu",
"memory": "2 gb",
{
"name":“scorekeep-frontend",
"image":"xxx.dkr.ecr.us-east-1.amazonaws.com/fe“,
"cpu": 256,
"memoryReservation": 512
},
{
"name":“scorekeep-api",
"image":"xxx.dkr.ecr.us-east-1.amazonaws.com/api",
"cpu": 768,
}
]
}
Task
Level
Resources
Container
Level
Resources
Task Definition Snippet

VPC Integration
Launch your Fargate Tasks into subnets
Under the hood :
• We create an Elastic Network Interface (ENI)
• The ENI is allocated a private IP from your subnet
• The ENI is attached to your task
• Your task now has a private IP from your subnet!
You can assign public IPs to your tasks
Configure security groups to control inbound & outbound
traffic
172.31.0.0/16
Subnet
172.31.1.0/24
Other Entities in VPC
EC2 LB DB etc.
Private IP
172.31.1.164
ENI Fargate
TaskPublic /
208.57.73.13 /

VPC Configuration
{
"family": "scorekeep",
"cpu": "1 vCpu",
"memory": "2 gb",
"networkMode": "awsvpc",
{
"name":“scorekeep-frontend",
"image":"xxx.dkr.ecr.us-east-1.amazonaws.com/fe",
"cpu": 256,
},
{
"name":“scorekeep-api",
"image":"xxx.dkr.ecr.us-east-1.amazonaws.com/api",
"cpu": 768,
}
]
}
$ aws ecs run-task ...
-- task-definition scorekeep:1
-- network-configuration
“awsvpcConfiguration = {
subnets=[subnet1-id, subnet2-id],
securityGroups=[sg-id]
}”
Enables ENI
creation &
attachment
to Task
Run Task
Task Definition

Private Task Setup
Public subnet Private subnet
Fargate
TaskENI
Private IP
172.31.1.164
NAT Gateway
Public EIP
34.214.162.237
Internet
Gateway
172.31.0.0/16
172.31.2.0/24 172.31.1.0/24
Destination Target
172.31.0.0/16 local
0.0.0.0/0 NAT Gateway
Destination Target
172.31.0.0/16 local
0.0.0.0/0 Internet Gateway
Route Tables
Internet
Attach Internet Gateway to VPC
Setup a Public Subnet with
• Route to Internet Gateway
• NAT Gateway
Setup Private Subnet with
• Fargate Task
• Route to NAT Gateway
Security Group to allow outbound traffic
Type Port Destination
All Traffic ALL 0.0.0.0/0
Outbound Security Group Rules

Public Task Setup
Outbound
Inbound
Public subnet
Fargate
Task
Public IP
54.191.135.66
Internet
Gateway
172.31.0.0/16
172.31.2.0/24
Destination Target
172.31.0.0/16 local
0.0.0.0/0 Internet Gateway
Route Table
Internet
ENI
$ aws ecs run-task ...
-- network-configuration
“awsvpcConfiguration = {
subnets=[public-subnet],
securityGroups=[sg-id],
}”
Launch the task into a Public subnet
Give it a public IP address
Security Group to allow the expected inbound traffic
Type Port Source
HTTP 8080 0.0.0.0/0
Inbound Security Group Rule
Type Port Destination
All Traffic ALL 0.0.0.0/0
Outbound Security Group Rules
assignPublicIp=ENABLED
Run Task

Internet Facing ELB VPC Setup
Public subnet Private subnet
Fargate
TaskENI
Private IP
172.31.1.16 :8080
ALB
172.31.0.0/16
172.31.2.0/24 172.31.1.0/24 Task in private subnet with private IP
Task Security GroupALB Security Group
Type Port Source
HTTP 80 0.0.0.0/0
Inbound Rule
Type Port Source
Custom
TCP
8080 ALB Security
Group
Inbound Rule
Destination Target
172.31.0.0/16 local
0.0.0.0/0 NAT G/W
Destination Target
172.31.0.0/16 local
0.0.0.0/0 Internet G/W
Public Subnet route table Private Subnet route table
ALB in public subnet
ALB Security group to allow inbound traffic from
internet
Task security group to allow inbound traffic from
ALB security group

Fargate Storage
Layer Storage Space :
• 10 GB layer storage available per task
across all containers in a single task
• Includes image layers
Ephemeral storage backed by Amazon EBS
Fargate volume Storage :
• 4 GB volume space per task
• Visible across containers
• Configure via task definitions
Image Layers
Writable Layer
Image Layers
Writable Layer
Container 1 Container 2
10 GB per Task
Container 1 Container 2
4 GB Volume Storage
mount
/var/container1/data /var/container2/data

PERMISSION TIERS
Cluster
Permissions
Application
Permissions
Task
Housekeeping
Permissions
Cluster
Fargate Task
Cluster Permissions:
Who can run/see tasks in the cluster?
Application (Task) Permissions:
Which of my AWS resources can this application access?
Housekeeping Permissions:
What permissions do I want to grant ECS to perform?
e.g.
• ECR Image Pull
• CloudWatch Logs pushing
• ENI creation
• Register/Deregister targets into ELB

COMPLIANCE
9001/27001/27017/27018

AWS Fargate Customers
”We don't want to
babysit any clusters.
That has nothing to do
with us”
Shimon Tolts
CTO, DATREE
“We moved to Fargate
because we need the
ability to scale quickly up
from baseline and get
fine-grained network
control, without having to
manage our own
infrastructure”
Product Hunt

Entire website runs as microservices. Ruby &
GraphQL backend with node.js frontend
Needed ability to scale quickly, schedule multi-
container workloads, network layer control
All in on AWS—Moved entire infrastructure to AWS
and Fargate in Jan 2018
Fargate scales quickly with traffic spikes, running
multiple services in production
Product Hunt: AWS Fargate

Fargate pricing
CPU Memory
256 (.25 vCPU) 512MB, 1GB, 2GB
512 (.5 vCPU) 1GB to 4GB
1024 (1 vCPU) 2GB to 8GB
1 vCPU = $0.04656/hour
1 GB Mem = $0.00511/hour
50 different CPU/memory configurations

게임에서 컨테이너 활용하기

Stateless
Scale Out
Stateful
Scale Out
Lifetime
Stateful
Scale Up

Dedicated Server 관리 방식 #1

Dedicated Server 관리 방식 #2
Agent가 대기
명령을 받으면 Dedi Server를 실행
Git이나 SVN 을 통해 형상관리
execute
node.js Agent
Dedi Server

Container로 운영하기 위한 선결 과제
• 포트 매핑
• static
• dynamic
• 퍼블릭 주소 알아내기
• 밖에서 알아내기
• 안에서 알아내기

EKS 에서 Dedicated Server 운영하기
• Agones
Dedicated Game Server Hosting and Scaling for Multiplayer Games on
Kubernetes
• 새로운 pod 타입
GameServer
GameServerAllocation
• PortPolicy
dynamic

ECS에서 Dedicated Server 운영하기
• Dynamic port mapping for ECS
• TaskDefinition에서 host port 영역을 0으로 설정
• Docker 1.6 이후에서 49153 ~ 65535영역을 매핑
• Public IP
EC2 Metadata
http://169.254.169.254/latest/meta-data/public-ipv4
• Public Port
ECS Task Metadata
V3: ${ECS_CONTAINER_METADATA_URI}/task
V2: http://169.254.170.2/v2/metadata

Thank you!

Amazon Container Services – 유재석 (AWS 솔루션즈 아키텍트)

More Related Content

Similar to Amazon Container Services – 유재석 (AWS 솔루션즈 아키텍트)

More from Amazon Web Services Korea

Recently uploaded

Amazon Container Services – 유재석 (AWS 솔루션즈 아키텍트)