Infrastructure as code using Kubernetes

Infrastructure as
code using GCP K8S
by Walter 2017/08/30

–Walter Liu
“Based on a true story. Not best practice :P”
覺得quote slide很有趣拿來⽤ :P

Overcrowded - Asia-east-c
服務要上線了，天啊！！！

Using GCP to create new cluster in
1 hour

Google Cloud Platform
❖ Container Engine
❖ Container Registry
❖ SQL
❖ Networking
❖ Load Balancer
❖ Firewall
❖ Monitoring (StackDriver)
❖ Logging
❖ Storage
❖ IAM & Service Account

Kubernetes
❖ Infrastructure as code
❖ Self-healing
❖ Horizontal scaling
❖ Automatized rollouts and rollback
❖ Service Discovery and Load Balancing
❖ Configuration
❖ ……

Write down notes or even script

Use K8S configs (apply -f) and
Template them to reduce complexity

Write operation scripts to deal with
daily jobs

TitanOPS script
❖ shell
❖ logs
❖ clogs
❖ deploy
❖ udeploy
❖ update
❖ images

GCP: Container Engine
❖ 3 * 2 CPU 7.5 GB memory VM
❖ get external IPs of Nodes for Google Cloud SQL (1st
generation)

GCP: Container Registry
❖ For Container Engine
❖ Items
❖ main_game_web
❖ nimo_server
❖ twemproxy

GCP: SQL
❖ Dee (failover)
❖ Nimo (failover)
❖ Nimo-kpi (MyISAM, 1st generation, IP-based)
❖ Nimo-chat (MyISAM, 1st generation, IP-based)
❖ 2 types of connection
❖ CloudSQLProxy (2nd generation)
❖ IP (1st generation)
❖ How to check connection
❖ python manage.py dbshell (dee platform)
❖ mysql -u xxxx -p xxx -h xxx -P (nimo)

Networking: Load Balancer
❖ Google Load Balancer (Prod)
❖ client => CDN => Google Load Balancer => K8S
❖ Kubernetes Load Balancer (Dev/Stg)
❖ with Firewall settings (in K8S config)

GCP: Monitor and Logging
❖ click and see them
❖ Command line logging shows only today

Storage
❖ Container Registry repo
❖ static web sites
❖ Use ./titan_control.py udeploy to deploy them.
❖ dev-static
❖ stg-static
❖ prod-static
❖ stg-nimo-res
❖ prod-nimo-res
❖ prod-tl-static

Kubernetes component used
❖ POD
❖ Deployment
❖ Statefulset
❖ Service
❖ Ingress

Deployment & Statefulset
❖ xxx-web-api
❖ xxx-web-celery-beat
❖ xxx-web-celery-worker
❖ xxx-web-cache (statuefulset)
❖ xxx-redis-queue
❖ xxx-nimo-server
❖ xxx-nimo-celery-beat
❖ xxx-nimo-celery-worker
❖ xxx-nimo-tool
❖ xxx-nimo-payment
❖ xxx-nimo-kvs (statuefulset)

Service
❖ External
❖ Any service with external IP
❖ Internal
❖ xxx-internal-nimo-xxx-service
❖ nimo-xxx-db-master/slave (SQL 1st generation)
❖ xxx-redis-queue-service

❖ GCP Load Balancer
❖ sites
❖ g.deegroup.com
❖ a-g.deegroup.com
❖ app-tl.deegroup.com
Ingress

CDN static site cache time
❖ 2 minutes client cache time
❖ 7 days CDN edge cache time
❖ Written in titan_control.py

K8S templating
❖ K8S doesn’t supports templating yet.
❖ ./titan_control.py deploy
❖ dev
❖ stg
❖ prod
❖ dev-nimo
❖ stg-nimo
❖ prod-nimo
❖ deploy/templates/*
❖ deploy/params/*

Deploy new code
❖ Update to git
❖ ./titan_control.py update
❖ modify image_tag: ?????? of
“deploy/params/xxxx.yaml”
❖ ./titan_control.py deploy dev/stg/prod ….

Deploy new static content
❖ Update to git/update to nimo res
❖ ./titan_control.py update (For nimo res, skip this)
❖ ./titan_control.py udeploy prod/prod-nimo-res …

Other commands
❖ Tracking logs
❖ shell to container (always chose first, for dev)
❖ images

How to create in 1 hour
❖ In my notes
❖ Create secrets
kubectl create secret generic cloudsql-credentials --from-file=credentials.json=secret/credentials.json
❖ Create config maps
kubectl apply -f configs/telegraf.yaml
gcloud compute http-health-checks list
gcloud compute http-health-checks update k8s-be-31451--XXXXXXXXXXX --host xxx.com
❖

Infrastructure as code using Kubernetes

More Related Content

Similar to Infrastructure as code using Kubernetes

More from Walter Liu

Recently uploaded

Infrastructure as code using Kubernetes