Kubernetes and container security

Kubernetes & Container Security
by Volodymyr Shynkar
Senior Lead DevOps Engineer
2021 | intellias.com

[devops@stage ~]$ cat ABOUT_ME.md
• 6+ years of commercial DevOps experience. Overall 8+ years of
Engineering
• Member of Technology Office
• Member of the Center of Excellence
• Successfully migrated, rolled out, consulted over 15 projects in
the healthcare, gambling, automotive, e-commerce industries
• Certified SAFe Agile Software Engineer
• Addicted to IoT and Smart Home
• Cyclist, promoter of a healthy lifestyle
Volodymyr Shynkar
Senior Lead DevOps Engineer at Intellias
[devops@stage ~]$

Agenda Overview
Container Security
Kubernetes Security
Other Tooling to help

Will talk about:
• Scan containers and Pods for vulnerabilities or misconfigurations.
• Run containers and Pods with the least privileges possible.
• Use network separation to control the amount of damage a
compromise can cause.
• Use firewalls to limit unneeded network connectivity and
encryption to protect confidentiality.
• Use strong authentication and authorization to limit user and
administrator access as well as to limit the attack surface.
• Periodically review all Kubernetes settings and use vulnerability
scans to help ensure risks are appropriately accounted for and
security patches are applied.

Attack Surface
Analysis for:
• Cloud and Host
• Kubernetes Cluster
• Container (images and running)
Goal: Reduce the attack surface

Attack Surface – Cloud & Host
There are at least few things that you do yo achive base
security level:
• Run instances in private network
• Expose services only through external services like LB or Proxy
• Block all external traffic except exposed ports, like 80, 443
• Do not expose SSH. Try to use SSM instead.
• Minimize privilege to applications running on the host
• Optional: for HTTP traffic, use WAF if posiable
Goal: Follow “Principle of least privilege”
Managed K8s services already goes with most of this features enabled

Dockerfile best practices
1.Avoid unnecessary privileges.
1. Avoid running containers as root.
2. Don’t bind to a specific UID.
3. Make executables owned by root and not writable.
2.Reduce attack surface.
1. Leverage multistage builds.
2. Use distroless images, or build your own from scratch.
3. Update your images frequently.
4. Watch out for exposed ports.
3.Prevent confidential data leaks.
1. Never put secrets or credentials in Dockerfile instructions.
2. Prefer COPY over ADD.
3. Be aware of the Docker context, and use .dockerignore.
4.Others.
1. Reduce the number of layers, and order them intelligently.
2. Add metadata and labels.
3. Leverage linters to automatize checks.
4. Scan your images locally during development.
5.Beyond image building.
1. Protect the docker socket and TCP connections.
2. Sign your images, and verify them on runtime.
3. Avoid tag mutability.
4. Don’t run your environment as root.
5. Include a health check.
6. Restrict your application capabilities. Source: https://sysdig.com/blog/dockerfile-best-practices/

Let’s start from scratch
First steps
Start from your app
• unprivileged user (rootless)
• read-only
• no shell, cat, grep, less, tail, echo, etc
• focus on fewer data stored inside the container - only
app, no source code, and build dependencies
• no backed secrets or bind through volume or
encrypted
Seriously :)

First steps
Start from your app
• read-only
encrypted
Serious :)

First steps
Start from your app
• read-only
encrypted
Seriously :)
More examples:
https://github.com/GoogleContainerTools/distroless

First steps
Start from your app
• read-only
• focus on fewer data stored inside the
container - only app, no source code, and
build dependencies
encrypted
Seriously :)

Scan your image
Docker and Snyk recently entered into a partnership to
provide container vulnerability scanning.
Alternative
Source: https://www.docker.com/blog/bringing-docker-scan-to-linux/

Next to discuss:
• Deployment to the cluster
• Pod Security Policy
• Open Policy Agent
• Network Policy
• Secrets
• Securing the Cluster
Kubernetes Hardening
Remember: follow the “Principle of least privilege”

Deployment to the cluster
kubectl apply –f my_app.yaml
helm install my_app charts/my_app

How to automate deploy?
There are two approaches:
• Push-based
• Regular CI (Jenkins, Gitlab, GitHub, CircleCI)

How to automate deploy?
There are two approaches:
• Pull-based
• GitOps strategy (ArgoCD, Flux)

How to manage manage
By only two resources: AppProject and Application

By only two resources: AppProject and Application
How to manage manage

How to grant access
ArgoCD provided with powerful UI
• Embedded security features
• Support of SAML/OKTA.
• Enhanced experience
• Role-based
• Easy to use

Pod Security Policy
PSP are one way to control the security-related attributes of pods,
including container privilege levels.
• Do not run application processes as root
• Do not allow privilege escalation
• Use a read-only root filesystem
• Use the default (masked) /proc filesystem mount
• Do not use the host network or process space
• Drop unused and unnecessary Linux capabilities
• Service Account control

Pod Security Policy
When a PSP resource is created, it does nothing. You need to
authorize using RBAC!

Pod Security Policies is going to be marked as
deprecated since v1.21 and will be removed in v1.25
The PSP Replacement Policy is the new enhancement:
https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/2579-psp-replacement
PSP Replacement Policy
The way PSPs are applied
to Pods has proven
confusing to nearly everyone
that has attempted to use
them.

Open Policy Agent (Gatekeeper)
Gatekeeper controller provides you ability to:
• Required labels
• Required resources section
• Mutate container images to always point to the corporate image registry.
• Set node and pod affinity and anti-affinity selectors to Deployments
• You can enforce anything that you want to see or not to see in configs
I need to control other pod fields or
any fields in other resource? How
can I achieve that?
The answer is:

Example: enforce to use only allowed container registries

Example: enforce to use only allowed container registries
The other examples can be found here:
https://github.com/open-policy-agent/gatekeeper/tree/master/demo

Network Policy
By default, namespaces are not automatically isolated.
For that we have networking policies and RBAC.
With this simple config you will
isolate NS from other NS

Network Policy
Network policy will not work with default CNI from
AWS EKS. You need to install Calico.
With this simple config you will allow
traffic from specific namespace

Network Policy
With AWS EKS CNI you can use different security
groups per pod which makes network policy redundant
With this simple config you will allow
traffic to a specific port
You can also specify protocol.

Secrets
Where do I version control my secrets?
What is solution for this?
• Paper/USB/CDR two fireproof safes?
• Vault (or actually Consul)?

Secrets
Where do I version control my secrets?
What is solution for this?
• Sealed Secrets (a Kubernetes controller and a tool for
one-way encrypted Secrets):
https://github.com/bitnami-labs/sealed-secrets
• Git Crypt - transparent file encryption in git:
https://www.agwa.name/projects/git-crypt/

Sealed Secrets example
As you can see the value of the secret is enrypted

Sealed Secrets example
The controller will unseal that into something like:

Securing the Cluster
API Server
By default, the API server will listen on what is rightfully
port 8080
• Close the insecure port by arg in API server’s --insecure-port flag to 0
• And --insecure-bind-address is not set.
Any requests to this port bypass authentication and authorization
checks.

etcd
The etcd backend database is a critical component and the
to secure within the cluster.
Close the insecure port by arg in API server’s --insecure-port flag to
0
• And --insecure-bind-address is not set.
etcd server should be configured to only trust certificates assigned to API
servers

Kubelet
The kubelet is the agent that is responsible launching pods (not
Check this parameters:
• Disable anonymous access with --anonymous-auth=false
• Ensure that requests are authorized by setting --
other than AlwaysAllow

Kubernetes Dashboard
The Dashboard has historically been used by attackers to gain
Kubernetes clusters.
Check this parameters:
• Allow only authenticated access. Only known users should be able
Dashboard.
• Use RBAC. Limit the privileges that users have so they can
they need to.
• Don’t expose your Dashboard to the public internet
• Unless you really know what you’re doing.

Protocol Direction Port Range Purpose
TCP Inbound 6443 or 8080 if not disabled Kubernetes API server
TCP Inbound 2379-2380 etcd server client API
TCP Inbound 10250 kubelet API
TCP Inbound 10251 kube-scheduler
TCP Inbound 10252 kube-controller-manager
TCP Inbound 10258 cloud-controller-manager (optional)
The following table lists the control plane ports and services.
You can try to curl on each port to check if it’s secured.

Kubescape
Kubescape is the first tool for testing if Kubernetes deployed securely
Source: https://github.com/armosec/kubescape

Kube-bench
The same tool as Kubescape but could be deployed as CronJob and executed
on regular bases
Source: https://github.com/aquasecurity/kube-bench

Kubesec
kubectl plugin for scanning Kubernetes pods,
deployments, daemonsets and statefulsets
Source: https://github.com/controlplaneio/kubectl-kubesec
Suggests what should be improved or changed

Kubeaudit
kubeaudit is a command line tool to audit Kubernetes clusters for various security
concerns, such as:
• run as non-root
• use a read-only root filesystem
• drop scary capabilities, don't add new ones
• don't run privileged
Source: https://github.com/Shopify/kubeaudit

Ksniff
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture
on any pod in your Kubernetes cluster.
Source: https://github.com/eldadru/ksniff

THANK
YOU!
Let's get connected!

Kubernetes and container security

In this document