Kubernetes Networks Solutions Comparison - Objectif Libre
Kubernetes Networks Solutions Comparison - Objectif Libre
+ 33 (0)5 82 95 65 36
contact@objectif-libre.com (mailto:contact@objectif-libre.com) (https://www.objectif-libre
Paris (https://goo.gl/maps/hHWcASts6Ho)
Toulouse (https://goo.gl/maps/MkGQ1k8Npc72)
Stockholm (https://goo.gl/maps/QFKBWhuke782)
(https://www.linkedin.com/company/objectif-
(https://www.objectif-
(https://www.facebook.com/ObjectifLibre)
(https://twitter.com/objectiflibre)
(https://www.youtube.com/user/objectiflibre)
libre) libre.com/en/feed/)
Newsletter
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 1/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
This blog post presents and compares different network solutions working with (https://www.objectif-libre
Kubernetes.
The f irst solution to conf igure the network is to create a virtual bridge with an IP
range, then add manually on each host a route between hosts. With Google or Amazon
cloud solutions, manual conf iguration is possible, but when you don’t perfectly stick
with it, it tends to be more diff icult to manage the conf igurations.
Hereafter a conf iguration example for bridge creation with one node and one
interface:
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 2/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
(https://www.objectif-libre
The second solution to setup Kubernetes network is to use the Container Network
Interface (CNI) and a network plugin. Basic conf igurations are automatically
generated. Creation and administration of networks become easier.
Linux networking can be def ined in two ways: underlay or overlay. Kubernetes makes it
possible to use both.
CNI is a group of specif ications and libraries (written in Go) that aim at helping
network plugins integration. A CNI plugin must be executed by the container system
management. It manages the interface setup (and its IP) in a namespace and its
conf iguration with the host (bridge connection and routes management).
The network’s type conf iguration used is written in the /etc/cni/net.d/xx-mynet.conf
f ile.
Network name, type (bridge, vlan, ipvlan, loopback, macvlan, ptp) are usually specif ied,
as well as its IPam (dhcp, host-local) with a type, subnetwork and routes linked to it.
Network plugins create their own types with the information given above.
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 3/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
Numerous networking solutions compatible with Kubernetes are available. They all use
or create a CNI plugin.
Three solutions are mainly used: Calico, Flannel and WeaveNet. Two others, Cilium
(https://www.objectif-libre
and Contiv, provide interesting features too.
In this blog post, we are going to present different solutions and their operations with
Kubernetes.
Note
Calico
Presentation
Operation
Felix: Calico agent, installed on each node supplies end points (external interface),
it shares ip table and routes between nodes.
BIRD (bpg): client (and route reflector) used with confd for BGP. Border Gateway
Protocol (BGP) is a routing protocol which shares its routing informations (tables)
with autonomous systems routers. BIRD is a daemon that act like a dynamic router
used by Calico with BGP. It is used to centralize routes distribution.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 4/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
confd: monitores etcd for BGP conf iguration changes. It automatically manages
BIRD conf igurations.
etcd or API Kubernetes: etcd is a key-value store.
(https://www.objectif-libre
Usage
Calico network
Calico networks and subnetworks can be represented as follow. In this example, Two
subnetworks are created and stretched accross nodes, allowing pods on differents
hosts to communicate on the same network.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 5/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
(https://www.objectif-libre
Selecting a network can be done when conf iguring a pod, using annotations:
metadata:
annotations:
"cni.projectcalico.org/ipv4pools": "[\"10.112.12.0/24\"]" # or ipv6po
ols if ipv6 is used
Deployment
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 6/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
(https://www.objectif-libre
(https://www.objectif-libre.com/f r/?attachment_id=12991)
Documentation : https://docs.projectcalico.org/v3.1/usage/
(https://docs.projectcalico.org/v3.1/usage/)
Cilium
Note
Kernel ≥ 4.8
Presentation
Cilium is a network solution for Kubernetes. It uses L3/L4 for the network part and L7
for the application part.
The L7 support allows adding high-level f ilter rules for web applications. It supports
ipv4 and ipv6. Cilium is the only solution to offer BPF f iltering.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 7/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
(https://www.objectif-libre
BPF is a packet f ilter solution which can replace iptables. The f ilter isn’t performed at
application level, but at the kernel level: it’s more eff icient and secure.
Cilium uses BPF to create and apply f ilter rules on packets, no iptable rule is created.
Filters are more effective and flexible.
Operation
Cilium works with the cilium-agent run on each node. Cilium-agent manages
operations and f ilters to share with hosts. It compiles BPF f ilters and redirects it to the
kernel’s host.
Usage
Cilium can work with overlay network (picture1, default) or with native routing
(picture2). IPv4 and IPv6 can be used in both cases. As we can see in the picture below,
a subnetwork is assigned on each node. Native routing is more complex to use.
Every outgoing local network packet is sent on routing kernel system which is used to
redirect packets.
Security rules are managed for IPv6; concerning IPv4, you have to use CIDR rules. IP
forwarding is enabled by Cilium, but redirecting rules must be managed manually or
with the BGP routing protocol.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 8/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
(https://www.objectif-libre
Cilium Network
Cilium adds its own networkPolicies usable with kubectl get ciliumnetworkpolicy (or
cnp) to f ilter L7 packets like http or kafka.
Kubernetes networkPolicies are applied automatically. All these conf igurations can be
written in yaml. The example below is a L7 f ilter rule made with CiliumNetworkPolicy
to f ilter http and allow “/” path access only for pods with the ‘access’ label set to true.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 9/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "Allow HTTP GET / from app=web to access=true"
metadata:
(https://www.objectif-libre
name: "rule1"
spec:
endpointSelector:
matchLabels:
app: web
ingress:
- fromEndpoints:
- matchLabels: # check labels
access: "true"
toPorts:
- ports:
- port: "80"
protocol: TCP
rules:
http:
- method: "GET"
path: "/"
A Kubernetes networkPolicy is visible with the cilium policy get command. The f ile
is in json format. Below, a networkPolicy example written in yaml applied with the
Kubernetes command, and its recording in json format returned by the cilium policy
get command.
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 10/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
Several commands are available to access the Cilium data state. The list is available
here http://cilium.readthedocs.io/en/latest/cheatsheet/
(http://cilium.readthedocs.io/en/latest/cheatsheet/). Some examples:
(https://www.objectif-libre
kubectl exec -n kube-system cilium-qpxvw
-- ls -al /sys/fs/bpf/tc/globals/ # list bpf entry
-- cilium bpf policy list -n 19898 # display policy with this number (i
ptables type display)
-- cilium status # give node status (--all-containers for all node)
-- cilium policy get # list all policies recieve by cilium (json forma
t)
Cilium offers identity system and allows to give priority level thanks to tags used on
pods.
Cilium operates with kube-proxy. But when ClusterIP (load balancing for pods traff ic)
is used, Cilium works as a proxy by adding and deleting BPF rules on each node. When
it is used with Istio, it uses Envoy as a proxy.
Deployment
Documentation: http://cilium.readthedocs.io/en/latest/
(http://cilium.readthedocs.io/en/latest/)
Contiv
Presentation
Operation
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 11/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
netplugin runs on each node. It implements the CNI plugin, and also manages pod
interface creations, IP allocations and so on.
netmaster runs on the master nodes as a DaemonSet. It manages network
requests and sends routes def initions to the netplugin components
(https://www.objectif-libre
The components use their host IP to communicate. A default contiv vlan is created to
mesh all the cluster nodes.
Usage
Networks can be created with the netctl command. The available networks are
visible with the netctl net ls command. IPs are def ined f rom pools with Nw Type
parameter set to data .
The network selection for pods is done in the yaml def inition using labels:
labels:
io.contiv.tenant: default
io.contiv.network: contiv-net
Contiv operates with overlays to def ine subnetworks, shared on all the nodes of the
cluster:
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 12/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
Contiv doesn’t support Kubernetes networkPolicies, but offers its own networkPolicies
with its own api for f iltering rules. It offers two rules types: Bandwitdth and Isolation.
The Bandwidth type can be used to limit bandwidth for a group. The Isolation type (https://www.objectif-libre
can be used to create whitelists or blacklists for an application in a group.
The netctl command allows to do these conf igurations but rule creation is also
possible with the web api offered by Contiv.
This interface allows to manage tenants, users (possible ldap authentif ication) and
networks too.
Note
Deployment
Documentation : http://contiv.github.io/documents/gettingStarted/
(http://contiv.github.io/documents/gettingStarted/).
Flannel
Presentation
Operation
Flannel uses etcd to store its conf iguration and the managed networks information. It
implements subnetworks on each host using a flanneld agent.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 13/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
Flannel creates only one VxLAN. Every new node is attached to this VxLAN with a veth.
Flannel doesn’t support running multiple networks on a single daemon, but it is
possible to run multiple daemons on each host.
(https://www.objectif-libre
Usage
Flannel creates a cni0 bridge on each node and attaches veth interfaces to it. Each
node manages a subnetwork of the Flannel pool. The communication is possible using
VxLAN tunnels created by flanneld on each host.
Flannel network
It is possible to enable packet routing using VxLAN-GBP when many hosts are on the
same network. VxLAN networks are then used only if the network is different.
Flannel manages ipv4 traff ic only between cluster nodes. It focuses on network and
does not support Kubernetes networkPolicies.
Deployment
Flannel is deployed as a DaemonSet. Before deploying it, the following options need to
be added in /etc/kubernetes/manifests/kube-controller-manager.manifest: --
allocate-node-cidrs=true --cluster-cidr=10.32.0.0/16 . The kubelet service must be
restarted. Installation is described here. (https://github.com/coreos/flannel#deploying-
flannel-manually)
Documentation : https://github.com/coreos/flannel#flannel
(https://github.com/coreos/flannel#flannel)
WeaveNet
Note
kernel ≥ 3.8, docker ≥1.10.0, Kubernetes ≥ 1.4, master with at least 2 CPU.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 14/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
(https://www.objectif-libre
Presentation
Weave net provides VxLAN on layer 2 networking for Kubernetes. It uses kube-proxy
and kube-dns. It supports IPv4 and IPv6.
Operation
Unlike other network solutions using etcd to stock data, WeaveNet saves its settings
and data in a /weavedb/weave-netdata.db f ile and shares it on each pod createy by the
DaemonSet. Each pod owns a node physical interface IP address when it is created.
Each pod has two containers: weave and weave-npc (Network Policy Controller).
Weave containers manages all Weave operations on the node. The weave-npc
container manages the Kubernetes NetworkPolicies.
Usage
Weave Network
As we can see on the picture below, on each pod a “weave” bridge is created.
Containers are connected to the host bridge using a virtual interface. Communication
between hosts is encapsulated using VxLAN.
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 15/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
It is possible to def ine more subnetworks. Subnetworks allocation are made with
IPAM and can be done on different modes:
seed: You should arrange a f irst cluster with f ixed devices number. A subnetwork is
assigned on each device. It is possible to add the device into the cluster, which can
either do an integral part of the cluster (a new f ixed device) or be integrated
dynamically according to the requirement, the device can be deleted.
consensus : Determines networks thanks to consensus algorithm (default choice). It
is used to put weave in interactive mode or to put f ix cluster where the addition or
deletion devices are rare. This mode uses weave prime command who
automatically put nodes IP allocation.
observer : Offers to add nodes as observer role. It asks IP pool if needed (in overload
node case) for a node to split its IP pool. It allows to dynamically add cluster nodes
according to requirement (autoscaling).
Deployment
Documentation : https://www.weave.works/docs/net/latest/kubernetes/kube-addon/
(https://www.weave.works/docs/net/latest/kubernetes/kube-addon/)
Conclusion
Network plugins are a real advantage and make network management easier. Each
provides a different set of features.
For a POC or if we want to quickly setup the network, it is best to use Flannel or
WeaveNet.
Calico, Contiv, and Cilium offer to use an underlay network (including BGP) directly,
and avoid VxLAN encapsulation.
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 16/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
Several solutions (Calico, Contiv) offer to add many virtual networks for the whole
cluster, pods can connect on a same network f rom different nodes.
(https://www.objectif-libre
Cilium is more security focused and offers application layer f iltering. It uses BPF to
f ilter at the kernel level. BPF f ilter offers better performances that iptables f ilters.
Summary
Kubernetes Layers Co
Solution networkPolicies IPv6 used Networks Deployment lig
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 17/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
PREVIOUS (HTTPS://WWW.OBJECTIF-LIBRE.COM/EN/BLOG/2018/06/11/SELF-
HOSTING-A-SECURE-DOCKER-REGISTRY-WITH-PORTUS/) (https://www.objectif-libre
NEXT (HTTPS://WWW.OBJECTIF-LIBRE.COM/EN/BLOG/2018/07/10/COMPLIANCE-
ENFORCEMENT-OPENSTACK-CLOUD-PUPPET-FOREMAN-OPENSCAP/)
GET IN TOUCH
+ 33 (0)5 82 95 65 36
contact@objectif-libre.com (mailto:contact@objectif-libre.com)
Toulouse :
5 rue du Moulin Bayard
31000 Toulouse
Paris :
359 rue St Martin
75003 Paris
Stockholm :
Embassy House
Linnégatan 89C
115 23 Stockholm
Sweden
Meet us (https://www.objectif-libre.com/en/about-us-2/contact/)
CONTACT US
actu (https://www.objectif-
libre.com/en/blog/tag/actu/) Ansible
(https://www.objectif-libre
(https://www.objectif-libre.com/en/blog/tag/ansible/) Bilan
(https://www.objectif-libre.com/en/blog/tag/bilan/) business (https://www.objectif-
libre.com/en/blog/tag/business/) Ceph (https://www.objectif-
libre.com/en/blog/tag/ceph/) Client (https://www.objectif-libre.com/en/blog/tag/client/)
Cloud (https://www.objectif-
libre.com/en/blog/tag/cloud/) CloudKitty
(https://www.objectif-libre.com/en/blog/tag/cloudkitty/) COA
(https://www.objectif-libre.com/en/blog/tag/coa/) Conference
(https://www.objectif-
libre.com/en/blog/tag/conference/) Containers
(https://www.objectif-
libre.com/en/blog/tag/containers/) DevOps
(https://www.objectif-
libre.com/en/blog/tag/devops/) DigitalPlace (https://www.objectif-
Docker (https://www.objectif-
libre.com/en/blog/tag/digitalplace/)
/ / / / /
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 19/20
23/04/2019 Kubernetes networks solutions comparison - Objectif Libre
libre.com/en/blog/tag/pause/) petit-déjeuner
(https://www.objectif-libre.com/en/blog/tag/petit-dejeuner/) POSS
(https://www.objectif-libre.com/en/blog/tag/poss/) (https://www.objectif-libre
Prometheus (https://www.objectif-
libre.com/en/blog/tag/prometheus/) R&D (https://www.objectif-libre.com/en/blog/tag/rd/)
libre.com/en/blog/tag/tech/) toulouse
(https://www.objectif-libre.com/en/blog/tag/toulouse/) Webinaire
(https://www.objectif-
libre.com/en/blog/tag/webinaire/) Free software
(https://www.objectif-libre.com/en/blog/tag/logiciels-libres/)
événement (https://www.objectif-
libre.com/en/blog/tag/evenement/)
Mentions légales
https://www.objectif-libre.com/en/blog/2018/07/05/k8s-network-solutions-comparison/ 20/20