KEMBAR78
Extending the Kube API | PDF
Extending the Kube	API
Dr.	Stefan	Schimanski
sttts@redhat.com /	sttts @	Github
@the1stein
Kubernetes Meetup Hamburg,	September	21	2017
Disclaimer:
Kubernetes happens to be able to launch pods.	It‘s even quite good at	it.
We will	not	launch pods today.
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/<name>
/api/v1/pods/<name>/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alpha1/jobs
/apis/batch/v2alpha1/cronjobs
/apis/batch/v1
/apis/batch/v1/jobs
kube-apiserver
kubeletproxy
Node:
$	kubectl create -f	foo.yaml
User:
scheduler
controller
manager
apiserver
ingress
controller
Master:
cloud native	apps
Pods:
$	kube-apiserver
--secure-port 0	
--etcd-servers	http://127.0.0.1:2379
--service-cluster-ip-range	10.0.0.0/16
--storage-backend	etcd2
--storage-media-type	application/json
$ etcd
$ kubectl config set-cluster	local --server=http://127.0.0.1:8080
$	kubectl config set-context local --cluster=local
$	kubectl config use-context local
$ kubectl get namespaces --v=7
$ kubectl get namespace default -o	json
$ kubectl annotate namespace default meetup=hello
$	curl http://127.0.0.1:8080/api/v1/namespaces/default
$ etcdctl ls --recursive
$ etcdctl get /registry/namespaces/default
$ etcdctl -o	extended get /registry/namespaces/default
$	http	GET http://127.0.0.1:8080/api/v1/namespaces/default
{
"apiVersion":	"v1",				
"kind": "Namespace",
"metadata": {
"annotations": {
"meetup": “hallo"
},
"creationTimestamp":	"2017-03-10T07:51:39Z",								
"name": "default",
"resourceVersion": “73",
},
"spec": { "finalizers": ["kubernetes“]	},
"status": { "phase": "Active“ }
}
$	http	GET http://127.0.0.1:8080/api/v1/namespaces/default |	
jq ".metadata.annotations["meetup"]	= "$(date)""	|	
http	PUT http://127.0.0.1:8080/api/v1/namespaces/default
HTTP/1.1	200	OK
Content-Length:	341
Content-Type:	application/json
Date:	Fri,	10	Mar	2017	08:28:01	GMT
$	while true;	do
http	GET	http://127.0.0.1:8080/api/v1/namespaces/default |	
jq ".metadata.annotations["meetup"]	=	"$RANDOM""	|	
http	--check-status	PUT	http://127.0.0.1:8080/api/v1/namespaces/default ||	break
done
HTTP/1.1	409	Conflict
Content-Length:	310
Content-Type:	application/json
Date:	Fri,	10	Mar	2017	08:27:58	GMT
{				
"apiVersion":	"v1",				
"code":	409,			
"details":	{							
"kind":	"namespaces",								
"name":	"default“
},				
"kind":	"Status",				
"message":	"Operation	cannot be fulfilled on	namespaces "default":	the object has been
modified;	please apply your changes to the latest version and try again",				
"metadata":	{},				
"reason":	"Conflict",				
"status":	"Failure“
/apis/batch/v2alpha1/jobs
Apigroup Version			Resource
HTTP	paths:
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
/namespaces/<name>/
{				
"apiVersion":	"v1",				
"kind":	"Status",				
"metadata":	{},				
"code":	409,			
"message":	“...",				
"status":	"Failure“
}
/apis/batch/v2alpha1/jobs
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
Resource vs.	Kind
http	path vs.	logical object
*	I	omitted the namespace in	/apis/batch/v1/jobs/namespaces/default/nightly
/apis/extensions/v1alpha1/jobs
/apis/batch/v2alpha1/jobs
/apis/batch/v1/jobs {
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
*	I	omitted the namespace in	/apis/batch/v1/jobs/namespaces/default/nightly
/apis/extensions/v1alpha1/jobs
/apis/batch/v2alpha1/jobs
/apis/batch/v1/jobs {
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
*	I	omitted the namespace in	/apis/batch/v1/jobs/namespaces/default/nightly
/apis/extensions/v1alpha1/jobs/nightly
/apis/batch/v2alpha1/jobs/nightly
/apis/batch/v1/jobs/nightly
{
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{		...	}
}
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
*	I	omitted the namespace in	/apis/batch/v1/jobs/namespaces/default/nightly
{
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v1	“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
etcd
{
“apiVersion“:	“v1	“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
Protobuf
or
JSON
storage
version
encoding
I	want my own kinds
and store them in	the apiserver and use kubectl.
Discovery
how kubectl knows which kinds/resources exist
$	http	127.0.0.1:8080/apis/	
{
"groups":	[{
"name": "batch",
"preferredVersion": {"groupVersion": "batch/v1", "version": "v1“},
"versions": [{"groupVersion": "batch/v1",		"version":	"v1"}]
},	...]
}
$	http	127.0.0.1:8080/apis/batch/v1
{
"apiVersion":	"v1",
"groupVersion":	"batch/v1",
"kind":	"APIResourceList",
"resources":	[{
"kind":	"Job",
"name":	"jobs",
"namespaced":	true,
"verbs":	["create",	"delete",	"deletecollection",
"get",	"list",	"patch",	"update",	"watch“
]
},	...]
}
resource name ⇒ /apis/batch/v1/jobs
/namespaces/<name>/
Third	Party	Resources
Third	Party	Resources
*	they are marked as beta in	the API,	but	became beta before we even had alpha.
Third	Party	Resources
*	they are marked as beta in	the API,	but	became beta before we even had alpha.
ThirdPartyResources are limited
• no version conversion
• no defaulting
• no validation
• no subresources (scale,	status)
• no admission
• alpha ⇒ API	might change,	beta proposal is in	discussion
• demand is high,	expect improvements in	1.7+
• Today‘s users of TPRs:
https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c
https://github.com/kubernetes/features/issues/95
https://github.com/kubernetes/community/pull/524
Custom	Resource Definitions (CRDs)
apiextensions/v1beta1
Custom	Resource Definitions (CRDs)
apiextensions/v1beta1
• apiVersion:	apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:		
name:	databases.example.com
spec:
group: example.com
version: v1
names:
kind:	Database
plural:	databases
scope:	Namespacedmust	match
Custom	Resource Definitions
• CRDs	replace ThirdPartyResources in	1.7+	
• TPRs will	be removed in	1.8,	simple	migration exists
• single version per	resource
• consistent API	like	native	types
• stable basis for more features
• validation with JSON-Schema	
• defaulting via	JSON-Schema
• sub-resources (proposal is being written)
• initializers in	1.7
• garbage collection in	1.8
in	1.8,	Google	Summer	of Code	project
!
apiVersion:	apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:		
name:	databases.example.com
spec:
group: example.com
version: v1
names:
kind:	Database
plural:	databases
scope:	Namespaced
$	kubectl create –f	databases-crd.yaml $	kubectl describe crd databases.example.com
$	kubectl get crd databases.example.com -o	yaml
...
status:		
acceptedNames:				
kind: Database				
listKind: DatabaseList
plural:	databases
singular: database
conditions:	
- type:	NamesAccepted
message: no conflicts found
reason: NoConflicts
status: "True“
- type: Established
message: the initial	names have been accepted
reason:	InitialNamesAccepted
status: "True“
a	moment later
$	kubectl create –f	databases-crd.yaml $	kubectl create –f	wordpress-databases.yaml
apiVersion:	example.com/v1
kind:	Database
metadata:
name:	wordpress
spec:
user:	wp
password:	secret
encoding:	unicode
apiVersion:	apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:		
name:	databases.example.com
spec:
group: example.com
version: v1
names:
kind:	Database
plural:	databases
scope:	Namespaced
CustomResource (CR)CustomResourceDefinition (CRD)
$ kubectl get databases –w --no-headers
wordpress <none>				{"apiVersion":"example.com/v1","kind":"Databases",...
wordpress <none>				{"apiVersion":"example.com/v1","kind":"Databases",...
$ curl -f	'http://127.0.0.1:8080/apis/example.com/v1/namespaces/default/databases?watch=true&resourceVersion=434‘
{"type":"DELETED","object":{"apiVersion":"example.com/v1","kind":"Databases","metadata":{"name":"wordpress","names
pace":"default","selfLink":"/apis/example.com/v1/namespaces/default/databases/wordpress","uid":"8f5312c0-29c8-11e7-
88f9-4c3275978b79","resourceVersion":"435","creationTimestamp":"2017-04-
25T15:05:03Z"},"spec":{"encoding":"unicode","password":"secret","user":"wp"}}}
{"type":"ADDED","object":{"apiVersion":"example.com/v1","kind":"Databases","metadata":{"name":"wordpress","namespa
ce":"default","selfLink":"/apis/example.com/v1/namespaces/default/databases/wordpress","uid":"b4318cb5-29c8-11e7-
88f9-4c3275978b79","resourceVersion":"436","creationTimestamp":"2017-04-
25T15:06:05Z"},"spec":{"encoding":"unicode","password":"secret","user":"wp"}}}
ThirdPartyResources are limited
• no version conversion (only one version possible per	CRD)
• no validation (in	1.8,	Google	Summer	of Code)
• no defaulting (probably in	1.9,	Google	Summer	of Code)
• no subresources (scale,	status)	(proposal is being written)
• no admission (since 1.7:	admission webhooks +	initializers)
• alpha CRDs	are beta
• demand is high
• Today‘s users of TPRs:
https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c
CRDs
Validation	in	1.8+apiVersion:	apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:		
name:	databases.example.com
spec:
...
validation:
openAPIV3Schema:
properties:
spec:
properties:
user:
type:	string
password:
type:	string
encoding:
enum:	[“unicode“, “iso8859“]
required:	[“user“,	“password“,	“encoding“]
metadata:
properties:
name:
pattern:	“^[a-z0-9]+(-[a-z0-9]+)*$“
apiVersion:	example.com/v1
kind:	Databases
metadata:
name:	wordpress
spec:
user:	wp
password:	secret
encoding:	unicode
validates
Controllers
where is the business logic?
$	while true;	do
http	GET	http://127.0.0.1:8080/api/v1/namespaces/default |	
jq ".metadata.annotations["meetup"]	=	"$(date)""	|	
http	--check-status	PUT	http://127.0.0.1:8080/api/v1/namespaces/default ||	break
done
⟲
$	kubectl get namespaces --watch --no-headers	|	
while read NS	STATUS	TIME	;	do
#	do	whatever you like	here,	e.g.	change the namespace
echo	"$NS	changed“
done
⟲
$	curl -f	'http://127.0.0.1:8080/api/v1/namespaces?watch=true&resourceVersion=4711‘
{"type":"ADDED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
{"type":“MODIFIED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
{"type":“DELETED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
API	Aggregation
Why
• CustomResourceDefinitions will	always be limited	(declarative vs.	Turing-complete)
• no version conversion
• validations only via	JSON-Schema	(less expressive	than code)
• simple	defaulting via	JSON-Schema
• admission only out-of-process (slower than in-process)
• Some things need full power	of Go
• Service	catalog
• OpenShift PaaS
• other powerful	APIs
≫ Goal:	allow powerful	extensions without modifying Kubernetes itself
k8s.io/apiserver – since 1.6
• generic apiserver library in	Go
• today used inside
• kube-apiserver
• apiextensions-apiserver
• federation-apiserver
• service catalog
• OpenShift 3.6+
• allows creation of custom apiservers in	a	couple hundred lines of code
• each custom apiserver is its own process,	communicating via	HTTPS
• delegates authentication/authorization to kube-apiserver
• uses etcd storage (possibly shared with kube)
kube-apiserver service catalog PaaS
kube-aggregator
kube-apiserver service catalog PaaS
kube-aggregator
API-1 API-2 API-3
API	1+2+3
Pods Jobs ... announcement ... build test project
kube-apiserver kube-apiserver kube-apiserver
federation
apiserver
API API API
API
Pods EU	replicaset Pods US	replicaset Pods A	replicaset
deployment service
Europe US Asia
Not	this:	
federation
controllers⟲
federated resources:
availability zones +	regions:
kube-apiserver service catalog PaaS
kube-aggregator
discoverydiscovery
kube-apiserver service catalog PaaS
kube-aggregator
GET
GET	/apis/servicecatalog/subscription/database-prod-wordpress
kube-apiserver service catalog PaaS
kube-aggregator
GET	/apis/servicecatalog/subscription/database-prod-wordpress
GET
user:	meetup
roles:	admin
RBAC Namespace
kube-apiserver service catalog PaaS
kube-aggregator
GET
Vision:	
$ helm install service-catalog
$ kubectl create service-announcement ....
GET	/apis/servicecatalog/subscription/database-prod-wordpress
kube-apiserver service catalog PaaS
kube-aggregator
GET
Vision:	
$ helm install service-catalog
$ kubectl create service-announcement ....
GET	/apis/servicecatalog/subscription/database-prod-wordpress
apiextensions-
apiserver
service catalog PaaS
kube-aggregator
GET
Vision:	
$ helm install service-catalog
$ kubectl create service-announcement ....
GET	/apis/servicecatalog/subscription/database-prod-wordpress
kube-apiserver
apiVersion:	v1
kind:	Service
metadata:
name:	sample-apiserver
namespace:	default
spec:
ports:		
- port:	8443
protocol:	TCP
targetPort:	8443
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:		
name:	v1alpha1.wardle.k8s.io
spec:
group:	wardle.k8s.io
version: v1alpha1
priority: 200
service:				
name: sample-apiserver
namespace: default
$	http	--cert client-admin.crt --cert-key client-admin.key --verify no
http://localhost:8080/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders
Demo
• https://github.com/openshift/kube-projects:
$ ALLOW_ANY_TOKEN=true ENABLE_RBAC=true hack/local-up-cluster.sh
$ kubectl create ns project-openshift-io
$ kubectl create -f	https://.../apiregistration.k8s.io.yaml
$ kubectl create -f	https://.../rbac.authorization.k8s.io.yaml
$ kubectl create -f	https://.../core.k8s.io.yaml
$ kubectl get projects
Links
• https://docs.google.com/document/d/1y16jKL2hMjQO0trYBJJSczPA
Wj8vAgNFrdTZeCincmI/
Two Ways to Extend the K8s	API - Add	resources to a	Kubernetes API	
with TPR	or AA
• https://github.com/kubernetes/community/blob/master/contributor
s/design-proposals/aggregated-api-servers.md
• https://gist.github.com/philips/a97a143546c87b86b870a82a753db1
4c - Kubernetes Third-Party	Resource Users
https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md
Backup
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alpha1/jobs
/apis/batch/v2alpha1/cronjo
/apis/batch/v1beta1
/apis/batch/v1beta1/jobs
/apis/batch/v2alpha1/jobs
Group			Version			ResourceHTTP	paths:
In	Go: gvk :=	schema.GroupVersionKind{Group:	“batch“,	Version:	“v2alpha1“,	Kind:	“Job“}
obj :=	api.Scheme.New(gvk)
codec :=	api.Codecs.LegacyCodec(gvk.GroupVersion())
codec.Decode(reqBody,	gvk,	obj)
type	Job struct {
metav1.TypeMeta
metav1.ObjectMeta
Spec JobSpec
Status	JobStatus
}
pkg/apis/batch/v2alpha1/types.go
type	TypeMeta struct {
Kind	string
APIVersion string
}
type	ObjectMeta struct {
Name	string
...
}
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alpha1/jobs
/apis/batch/v2alpha1/cronjo
/apis/batch/v1beta1
/apis/batch/v1beta1/jobs
MaxInFlightLimit
TimeoutForNonLongRunningRequests
Panic	Recovery
CORS
Authentication
Audit
Impersonation
Authorization
k8s.io/apiserver/pkg/server.
DefaultBuildHandlerChain
„Filters“
k8s.io/apiserver/pkg/server/routes/index.go – /
k8s.io/apiserver/pkg/server/routes/version.go – /version
k8s.io/apiserver/pkg/server/routes/swagger.go – /swaggerapi
k8s.io/apiserver/pkg/server/routes/openapi.go – /swagger.json
„Routes“
mux
k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST
AddSupportedResourcesWebService – /apis/batch/v2alpha1
k8s.io/apiserver/pkg/endpoints.APIInstaller.Install
/apis/batch/v2alpha1/jobs
/apis/batch/v2alpha1/cronjobs
...
WithRequestInfo
ctx.RequestInfo
Restful
http	API
mux
k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST
AddSupportedResourcesWebService – /apis/batch/v2alpha1
k8s.io/apiserver/pkg/endpoints.APIInstaller.Install
/apis/batch/v2alpha1/jobs
/apis/batch/v2alpha1/cronjobs
...
pkg/apis/batch
type Jobs	struct
pkg/apis/batch/v2alpha1
type Jobs	struct
api.Scheme
k8s.io/apiserver
pkg/api
api.Scheme.Convert(&internalJob,	&v2alohaJob)
/apis/batch/v2alpha1/jobs
GET
PUT
POST
DELETE
...
/status
/scale
/proxy
...
subresources
Restful
http	API
mux
pkg/apis/batch
type Jobs	struct
pkg/apis/batch/v2alp
ha1
type Jobs	struct
api.Sche
me api.Scheme.Convert(&job,	&v1job)
POST
/apis/batch/v2alpha1/jobs
k8s.io/apiserver
pkg/endpoints/handlers.CreateNamedResource
binary
JSON
payload
Go	struct
v2alpha1.Job
HTTP
Request
Go	struct
internal.Job
Store
k8s.io/apiserver
pkg/registry/generic
Storage
k8s.io/apiserver
pkg/storage/etcd3
ProtoBuf
Job
Go	struct
v2alpha1.Job
etcd
type Scheme struct
• AddKnownTypes(gv, obj Object)
• Default(src Object)
• Copy(src Object) Object
• Convert(in, out interface{})
• New(gvk) Object
ApiGroup
pkg/apis/batch
pkg/apis/batch/v1
pkg/apis/batch/v2alpha1
pkg/apis/batch/register.go
pgk/apis/batch/install
Group
Version
Kind
Resource
type Object interface
• GetObjectKind() string
client-go/pkg/api.Scheme
client-go/pkg/api.Codecs
Discovery
type APIGroupList struct
type APIVersions struct
type APIResourceList struct
GroupVersionKind „gvk“
GroupVersionResource
Unversioned types
Unstructured
List
Registry / Storage
type Storage interface
type Lister interface
type Updater interface
type Getter interface
type Deleter interface
....
deepcopy-gen
conversion-gen
defaulting-gen
Code Generation
type OwnerReference struct
type ObjectReference struct
type TypeMeta struct
type ObjectMeta struct
Meta
api.Scheme
api.Codecs
api.Registry
api.GroupFactoryRegistry
Globals

Extending the Kube API