KEMBAR78
Kubernetes API code-base tour | PDF
API Codebase Tour
Stefan Schimanski / @the_sttts / Red Hat
Hacking the kube-apiserver
Defining API types
v1alpha1 types: staging/src/k8s.io/api/auditregistration/v1alpha1
• types.go – actual Golang types (with JSON and Proto tags)
• register.go – registration code: AddToScheme
internal types: pkg/apis/auditregistration
• types.go – internal (hub) Golang types (without JSON/Proto)
• register.go – registration code: AddToScheme
Installer: pkg/apis/auditregistration/install:
func Install(scheme *runtime.Scheme)
Golang types
Scheme: register Golang types & Golang funcs w/ GroupVersionKind
k8s.io/apimachinery/pkg/runtime.Scheme
GroupVersionKinds
conversions
defaulters
reflect.Type
Scheme
Codec
v1alpha1 types: staging/src/k8s.io/api/auditregistration/v1alpha1
• types.go – actual Golang types (with JSON and Proto tags)
• register.go – registration code: AddToScheme
internal types: pkg/apis/auditregistration
• types.go – internal (hub) Golang types (without JSON/Proto)
• register.go – registration code: AddToScheme
Installer: pkg/apis/auditregistration/install:
func Install(scheme *runtime.Scheme)
Golang types
Conversions: pkg/apis/auditregistration/v1alpha1
• conversion.go – custom conversions
• zz_generated.conversion.go – generated conversions
Defaults: zz_generated_defaults.go
DeepCopy: zz_generated_deepcopy.go
Generated Code
not in k8s.io/api!
Serving the API
apiserver binary
generic apiserver in k8s.io/apiserver
404
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
mux
data flow
calls back to
knows no API groups yetScheme
empty
/version
/apis
/openapi/v2
/swagger.json
/healthz
/metrics
apiserver binary
generic apiserver in k8s.io/apiserver
404
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
mux
data flow
calls back to
knows no API groups yetScheme
empty
/version
/apis
/openapi/v2
/swagger.json
/healthz
/metrics
func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
handler := genericapifilters.WithAuthorization(apiHandler, ...)
handler = genericfilters.WithMaxInFlightLimit(handler, ...)
handler = genericapifilters.WithImpersonation(handler, ...)
handler = genericapifilters.WithAudit(handler, ...)
failedHandler := genericapifilters.Unauthorized(...)
failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, ...)
handler = genericapifilters.WithAuthentication(handler, ..., failedHandler, ...)
handler = genericfilters.WithCORS(handler, ...)
handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, ...)
handler = genericfilters.WithWaitGroup(handler, ...)
handler = genericapifilters.WithRequestInfo(handler, ...)
handler = genericfilters.WithPanicRecovery(handler)
return handler
}
k8s.io/apiserver/pkg/server/config.go
kube-apiserver
generic apiserver
404
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
mux
data flow
calls back to
knows no API groups yetScheme
/version
/apis
/openapi/v2
/swagger.json
/healthz
/metrics
core/v1 Podcore/v1 Podcore/v1 Pod
kube-apiserver
apiserver
404
resource handler
request
conversion&
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
mux
data flow
calls back to
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
via InstallAPIGroup(info)
kube-apiserver
apiserver
resource handlerresource handler
404
resource handler
request
conversion&
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
mux
no storage logic yet
kube-apiserver
apiserver
resource handlerresource handler
404
etcd
resource handler
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
API Group “core”API Group “core”API Group “core”
PodStoragePodStoragePodStorage
Generic Registry Pod Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...mux
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
kube-apiserver
apiserver
resource handlerresource handler
404
etcd
resource handler
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
API Group “core”API Group “core”API Group “core”
PodStoragePodStoragePodStorage
Generic Registry Pod Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...mux
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
v1
v1
v1 int int
v1
int
v1
int
v2
v1
int
int
v1
hub/internal version
kube-apiserver
apiserver
resource handlerresource handler
404
etcd
resource handler
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
API Group “core”API Group “core”API Group “core”
PodStoragePodStoragePodStorage
Generic Registry Pod Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...mux
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
conversions
defaulting
kube-apiserver
CRDs
aggregator
kube-
aggregator
&
CRDs
apiserver
resource handlerresource handler
404
etcdaggregated
apiservers
resource handler
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
API Group “core”API Group “core”API Group “core”
PodStoragePodStoragePodStorage
Generic Registry Pod Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...mux
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
kube-apiserver
kube-
aggregator
apiserver
resource handlerresource handler
404
etcdaggregated
apiservers
resource handler
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
API Group “core”API Group “core”API Group “core”
PodStoragePodStoragePodStorage
Generic Registry Pod Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...mux
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
pkg/registry
pkg/apis + k8s.io/api
k8s.io/apiserver/pkg/endpoints/handlers
k8s.io/apiserver/pkg/admission
k8s.io/apiserver/plugin/pkg/admission
plugins/pkg/admission
k8s.io/apiserver/pkg/endpoints/filters
k8s.io/kube-aggregator
k8s.io/apiextensions-apiserver
k8s.io/apiserver/pkg/storage/etcd3
k8s.io/apiserver/pkg/registry/generic
API Group “core”API Group “core”API Group “auditregistration.k8s.io”
PodStoragePodStorageAuditSinkStorage
Generic Registry AuditSink Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...
staging/src/k8s.io/apiserver/pkg/registry/generic/registry
pkg/apis/auditregistration/validation
“The registry” of a resource
Plumbing into kube-apiserver
pkg/master/import_known_versions.go
import (
_ "k8s.io/kubernetes/pkg/apis/auditregistration/install"
)
pkg/master/master.go
import (
auditregistrationrest "k8s.io/kubernetes/pkg/registry/auditregistration/rest"
)
restStorageProviders := []RESTStorageProvider{
auditregistrationrest.RESTStorageProvider{},
autoscalingrest.RESTStorageProvider{},
…
}
apiserver.InstallAPIs(…, restStorageProviders…)
legacyscheme.Scheme
installs handlers into the mux
func init()
Build system plumbing
• hack/.golint_failures
ignore lint errors due to generated code
• hack/lib/init.sh
add to KUBE_AVAILABLE_GROUP_VERSIONS,
used by many hack/ scripts
• hack/update-generated-protobuf-dockerized.sh
generate Protobuf code, independent from
KUBE_AVAILABLE_GROUP_VERSIONS for some reason
$ make WHAT=cmd/hyperkube
$ RUNTIME_CONFIG=auditregistration.k8s.io/v1alpha1=true 
hack/local-up-cluster.sh
$ kubectl get --raw /apis | grep auditregistration.k8s.io
Live Debugging
kube-apiserver
CRDs
aggregator
kube-
aggregator
&
CRDs
apiserver
resource handlerresource handler
404
etcdaggregated
apiservers
resource handler
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
API Group “core”API Group “core”API Group “core”
PodStoragePodStoragePodStorage
Generic Registry Pod Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...mux
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
Live Debugging
* perfectly written down in xmudrii’s https://xmudrii.com/posts/debugging-kubernetes/
*
kube-apiserver
CRDs
aggregator
kube-
aggregator
&
CRDs
apiserver
resource handlerresource handler
404
etcdaggregated
apiservers
resource handler
request
conversion&
defaulting
storage
conversion &
defaulting
REST logic
result
conversion
validation
admission
decoding
GET
CREATE
LIST
UPDATE
DELETE
WATCH
PATCH
encoding
mutating
webhooks
validating
webhooks
authentication
authorization
impersonation
panic recovery
request-timeout
audit
max-in-flight
handlerchain
API Group “core”API Group “core”API Group “core”
PodStoragePodStoragePodStorage
Generic Registry Pod Strategy
- PrepareForUpdate
- PrepareForCreate
- Validate
...
create
update
...mux
Scheme
core/v1 Podcore/v1 Podcore/v1 Pod
data flow
calls back to
@lavalamp’s “Live API Code Review” after the break

Kubernetes API code-base tour

  • 1.
    API Codebase Tour StefanSchimanski / @the_sttts / Red Hat Hacking the kube-apiserver
  • 2.
  • 4.
    v1alpha1 types: staging/src/k8s.io/api/auditregistration/v1alpha1 •types.go – actual Golang types (with JSON and Proto tags) • register.go – registration code: AddToScheme internal types: pkg/apis/auditregistration • types.go – internal (hub) Golang types (without JSON/Proto) • register.go – registration code: AddToScheme Installer: pkg/apis/auditregistration/install: func Install(scheme *runtime.Scheme) Golang types
  • 5.
    Scheme: register Golangtypes & Golang funcs w/ GroupVersionKind k8s.io/apimachinery/pkg/runtime.Scheme GroupVersionKinds conversions defaulters reflect.Type Scheme Codec
  • 6.
    v1alpha1 types: staging/src/k8s.io/api/auditregistration/v1alpha1 •types.go – actual Golang types (with JSON and Proto tags) • register.go – registration code: AddToScheme internal types: pkg/apis/auditregistration • types.go – internal (hub) Golang types (without JSON/Proto) • register.go – registration code: AddToScheme Installer: pkg/apis/auditregistration/install: func Install(scheme *runtime.Scheme) Golang types
  • 7.
    Conversions: pkg/apis/auditregistration/v1alpha1 • conversion.go– custom conversions • zz_generated.conversion.go – generated conversions Defaults: zz_generated_defaults.go DeepCopy: zz_generated_deepcopy.go Generated Code not in k8s.io/api!
  • 8.
  • 9.
    apiserver binary generic apiserverin k8s.io/apiserver 404 authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain mux data flow calls back to knows no API groups yetScheme empty /version /apis /openapi/v2 /swagger.json /healthz /metrics
  • 10.
    apiserver binary generic apiserverin k8s.io/apiserver 404 authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain mux data flow calls back to knows no API groups yetScheme empty /version /apis /openapi/v2 /swagger.json /healthz /metrics
  • 11.
    func DefaultBuildHandlerChain(apiHandler http.Handler,c *Config) http.Handler { handler := genericapifilters.WithAuthorization(apiHandler, ...) handler = genericfilters.WithMaxInFlightLimit(handler, ...) handler = genericapifilters.WithImpersonation(handler, ...) handler = genericapifilters.WithAudit(handler, ...) failedHandler := genericapifilters.Unauthorized(...) failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, ...) handler = genericapifilters.WithAuthentication(handler, ..., failedHandler, ...) handler = genericfilters.WithCORS(handler, ...) handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, ...) handler = genericfilters.WithWaitGroup(handler, ...) handler = genericapifilters.WithRequestInfo(handler, ...) handler = genericfilters.WithPanicRecovery(handler) return handler } k8s.io/apiserver/pkg/server/config.go
  • 12.
    kube-apiserver generic apiserver 404 authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain mux dataflow calls back to knows no API groups yetScheme /version /apis /openapi/v2 /swagger.json /healthz /metrics core/v1 Podcore/v1 Podcore/v1 Pod
  • 13.
  • 14.
    kube-apiserver apiserver resource handlerresource handler 404 resourcehandler request conversion& defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to mux no storage logic yet
  • 15.
    kube-apiserver apiserver resource handlerresource handler 404 etcd resourcehandler request conversion& defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain API Group “core”API Group “core”API Group “core” PodStoragePodStoragePodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ...mux Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to
  • 16.
    kube-apiserver apiserver resource handlerresource handler 404 etcd resourcehandler request conversion& defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain API Group “core”API Group “core”API Group “core” PodStoragePodStoragePodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ...mux Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to v1 v1 v1 int int v1 int v1 int v2 v1 int int v1 hub/internal version
  • 17.
    kube-apiserver apiserver resource handlerresource handler 404 etcd resourcehandler request conversion& defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain API Group “core”API Group “core”API Group “core” PodStoragePodStoragePodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ...mux Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to conversions defaulting
  • 18.
    kube-apiserver CRDs aggregator kube- aggregator & CRDs apiserver resource handlerresource handler 404 etcdaggregated apiservers resourcehandler request conversion& defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain API Group “core”API Group “core”API Group “core” PodStoragePodStoragePodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ...mux Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to
  • 19.
    kube-apiserver kube- aggregator apiserver resource handlerresource handler 404 etcdaggregated apiservers resourcehandler request conversion& defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain API Group “core”API Group “core”API Group “core” PodStoragePodStoragePodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ...mux Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to pkg/registry pkg/apis + k8s.io/api k8s.io/apiserver/pkg/endpoints/handlers k8s.io/apiserver/pkg/admission k8s.io/apiserver/plugin/pkg/admission plugins/pkg/admission k8s.io/apiserver/pkg/endpoints/filters k8s.io/kube-aggregator k8s.io/apiextensions-apiserver k8s.io/apiserver/pkg/storage/etcd3 k8s.io/apiserver/pkg/registry/generic
  • 20.
    API Group “core”APIGroup “core”API Group “auditregistration.k8s.io” PodStoragePodStorageAuditSinkStorage Generic Registry AuditSink Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ... staging/src/k8s.io/apiserver/pkg/registry/generic/registry pkg/apis/auditregistration/validation “The registry” of a resource
  • 21.
    Plumbing into kube-apiserver pkg/master/import_known_versions.go import( _ "k8s.io/kubernetes/pkg/apis/auditregistration/install" ) pkg/master/master.go import ( auditregistrationrest "k8s.io/kubernetes/pkg/registry/auditregistration/rest" ) restStorageProviders := []RESTStorageProvider{ auditregistrationrest.RESTStorageProvider{}, autoscalingrest.RESTStorageProvider{}, … } apiserver.InstallAPIs(…, restStorageProviders…) legacyscheme.Scheme installs handlers into the mux func init()
  • 22.
    Build system plumbing •hack/.golint_failures ignore lint errors due to generated code • hack/lib/init.sh add to KUBE_AVAILABLE_GROUP_VERSIONS, used by many hack/ scripts • hack/update-generated-protobuf-dockerized.sh generate Protobuf code, independent from KUBE_AVAILABLE_GROUP_VERSIONS for some reason
  • 23.
    $ make WHAT=cmd/hyperkube $RUNTIME_CONFIG=auditregistration.k8s.io/v1alpha1=true hack/local-up-cluster.sh $ kubectl get --raw /apis | grep auditregistration.k8s.io
  • 24.
  • 25.
    kube-apiserver CRDs aggregator kube- aggregator & CRDs apiserver resource handlerresource handler 404 etcdaggregated apiservers resourcehandler request conversion& defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain API Group “core”API Group “core”API Group “core” PodStoragePodStoragePodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ...mux Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to
  • 26.
    Live Debugging * perfectlywritten down in xmudrii’s https://xmudrii.com/posts/debugging-kubernetes/ *
  • 27.
    kube-apiserver CRDs aggregator kube- aggregator & CRDs apiserver resource handlerresource handler 404 etcdaggregated apiservers resourcehandler request conversion& defaulting storage conversion & defaulting REST logic result conversion validation admission decoding GET CREATE LIST UPDATE DELETE WATCH PATCH encoding mutating webhooks validating webhooks authentication authorization impersonation panic recovery request-timeout audit max-in-flight handlerchain API Group “core”API Group “core”API Group “core” PodStoragePodStoragePodStorage Generic Registry Pod Strategy - PrepareForUpdate - PrepareForCreate - Validate ... create update ...mux Scheme core/v1 Podcore/v1 Podcore/v1 Pod data flow calls back to
  • 28.
    @lavalamp’s “Live APICode Review” after the break