Downloaded 38 times
![21
OTEL-Collector configuration
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
mode: deployment
config: |
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:14250
exporters:
logging:
loglevel: debug
service:
extensions: [headers_setter, ...]
pipelines:
traces:
processors: [batch,...]
receivers: [otlp]
exporters: [logging]
# Data src: traces, metrics, logs
processors: # <optional>
batch:
send_batch_size: 10000
timeout: 10s
resourcedetection/openshift:
detectors: openshift
timeout: 2s
override: false
extensions: # <optional>
headers_setter:
- key: X-Scope-OrgID
from_context: tenant_id
- key: User-ID
value: user_id
sidecar.opentelemetry.io/inject: "true"](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/85/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-21-320.jpg)
![22
OpenTelemetry Operator - Collector Kind
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
config: |
receivers:
otlp:
protocols:
grpc:
http:
jaeger:
protocols:
grpc:
thrift_compact:
processors:
exporters:
jaeger:
endpoint: jaeger-collector.headless:14250
service:
pipelines:
traces:
receivers: [otlp, jaeger]
processors: []
exporters: [jaeger]
EOF
sidecar.opentelemetry.io/inject: "true"](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/85/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-22-320.jpg)
![35
OTEL-Collector configuration
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
mode: statefulset
targetAllocator:
enabled: true
allocationStrategy: least-weighted
replicas: 3
serviceAccount: ta
prometheusCR: {}
config: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: ‘otel-collector’
scrape_interval: 10s
static_configs:
- targets: [`0.0.0.0:8888`]
target_allocator:
endpoint: http://my-ta-svc
interval: 30s
collector_id: collector-1](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/85/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-35-320.jpg)
![37
OTEL-Collector configuration
Be aware of:
● Prometheus does not support UTF-8 labels
[prometheus#11700] [grafana#42615]
Therefore OTEL-Prometheus-Remote-Write converts metrics to Prom
Naming convention. [otel#normalization]
● Other useful metric receiver
[host-metrics]](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/85/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-37-320.jpg)
![43
OpenTelemetry Operator - collecting logs, logging exporter 1
2023-01-19T16:28:48.675Z info ResourceLog #0
Resource SchemaURL:
Resource attributes:
-> k8s.namespace.name: Str(cert-manager)
-> k8s.pod.name: Str(cert-manager-cainjector-857ff8f7cb-xmr6p)
-> k8s.container.restart_count: Str(0)
-> k8s.pod.uid: Str(53574bad-1d91-4880-ab87-8f5a88bfffee)
-> k8s.container.name: Str(cert-manager)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2023-01-19 16:28:48.674124625 +0000 UTC
Timestamp: 2023-01-18 12:58:21.446092114 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str({"log":"I0118 12:58:21.445823 1 controller.go:178]
cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcede
finition "msg"="Starting EventSource" "source"="u0026{{%!s(*v1.Certificate=u0026{{ } { 0
{{0 0 u
003cnilu003e}} u003cnilu003e u003cnilu003e map[] map[] [] [] []} {u003cnilu003e
u003cnilu003e u003cnilu003e [] [] [] [] u003cnilu003e u003cnilu003e { } false []
u003cnilu003e u003cnilu003e u003cnilu003e []} {[] u003cnilu003e u003cnilu003e u003cnilu00
3e u003cnilu003e u003cnilu003e u003cnilu003e u003cnilu003e}})
%!s(*cache.informerCache=u0026{0xc000438380}) %!s(chan error=u003cnilu003e)
%!s(func()=u003cnilu003e)}}"n","stream":"stderr","time":"2023-01-18T12:58:21.446092114Z"})](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/85/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-43-320.jpg)
![44
OpenTelemetry Operator - collecting logs, logging exporter 2
Attributes:
-> log.file.path:
Str(/var/log/pods/cert-manager_cert-manager-cainjector-857ff8f7cb-xmr6p_53574bad-1d91-4880-ab
87-8f5a88bfffee/cert-manager/0.log)
-> log: Str(I0118 12:58:21.445823 1 controller.go:178]
cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-custo
mresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } {
0 {{0 0 <nil>}} <nil> <nil>
map[] map[] [] [] []} {<nil> <nil> <nil> [] [] [] [] <nil> <nil> { } false [] <nil>
<nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}})
%!s(*cache.informerCache=&{0xc000438380}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
)
-> time: Str(2023-01-18T12:58:21.446092114Z)
-> log.iostream: Str(stderr)
Trace ID:
Span ID:
Flags: 0](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/85/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-44-320.jpg)
![21
OTEL-Collector configuration
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
mode: deployment
config: |
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:14250
exporters:
logging:
loglevel: debug
service:
extensions: [headers_setter, ...]
pipelines:
traces:
processors: [batch,...]
receivers: [otlp]
exporters: [logging]
# Data src: traces, metrics, logs
processors: # <optional>
batch:
send_batch_size: 10000
timeout: 10s
resourcedetection/openshift:
detectors: openshift
timeout: 2s
override: false
extensions: # <optional>
headers_setter:
- key: X-Scope-OrgID
from_context: tenant_id
- key: User-ID
value: user_id
sidecar.opentelemetry.io/inject: "true"](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/75/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-21-2048.jpg)
![22
OpenTelemetry Operator - Collector Kind
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
config: |
receivers:
otlp:
protocols:
grpc:
http:
jaeger:
protocols:
grpc:
thrift_compact:
processors:
exporters:
jaeger:
endpoint: jaeger-collector.headless:14250
service:
pipelines:
traces:
receivers: [otlp, jaeger]
processors: []
exporters: [jaeger]
EOF
sidecar.opentelemetry.io/inject: "true"](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/75/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-22-2048.jpg)
![35
OTEL-Collector configuration
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
spec:
mode: statefulset
targetAllocator:
enabled: true
allocationStrategy: least-weighted
replicas: 3
serviceAccount: ta
prometheusCR: {}
config: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: ‘otel-collector’
scrape_interval: 10s
static_configs:
- targets: [`0.0.0.0:8888`]
target_allocator:
endpoint: http://my-ta-svc
interval: 30s
collector_id: collector-1](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/75/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-35-2048.jpg)
![37
OTEL-Collector configuration
Be aware of:
● Prometheus does not support UTF-8 labels
[prometheus#11700] [grafana#42615]
Therefore OTEL-Prometheus-Remote-Write converts metrics to Prom
Naming convention. [otel#normalization]
● Other useful metric receiver
[host-metrics]](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/75/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-37-2048.jpg)
![43
OpenTelemetry Operator - collecting logs, logging exporter 1
2023-01-19T16:28:48.675Z info ResourceLog #0
Resource SchemaURL:
Resource attributes:
-> k8s.namespace.name: Str(cert-manager)
-> k8s.pod.name: Str(cert-manager-cainjector-857ff8f7cb-xmr6p)
-> k8s.container.restart_count: Str(0)
-> k8s.pod.uid: Str(53574bad-1d91-4880-ab87-8f5a88bfffee)
-> k8s.container.name: Str(cert-manager)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2023-01-19 16:28:48.674124625 +0000 UTC
Timestamp: 2023-01-18 12:58:21.446092114 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str({"log":"I0118 12:58:21.445823 1 controller.go:178]
cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcede
finition "msg"="Starting EventSource" "source"="u0026{{%!s(*v1.Certificate=u0026{{ } { 0
{{0 0 u
003cnilu003e}} u003cnilu003e u003cnilu003e map[] map[] [] [] []} {u003cnilu003e
u003cnilu003e u003cnilu003e [] [] [] [] u003cnilu003e u003cnilu003e { } false []
u003cnilu003e u003cnilu003e u003cnilu003e []} {[] u003cnilu003e u003cnilu003e u003cnilu00
3e u003cnilu003e u003cnilu003e u003cnilu003e u003cnilu003e}})
%!s(*cache.informerCache=u0026{0xc000438380}) %!s(chan error=u003cnilu003e)
%!s(func()=u003cnilu003e)}}"n","stream":"stderr","time":"2023-01-18T12:58:21.446092114Z"})](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/75/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-43-2048.jpg)
![44
OpenTelemetry Operator - collecting logs, logging exporter 2
Attributes:
-> log.file.path:
Str(/var/log/pods/cert-manager_cert-manager-cainjector-857ff8f7cb-xmr6p_53574bad-1d91-4880-ab
87-8f5a88bfffee/cert-manager/0.log)
-> log: Str(I0118 12:58:21.445823 1 controller.go:178]
cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-custo
mresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } {
0 {{0 0 <nil>}} <nil> <nil>
map[] map[] [] [] []} {<nil> <nil> <nil> [] [] [] [] <nil> <nil> { } false [] <nil>
<nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}})
%!s(*cache.informerCache=&{0xc000438380}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
)
-> time: Str(2023-01-18T12:58:21.446092114Z)
-> log.iostream: Str(stderr)
Trace ID:
Span ID:
Flags: 0](https://image.slidesharecdn.com/devnation2023exploringthepowerofopentelemetryonkubernetes-230126194106-fdcb0ed5/75/Exploring-the-power-of-OpenTelemetry-on-Kubernetes-44-2048.jpg)
Uploaded byRed Hat Developers
Exploring the power of OpenTelemetry on Kubernetes
The document discusses the OpenTelemetry project and its implementation in Kubernetes, focusing on data collection for traces, metrics, and logs through various instrumentation techniques and the OpenTelemetry Kubernetes operator. It outlines the history of observability technologies, the architecture for data collection, and examples of configurations for collectors and instrumentations. The discussion concludes with future prospects for OpenTelemetry, including auto-instrumentation and profiling capabilities.
More Related Content
Similar to Exploring the power of OpenTelemetry on Kubernetes
![UiPath Automation Suite Installation (Hands-On) [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/automationsuitecommunitysession2-251015095633-a6d862f1-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Exploring the power of OpenTelemetry on Kubernetes
- 1. 1 Exploring the powerof OpenTelemetry on Kubernetes Pavol Loffay Principal Software Engineer Benedikt Bongartz Senior Software Engineer
- 2. 2 ● Pavol Loffay ●Principal Software Engineer ● OpenTelemetry, Jaeger maintainer ● Follow me @ploffay Bio
- 3. 3 ● Benedikt Bongartz ●Senior Software Engineer ● Working on OpenTelemetry ● Contact ○ @frzifus@fosstodon.org ○ frzifus:matrix.org Bio
- 4. 4 ● History observability ●Intro to OpenTelemetry (OTEL) ○ Instrumentation ○ Collector ● OpenTelemetry Kubernetes operator ○ Instrumentation CR ○ Collector CR ● Data collection use-cases - Traces, Metrics, Logs ● What is next? Agenda
- 5. A peek intohistory of OSS observability technology advancement and less vendor locking
- 6. 6 OSS distributed tracinghistory X-Trace 2007 Dapper paper 2010 Zipkin 2012 OpenTracing 2015 Jaeger 2017 OpenTelemetry DataDog 2019 New Relic 2020 OpenCensus 2016 Hypetrace Grafana Tempo 2020 SigNoz 2021 Apache SkyWalking 2015 Data model bespoke APIs SDK Collector auto-instrumentation Spec API Instrumentation libraries some instrumentation reusable instrumentation libraries Agents/ auto-instrumentation
- 7. OpenTelemetry a.k.a. OTEL Open-sourcedata collection project
- 8. 8 ● Open source! ●Cloud Native Computing Foundation (CNCF) ● Vendor neutral telemetry data collection ● Specification, API, SDK, data model - OTLP, auto-instrumentation, collector ● Helm chart, Kubernetes operator
- 9. 9 OpenTelemetry is onlydata collection It is not platform, storage nor query API
- 10. OpenTelemetry instrumentation How telemetrydata is created
- 11.
- 12. ● Manual /Explicit ○ Better control ○ Might have better performance ○ Requires code change, recompilation, redeploy ● Direct integration in runtimes - e.g. Quarkus, Wildfly ● Auto-instrumentation / agent ○ Easy to roll out ○ Wide framework coverage 12 Instrumentation java -javaagent:/opt/javaagent.jar -jar target/demo-0.0.1-SNAPSHOT.jar tracing.enabled = true io.opentelemetry:opentelemetry-api:1.22.0 io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:1.19.1
- 13. ● Java ● C# ●NodeJS ● Python ● PHP ● Golang 13 OpenTelemetry auto-instrumentation java -javaagent:/opt/javaagent.jar -jar target/demo-0.0.1-SNAPSHOT.jar JAVA_TOOL_OPTIONS = -javaagent:/otel-auto-instrumentation/javaagent.jar CORECLR_ENABLE_PROFILING, CORECLR_PROFILER_PATH, DOTNET_ADDITIONAL_DEPS, DOTNET_SHARED_STORE NODE_OPTIONS = --require /otel-auto-instrumentation/autoinstrumentation.js PYTHONPATH = /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation//otel-auto-instrumentation
- 14. OpenTelemetry collector How telemetrydata is collected and exported to a platform
- 15. ● Written inGolang ● Docs - opentelemetry.io/docs/collector/ ● Two distributions ○ Core - opentelemetry-collector-releases/distributions/otelcol ○ Contrib - opentelemetry-collector-releases/distributions/otelcol-contrib ○ Build your own ● Many components open-telemetry/opentelemetry-collector-contrib ● Distributed as: binary, container image, APK/DEB/RPM 15 OpenTelemetry collector
- 16. 16 Instrumentation and collector app/workload/pod (auto)Instrumentation API/SDK RPC framework (e.g. servlet) Jaeger OpenTelemetry Zipkin OpenCensus Batching PII filter Re-labeling Custom Splunk Kafka Jaeger Batching Data drop Collector Receivers Processors Exporters
- 17. OpenTelemetry Kubernetes Operator ManagesOpenTelemetry collector and Instrumentation
- 18. ● https://github.com/open-telemetry/opentelemetry-operator ● Kubernetesoperator Custom Resource Definitions (CRDs): ○ opentelemetrycollectors.opentelemetry.io / otelcol ○ instrumentation.opentelemetry.io / otelinst ● The operator can be installed via OLM, Helm chart or K8s manifests 18 OpenTelemetry operator
- 19.
- 20. ● Deployment modes:.spec.mode ○ sidecar, deployment, daemonset, statefulset ● Auto scaling or kubectl scale --replicas=5 otelcols/simplest ● Expose outside of the cluster .spec.ingress ● Use custom collector image .spec.image 20 OpenTelemetry Operator - Collector Kind
- 21. 21 OTEL-Collector configuration apiVersion: opentelemetry.io/v1alpha1 kind:OpenTelemetryCollector metadata: name: otel spec: mode: deployment config: | receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:14250 exporters: logging: loglevel: debug service: extensions: [headers_setter, ...] pipelines: traces: processors: [batch,...] receivers: [otlp] exporters: [logging] # Data src: traces, metrics, logs processors: # <optional> batch: send_batch_size: 10000 timeout: 10s resourcedetection/openshift: detectors: openshift timeout: 2s override: false extensions: # <optional> headers_setter: - key: X-Scope-OrgID from_context: tenant_id - key: User-ID value: user_id sidecar.opentelemetry.io/inject: "true"
- 22. 22 OpenTelemetry Operator -Collector Kind kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: config: | receivers: otlp: protocols: grpc: http: jaeger: protocols: grpc: thrift_compact: processors: exporters: jaeger: endpoint: jaeger-collector.headless:14250 service: pipelines: traces: receivers: [otlp, jaeger] processors: [] exporters: [jaeger] EOF sidecar.opentelemetry.io/inject: "true"
- 23. 23 OpenTelemetry Operator -Collector Kind - Processor processors: k8sattributes: # Add k8s attributes attributes: # Delete db.table attribute actions: - key: db.table action: delete resource: # Add k8s cluster name attributes: - key: k8s.cluster.name from_attribute: k8s-cluster action: insert Other use cases: ● Tail base sampling ● Derive RED metrics from traces
- 24.
- 25. 25 OpenTelemetry Operator -Instrumentation Kind apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: exporter: endpoint: http://otel-collector:4317 propagators: - tracecontext - baggage - b3 sampler: type: parentbased_traceidratio argument: "0.25" resource: attributes: k8s.cluster.name: test-1-23 instrumentation.opentelemetry.io/inject-java: "true" instrumentation.opentelemetry.io/inject-python: "true" instrumentation.opentelemetry.io/inject-nodejs: "true" instrumentation.opentelemetry.io/inject-dotnet: "true" instrumentation.opentelemetry.io/inject-sdk: "true" instrumentation.opentelemetry.io/container-names: "app"
- 26.
- 27. 27 OpenTelemetry Architecture -Tracing Pod Container 1 OTEL-Client OpenTelemetry API Instrumentation Application push Platform (Jaeger/Loki/Splunk…) Platform instrumentation.opentelemetry.io/inject-java: "true" apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my–java-instrumentation spec: exporter: . . .
- 28.
- 29.
- 30.
- 31.
- 32.
- 33. 33 Collector Platform (Prometheus/Thanos/Splunk…) Platform Pod 2 Node 1 OTEL-Collectorconfiguration Pod 1 Pull Push
- 34. 34 OTEL-Collector configuration apiVersion: opentelemetry.io/v1alpha1 kind:OpenTelemetryCollector metadata: name: otel spec: targetAllocator: enabled: true allocationStrategy: least-weighted replicas: 3 serviceAccount: ta prometheusCR: enabled: true serviceMonitorSelector: team: backend-1 config: | receivers: prometheus: config: scrape_configs: - job_name: ‘otel-collector’ scrape_interval: 10s static_configs: ...
- 35. 35 OTEL-Collector configuration apiVersion: opentelemetry.io/v1alpha1 kind:OpenTelemetryCollector metadata: name: otel spec: mode: statefulset targetAllocator: enabled: true allocationStrategy: least-weighted replicas: 3 serviceAccount: ta prometheusCR: {} config: | receivers: prometheus: config: scrape_configs: - job_name: ‘otel-collector’ scrape_interval: 10s static_configs: - targets: [`0.0.0.0:8888`] target_allocator: endpoint: http://my-ta-svc interval: 30s collector_id: collector-1
- 36. 36 OTEL-Collector configuration … receivers: prometheus: config: global: scrape_interval: 1m scrape_timeout:10s evaluation_interval: 1m scrape_configs: - job_name: otel-collector honor_timestamps: true scrape_interval: 10s scrape_timeout: 10s metrics_path: /metrics scheme: http follow_redirects: true http_sd_configs: - follow_redirects: false url: http://metrics-ta-svc:80/jobs/otel-collector/targets?collector_id=$POD_NAME spec.targetAllocator.enabeld: "true"
- 37. 37 OTEL-Collector configuration Be awareof: ● Prometheus does not support UTF-8 labels [prometheus#11700] [grafana#42615] Therefore OTEL-Prometheus-Remote-Write converts metrics to Prom Naming convention. [otel#normalization] ● Other useful metric receiver [host-metrics]
- 38.
- 39. 39 ● For metricsand traces OpenTelemetry takes the approach of a clean-sheet design, specifies a new API and provides full implementations of this API in multiple languages. ● Our approach with logs is somewhat different. For OpenTelemetry to be successful in logging space we need to support existing legacy of logs and logging libraries, while offering improvements and better integration with the rest of observability world where possible. OpenTelemetry logs
- 40. 40 OpenTelemetry Operator -collecting logs ● File log receiver ○ Available in the “contrib” docker image ○ Example config - opentelemetry-collector-contrib/otel-collector-config.yml ○ Includes: /var/log/pods/*/*/*.log ○ Set of operators to parse data: json_parser, regex_parser, move… ○ Collectors resource attributes: namespace, pod name/UID/restarts, container name ● Fluentforwardreceiver - receive data from
- 41. 41 DaemonSet collector Platform (Loki/OpenSearch/Splunk…) Platform Pod 1 Node 1 Pod2 Pod 3 DaemonSet collector Pod 1 Node 2 Pod 2 Pod 3 DaemonSet collector Pod 1 Node 3 Pod 2 Pod 3 Kubernetes cluster
- 42. 42 OpenTelemetry Operator -collecting logs, CR apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel-logs spec: mode: daemonset image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0. 69.0 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers volumeMounts: - mountPath: /var/log name: varlog readOnly: true - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true
- 43. 43 OpenTelemetry Operator -collecting logs, logging exporter 1 2023-01-19T16:28:48.675Z info ResourceLog #0 Resource SchemaURL: Resource attributes: -> k8s.namespace.name: Str(cert-manager) -> k8s.pod.name: Str(cert-manager-cainjector-857ff8f7cb-xmr6p) -> k8s.container.restart_count: Str(0) -> k8s.pod.uid: Str(53574bad-1d91-4880-ab87-8f5a88bfffee) -> k8s.container.name: Str(cert-manager) ScopeLogs #0 ScopeLogs SchemaURL: InstrumentationScope LogRecord #0 ObservedTimestamp: 2023-01-19 16:28:48.674124625 +0000 UTC Timestamp: 2023-01-18 12:58:21.446092114 +0000 UTC SeverityText: SeverityNumber: Unspecified(0) Body: Str({"log":"I0118 12:58:21.445823 1 controller.go:178] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcede finition "msg"="Starting EventSource" "source"="u0026{{%!s(*v1.Certificate=u0026{{ } { 0 {{0 0 u 003cnilu003e}} u003cnilu003e u003cnilu003e map[] map[] [] [] []} {u003cnilu003e u003cnilu003e u003cnilu003e [] [] [] [] u003cnilu003e u003cnilu003e { } false [] u003cnilu003e u003cnilu003e u003cnilu003e []} {[] u003cnilu003e u003cnilu003e u003cnilu00 3e u003cnilu003e u003cnilu003e u003cnilu003e u003cnilu003e}}) %!s(*cache.informerCache=u0026{0xc000438380}) %!s(chan error=u003cnilu003e) %!s(func()=u003cnilu003e)}}"n","stream":"stderr","time":"2023-01-18T12:58:21.446092114Z"})
- 44. 44 OpenTelemetry Operator -collecting logs, logging exporter 2 Attributes: -> log.file.path: Str(/var/log/pods/cert-manager_cert-manager-cainjector-857ff8f7cb-xmr6p_53574bad-1d91-4880-ab 87-8f5a88bfffee/cert-manager/0.log) -> log: Str(I0118 12:58:21.445823 1 controller.go:178] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-custo mresourcedefinition "msg"="Starting EventSource" "source"="&{{%!s(*v1.Certificate=&{{ } { 0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] [] []} {<nil> <nil> <nil> [] [] [] [] <nil> <nil> { } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc000438380}) %!s(chan error=<nil>) %!s(func()=<nil>)}}" ) -> time: Str(2023-01-18T12:58:21.446092114Z) -> log.iostream: Str(stderr) Trace ID: Span ID: Flags: 0
- 45.
- 46. ● OpenTelemetry Operator ○Auto-instrumentation for Webservers ● Profiling vision see 0212-profiling-vision.md ● OpAMP: Open Agent Management Protocol see open-telemetry/opamp-spec 46 What is next in OpenTelemetry?
- 47. Thank you 47 Pavol Loffay @ploffay BenediktBongartz @frzifus