Observability on Board

Episode 11traces

The Kubernetes API server already traces every request

The Kubernetes API server ships with OpenTelemetry tracing. A three-line file and one flag in its static pod manifest turn every request into a trace.

Ship telemetry toUse an OTLP endpoint you already have, from another vendor or your platform team. This one needs a one-time setup step: save your endpoint and the episodes fill it in.Set your endpointRun an OpenTelemetry Collector on your machine and watch telemetry arrive in its terminal. No account needed.More detailsShip straight to Bronto. You need an account and an API key with ingest permission.More details

Switch it on

Run a Collector on the control-plane node, then add /etc/kubernetes/tracing.yaml there and point the API server at it in its static pod manifest. Saving the manifest restarts the API server, and every request after that becomes a trace that includes each admission webhook it waited on. Needs Kubernetes 1.34 or newer.

Save this as otel-collector.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector
  namespace: kube-system
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: localhost:4317
    exporters:
      otlp_http:
        endpoint: https://ingestion.eu.bronto.io
        headers:
          x-bronto-api-key: ${env:BRONTO_API_KEY}
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp_http]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      hostNetwork: true
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.158.0
        args: [--config=/etc/otelcol/config.yaml]
        envFrom:
        - secretRef:
            name: bronto
        volumeMounts:
        - name: config
          mountPath: /etc/otelcol
      volumes:
      - name: config
        configMap:
          name: otel-collector

In a terminal:

kubectl -n kube-system create secret generic bronto \
  --from-literal=BRONTO_API_KEY="$BRONTO_API_KEY"
kubectl apply -f otel-collector.yaml

On the node, /etc/kubernetes/tracing.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: TracingConfiguration
samplingRatePerMillion: 1000000

In /etc/kubernetes/manifests/kube-apiserver.yaml:

    command:
    - kube-apiserver
+   - --tracing-config-file=/etc/kubernetes/tracing.yaml
  …
    volumeMounts:
+   - mountPath: /etc/kubernetes/tracing.yaml
+     name: tracing
  …
  volumes:
+ - hostPath:
+     path: /etc/kubernetes/tracing.yaml
+   name: tracing

On the node, /etc/kubernetes/tracing.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: TracingConfiguration
samplingRatePerMillion: 1000000

In /etc/kubernetes/manifests/kube-apiserver.yaml:

    command:
    - kube-apiserver
+   - --tracing-config-file=/etc/kubernetes/tracing.yaml
  …
    volumeMounts:
+   - mountPath: /etc/kubernetes/tracing.yaml
+     name: tracing
  …
  volumes:
+ - hostPath:
+     path: /etc/kubernetes/tracing.yaml
+   name: tracing

This assumes the Local Collector running on the control-plane node, with a grpc receiver on localhost:4317 in place of http.

This uses the endpoint and auth header you saved in the setup guide.

Save this as otel-collector.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector
  namespace: kube-system
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: localhost:4317
    exporters:
      otlp_http:
        endpoint: YOUR_OTLP_ENDPOINT
        headers:
          YOUR_AUTH_HEADER: YOUR_AUTH_VALUE
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp_http]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      hostNetwork: true
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.158.0
        args: [--config=/etc/otelcol/config.yaml]
        volumeMounts:
        - name: config
          mountPath: /etc/otelcol
      volumes:
      - name: config
        configMap:
          name: otel-collector

In a terminal:

kubectl apply -f otel-collector.yaml

On the node, /etc/kubernetes/tracing.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: TracingConfiguration
samplingRatePerMillion: 1000000

In /etc/kubernetes/manifests/kube-apiserver.yaml:

    command:
    - kube-apiserver
+   - --tracing-config-file=/etc/kubernetes/tracing.yaml
  …
    volumeMounts:
+   - mountPath: /etc/kubernetes/tracing.yaml
+     name: tracing
  …
  volumes:
+ - hostPath:
+     path: /etc/kubernetes/tracing.yaml
+   name: tracing