eBPF and Observability in Kubernetes: Deep Microservice Monitoring Without Code Changes

Introduction: eBPF — A Revolution in Observability

The traditional approach to observability in Kubernetes revolved around sidecar agents, code instrumentation, and heavyweight APM solutions. Every microservice required manual SDK integration, tracing configuration, and an additional agent container alongside the main one. In 2026, this approach is increasingly giving way to a revolutionary technology — eBPF (extended Berkeley Packet Filter).

eBPF allows safe, verified code to run directly in the Linux kernel without modifying its source or rebooting the system. For Kubernetes and microservices, this means deep observability at the syscall, network, and CPU level — without changing a single line of application code. eBPF-based tools collect telemetry at the kernel level and pass it to userspace with minimal overhead.

eBPF does for Linux what JavaScript did for browsers: it lets you dynamically extend kernel capabilities without changing it. — Brendan Gregg

How eBPF Works in a Kubernetes Context

In a Kubernetes environment, every Pod runs in an isolated Linux namespace (network namespace, PID namespace). eBPF programs loaded into the kernel attach to various tracing points:

• kprobes/kretprobes — intercepting kernel function calls
• tracepoints — static tracing points inside the kernel
• XDP (eXpress Data Path) — capturing network packets at the NIC level
• tc (traffic control) — intercepting traffic at the network stack level
• uprobe — tracing functions in userspace

The key advantage in a Kubernetes context is that a single eBPF agent per node (DaemonSet) covers all pods on that node. No separate sidecar container is needed for each microservice. This reduces CPU and memory overhead significantly and simplifies operational management.

eBPF programs use special data structures called maps to exchange data between the kernel and userspace. Information about system calls, network connections, latencies, and errors is collected in real time and exported in formats compatible with Prometheus, OpenTelemetry, and Grafana.

eBPF-Based Tools: Cilium, Hubble, Pixie, Tetragon

Cilium

Cilium is a CNI plugin (Container Network Interface) for Kubernetes built entirely on eBPF. It replaces traditional iptables rules with eBPF programs, delivering higher performance and more flexible network policies. Cilium supports L3/L4/L7 policies, WireGuard/IPsec encryption, and service mesh integration without Envoy sidecars.

Hubble

Hubble is an observability layer on top of Cilium. It provides full visibility into network flows between services: who communicates with whom, which HTTP endpoints are called, and where latencies and errors occur. Hubble works without modifying application code or adding sidecars — all information is extracted from Cilium's eBPF programs.

Pixie

Pixie (from New Relic, open-source) is an eBPF-based observability platform for Kubernetes. It supports automatic protocol detection (HTTP/2, gRPC, MySQL, PostgreSQL, Redis), request tracing, and CPU profiling. It uses its own PxL query language, similar to Python/pandas. Pixie runs entirely in-cluster and does not send data outside by default.

Tetragon

Tetragon by Isovalent is a runtime security and observability tool built on eBPF. It provides visibility at the syscall, process execution, network connections, and file access level. It can enforce security policies directly in the kernel, blocking suspicious actions in real time.

Comparing the tools: Cilium + Hubble is the best choice for network observability and policies; Pixie is ideal for a quick full-stack start with no configuration; Tetragon is best for security scenarios and compliance. In production in 2026, all three are often used together.

Hands-On: Installing Cilium and Hubble in a Kubernetes Cluster

Installing Cilium via Helm is the standard approach for a production cluster. Make sure your Linux kernel version is at least 5.4 (5.15+ is recommended).

# Add the Cilium Helm repository
helm repo add cilium https://helm.cilium.io/
helm repo update

# Install Cilium with Hubble enabled
helm install cilium cilium/cilium \
  --version 1.15.0 \
  --namespace kube-system \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2}" \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=YOUR_API_SERVER_IP \
  --set k8sServicePort=6443

After installation, verify the Cilium status:

# Check agent status
kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium

# Check via cilium CLI
cilium status --wait

# Check Hubble
cilium hubble port-forward &
hubble status
hubble observe --follow

The Hubble UI is accessible via port-forward:

kubectl port-forward -n kube-system svc/hubble-ui 12000:80

In your browser at http://localhost:12000, you'll see a real-time service dependency graph: which pods are communicating, on which ports, and with which HTTP response codes. This works for any microservices without any changes to their code.

Example L7 NetworkPolicy via Cilium CRD:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-api-to-backend
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend-service
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: api-gateway
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/v1/.*"
        - method: "POST"
          path: "/api/v1/orders"

Profiling Go Microservices with eBPF: Continuous Profiling

Profiling Go services in production has historically been painful: the built-in pprof required explicitly enabling an HTTP endpoint, and continuous profiling consumed significant resources. eBPF changes this fundamentally.

Parca and Pyroscope (now part of the Grafana Stack as Grafana Pyroscope) use eBPF for continuous CPU profiling without code instrumentation. They periodically capture stack traces via eBPF programs attached to perf_event and build flame graphs in real time.

Installing Pyroscope via Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install pyroscope grafana/pyroscope \
  --namespace monitoring \
  --create-namespace \
  --set pyroscope.components.querier.resources.limits.memory=512Mi

Installing Grafana Alloy (agent) with eBPF profiling:

helm install alloy grafana/alloy \
  --namespace monitoring \
  --set alloy.configMap.content='
pyroscope.ebpf "ebpf_profiler" {
  targets = discovery.kubernetes.pods.targets
  forward_to = [pyroscope.write.default.receiver]
}

pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

discovery.kubernetes "pods" {
  role = "pod"
}'

For Go services, the eBPF profiler automatically unwinds call stacks using DWARF debug info or frame pointers. Starting with Go 1.21, frame pointers are enabled by default, which significantly improves profile quality. Flame graphs in Grafana let you see which functions in a microservice consume the most CPU — without changing a single line of service code.

Important: for accurate eBPF profiling of Go binaries, it is recommended to compile with the -gcflags="all=-trimpath" flag and ensure the binary is not stripped (or that debug symbols are available separately).

Security with eBPF: Tetragon and Runtime Security

Tetragon takes runtime security to a new level. Instead of analyzing logs or events after the fact, it operates in-kernel: eBPF programs intercept system calls and can both log and block actions in real time.

Installing Tetragon:

helm repo add cilium https://helm.cilium.io/
helm install tetragon cilium/tetragon \
  --namespace kube-system \
  --set tetragon.grpc.address="localhost:54321"

Example TracingPolicy for detecting suspicious binary execution:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-shell-execution
spec:
  kprobes:
  - call: "sys_execve"
    syscall: true
    args:
    - index: 0
      type: "string"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Postfix"
        values:
        - "/sh"
        - "/bash"
        - "/python3"
      matchActions:
      - action: Sigkill

This policy automatically kills a process if a shell is launched inside a container — a classic indicator of an attack. Tetragon exports events in JSON format compatible with SIEM systems (Elastic, Splunk). Integration with Falco and OPA enables multi-layered defense.

In the context of Kubernetes, Tetragon sees every fork/exec, every network connection, and every filesystem operation inside pods — without modifying deployment manifests.

Comparison with the Classic Approach: Sidecar + OpenTelemetry

The classic observability approach in Kubernetes involves:

• A sidecar container (Envoy, Jaeger agent) in every pod
• Code instrumentation via the OpenTelemetry SDK
• Manual tracing configuration in every microservice
• Additional resource consumption: ~50–100 MB RAM and ~0.1 CPU per pod

The eBPF-based approach offers:

• One DaemonSet per node instead of a sidecar in every pod
• Zero-code instrumentation — application code remains unchanged
• Kernel-level data: more complete and reliable
• Lower overhead: ~10–30 MB RAM and ~0.05 CPU per node

When to use OpenTelemetry + sidecar: when you need business-logic tracing (request context, custom attributes), detailed spans within a single service, or when using older Linux kernel versions (below 5.4). In CI/CD pipelines for service testing, code instrumentation provides more precise data about internal behavior.

When to use eBPF: for infrastructure monitoring, network observability, security, profiling, and in situations where changing application code is undesirable or impossible (third-party services, legacy systems). In 2026, the best practice is a hybrid approach: eBPF for the infrastructure layer, OpenTelemetry for business tracing.

eBPF Limitations: Kernel Versions, Permissions, Compatibility

eBPF is a powerful technology, but it comes with important limitations to consider in production:

• Linux kernel version: minimum 4.9 for basic eBPF, 5.4+ for most features, 5.15+ for the full feature set (BTF, CO-RE). Managed Kubernetes (EKS, GKE, AKS) typically uses kernels 5.15+, but verify specific AMIs/node images.
• Permissions: loading eBPF programs requires CAP_BPF (Linux 5.8+) or CAP_SYS_ADMIN. In Kubernetes, this means the DaemonSet agent must run with elevated privileges — requiring a separate security review.
• Windows nodes: eBPF for Windows exists (a Microsoft project), but falls significantly short of the Linux implementation. Mixed clusters with Windows nodes will require a separate observability strategy.
• The eBPF verifier: programs undergo strict in-kernel verification. Complex programs may fail verification on older kernels due to instruction count limits.
• BTF (BPF Type Format): CO-RE (Compile Once, Run Everywhere) requires BTF support in the kernel (CONFIG_DEBUG_INFO_BTF=y). Most modern distributions enable this by default.
• eBPF in managed Kubernetes: Fargate (EKS) and some managed node pools have eBPF restrictions due to hypervisor architecture.

Observability Roadmap in 2026

The eBPF-based observability ecosystem is evolving rapidly in 2026. Key trends include:

• eBPF + OpenTelemetry: the OpenTelemetry eBPF Agent project (otebi) enables automatic generation of OTLP-compatible telemetry from eBPF data, bridging the gap between the kernel and application layers.
• Profiling as a standard: continuous profiling is becoming a core observability pillar alongside metrics, logs, and traces — the so-called "four pillars" instead of three.
• AI-assisted anomaly detection: eBPF data, with its high granularity, is becoming the foundation for ML models that detect anomalies in real time.
• Kubernetes Gateway API + Cilium: Cilium is becoming the reference implementation for the Gateway API, unifying ingress, service mesh, and observability in a single solution.
• eBPF in CI/CD: using eBPF in test environments to automatically build dependency maps and detect unexpected network interactions in integration tests.

Go microservices benefit particularly: the Go runtime is well-suited to eBPF tracing, and the gc compiler starting with version 1.21 generates frame pointers by default — which is critical for high-quality CPU profiling via eBPF.

Conclusion

eBPF fundamentally transforms the approach to observability in Kubernetes. The ability to obtain deep telemetry — network, system, profiling, and security — without modifying microservice code and without the overhead of a sidecar architecture makes eBPF a must-have tool for DevOps engineers and SREs in 2026.

A practical starting point: install Cilium as your CNI with Hubble enabled for immediate network visibility, add Pyroscope for continuous profiling of Go services, and configure Tetragon for runtime security. This stack covers three of the four observability pillars without changing a single line of application code.

The classic OpenTelemetry approach isn't disappearing — it's evolving toward integration with the eBPF layer, creating a hybrid model where the kernel level provides infrastructure telemetry and the SDK provides business context. This combination defines the future of observability in cloud-native microservice architectures.