Distributed Tracing in Go: OpenTelemetry, Jaeger, and Microservices

Introduction: Why Distributed Tracing Matters in 2026

When a monolith breaks apart into dozens of microservices, debugging becomes a non-trivial challenge. A user request travels through an API Gateway, several business services, a message queue, a Redis cache, and PostgreSQL — and somewhere along that chain, an 800 ms latency appears that clients are complaining about. Finding the bottleneck without observability tools is nearly impossible.

Distributed tracing is a mechanism for end-to-end request tracking across all components of a distributed system. Each request receives a unique trace_id, and each operation within a service gets a span with timestamps, attributes, and relationships. The result is a complete picture: where time was spent, which services were called, and where errors occurred.

By 2026, OpenTelemetry has become the de facto standard — a vendor-neutral CNCF project that unified OpenTracing and OpenCensus. Combined with Jaeger as a backend for storing and visualizing traces, and Go as the microservice implementation language, this forms a production-ready observability stack.

OpenTelemetry Overview: Standards, SDK, and Exporters

OpenTelemetry is a set of APIs, SDKs, and tools for collecting telemetry data: traces, metrics, and logs. The architecture consists of several key components:

API — interfaces independent of any specific implementation. Libraries are instrumented via the API without being tied to any SDK.
SDK — the API implementation with batching, sampling, and data processing.
Exporters — components that send data to a backend: Jaeger, Zipkin, OTLP, Prometheus.
Collector — an optional agent/proxy that receives data from applications and routes it to one or more backends.
Instrumentation Libraries — ready-made integrations for net/http, gRPC, database/sql, Redis, and more.

For Go, the main package is go.opentelemetry.io/otel. The tracing SDK is go.opentelemetry.io/otel/sdk/trace. The Jaeger exporter via OTLP gRPC is go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc.

Instrumenting Go Services with otel-go

Initializing the Tracer Provider

The first step is to configure the TracerProvider — the central object that manages the lifecycle of traces. This is typically done at application startup:

package telemetry

import (
    "context"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func InitTracer(ctx context.Context, serviceName, collectorAddr string) (func(), error) {
    conn, err := grpc.DialContext(ctx, collectorAddr,
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithBlock(),
    )
    if err != nil {
        return nil, err
    }

    exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
    if err != nil {
        return nil, err
    }

    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName(serviceName),
            semconv.ServiceVersion("1.0.0"),
            semconv.DeploymentEnvironment("production"),
        ),
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter,
            sdktrace.WithBatchTimeout(5*time.Second),
            sdktrace.WithMaxExportBatchSize(512),
        ),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.1), // 10% sampling in prod
        )),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    shutdown := func() {
        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()
        _ = tp.Shutdown(ctx)
    }

    return shutdown, nil
}

Creating Spans and Propagating Context

The key principle is that Go's context.Context carries the current trace information. Pass it through every layer of your application:

package service

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
)

var tracer = otel.Tracer("order-service")

type OrderService struct {
    repo   OrderRepository
    cache  CacheClient
}

func (s *OrderService) GetOrder(ctx context.Context, orderID string) (*Order, error) {
    ctx, span := tracer.Start(ctx, "OrderService.GetOrder")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.String("component", "order-service"),
    )

    // Check cache
    order, err := s.cache.Get(ctx, "order:"+orderID)
    if err == nil {
        span.SetAttributes(attribute.Bool("cache.hit", true))
        return order, nil
    }

    span.SetAttributes(attribute.Bool("cache.hit", false))

    // Query the database
    order, err = s.repo.FindByID(ctx, orderID)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return nil, fmt.Errorf("repo.FindByID: %w", err)
    }

    return order, nil
}

Setting Up Jaeger as a Trace Collection Backend

Jaeger is an open-source tracing system from Uber, adopted by CNCF. In development mode, the all-in-one image is convenient to use. Here is a docker-compose.yml configuration:

version: "3.9"

services:
  jaeger:
    image: jaegertracing/all-in-one:1.55
    environment:
      - COLLECTOR_OTLP_ENABLED=true
      - SPAN_STORAGE_TYPE=badger
      - BADGER_EPHEMERAL=false
      - BADGER_DIRECTORY_VALUE=/badger/data
      - BADGER_DIRECTORY_KEY=/badger/key
    volumes:
      - jaeger-data:/badger
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
      - "14268:14268"   # Jaeger HTTP collector
    restart: unless-stopped

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.95.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4319:4317"  # OTLP gRPC from applications
    depends_on:
      - jaeger

volumes:
  jaeger-data:

OpenTelemetry Collector configuration (otel-collector-config.yaml):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  logging:
    loglevel: warn

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger, logging]

For production, it is recommended to use Jaeger with Elasticsearch or Cassandra as the storage backend instead of badger. Set SPAN_STORAGE_TYPE=elasticsearch and specify your cluster address.

Integrating Tracing with REST API and gRPC Endpoints

Middleware for REST API (net/http)

For HTTP servers, use the go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp library:

package middleware

import (
    "net/http"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func NewTracingMiddleware(serviceName string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return otelhttp.NewHandler(next, serviceName,
            otelhttp.WithMessageEvents(
                otelhttp.ReadEvents,
                otelhttp.WriteEvents,
            ),
        )
    }
}

// HTTP client with tracing
func NewTracedHTTPClient() *http.Client {
    return &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
    }
}

Interceptors for gRPC

For gRPC services, use interceptors from the go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc package:

package server

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

func NewGRPCServer() *grpc.Server {
    return grpc.NewServer(
        grpc.StatsHandler(otelgrpc.NewServerHandler(
            otelgrpc.WithMessageEvents(
                otelgrpc.ReceivedEvents,
                otelgrpc.SentEvents,
            ),
        )),
    )
}

func NewGRPCClientConn(target string) (*grpc.ClientConn, error) {
    return grpc.Dial(target,
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    )
}

Correlating Traces with Logs and Metrics

Maximum observability value is achieved when traces, logs, and metrics are linked together. The key is injecting trace_id and span_id into every log record:

package logger

import (
    "context"
    "log/slog"
    "os"

    "go.opentelemetry.io/otel/trace"
)

type TraceHandler struct {
    handler slog.Handler
}

func (h *TraceHandler) Handle(ctx context.Context, r slog.Record) error {
    span := trace.SpanFromContext(ctx)
    if span.IsRecording() {
        sc := span.SpanContext()
        r.AddAttrs(
            slog.String("trace_id", sc.TraceID().String()),
            slog.String("span_id", sc.SpanID().String()),
            slog.String("trace_flags", sc.TraceFlags().String()),
        )
    }
    return h.handler.Handle(ctx, r)
}

func NewLogger() *slog.Logger {
    base := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        Level: slog.LevelInfo,
    })
    return slog.New(&TraceHandler{handler: base})
}

With this approach, you can instantly find all logs related to a specific request in Grafana or Kibana using the trace_id from Jaeger. For metrics, use go.opentelemetry.io/otel/metric — add service.name and environment attributes to every metric so that correlation works via exemplars in Prometheus.

Practical Example: Tracing a Chain of Three Go Services

Consider a system of three services: API Gateway, Order Service, and Inventory Service. An order creation request flows through the entire chain, touching Redis (cache) and PostgreSQL (storage).

Instrumenting PostgreSQL via database/sql

package db

import (
    "context"
    "database/sql"

    "github.com/XSAM/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    _ "github.com/lib/pq"
)

func NewPostgresDB(dsn string) (*sql.DB, error) {
    db, err := otelsql.Open("postgres", dsn,
        otelsql.WithAttributes(
            semconv.DBSystemPostgreSQL,
        ),
        otelsql.WithSpanOptions(otelsql.SpanOptions{
            Ping:                 true,
            RowsAffected:        true,
            DisableErrSkip:      true,
        }),
    )
    if err != nil {
        return nil, err
    }
    
    if err := otelsql.RegisterDBStatsMetrics(db,
        otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    ); err != nil {
        return nil, err
    }
    
    return db, nil
}

Instrumenting Redis

package cache

import (
    "context"

    "github.com/redis/go-redis/extra/redisotel/v9"
    "github.com/redis/go-redis/v9"
)

func NewRedisClient(addr string) (*redis.Client, error) {
    rdb := redis.NewClient(&redis.Options{
        Addr: addr,
        DB:   0,
    })

    // Enable Redis tracing and metrics
    if err := redisotel.InstrumentTracing(rdb,
        redisotel.WithDBStatement(true),
    ); err != nil {
        return nil, err
    }

    return rdb, nil
}

Full Order Handler with Child Spans

package handler

import (
    "context"
    "encoding/json"
    "net/http"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

var tracer = otel.Tracer("api-gateway")

type CreateOrderRequest struct {
    UserID    string   `json:"user_id"`
    ProductID string   `json:"product_id"`
    Quantity  int      `json:"quantity"`
}

func (h *Handler) CreateOrder(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    ctx, span := tracer.Start(ctx, "CreateOrder",
        trace.WithSpanKind(trace.SpanKindServer),
    )
    defer span.End()

    var req CreateOrderRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "invalid request body")
        http.Error(w, "Bad Request", http.StatusBadRequest)
        return
    }

    span.SetAttributes(
        attribute.String("user.id", req.UserID),
        attribute.String("product.id", req.ProductID),
        attribute.Int("order.quantity", req.Quantity),
    )

    // Check product availability in Inventory Service
    available, err := h.checkInventory(ctx, req.ProductID, req.Quantity)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "inventory check failed")
        http.Error(w, "Internal Server Error", http.StatusInternalServerError)
        return
    }
    span.SetAttributes(attribute.Bool("inventory.available", available))

    if !available {
        span.SetStatus(codes.Error, "out of stock")
        http.Error(w, "Product out of stock", http.StatusConflict)
        return
    }

    // Create order in Order Service
    orderID, err := h.createOrder(ctx, req)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "order creation failed")
        http.Error(w, "Internal Server Error", http.StatusInternalServerError)
        return
    }

    span.SetAttributes(attribute.String("order.id", orderID))
    span.SetStatus(codes.Ok, "order created")

    json.NewEncoder(w).Encode(map[string]string{"order_id": orderID})
}

func (h *Handler) checkInventory(ctx context.Context, productID string, qty int) (bool, error) {
    ctx, span := tracer.Start(ctx, "checkInventory",
        trace.WithSpanKind(trace.SpanKindClient),
    )
    defer span.End()

    span.SetAttributes(
        attribute.String("rpc.service", "inventory-service"),
        attribute.String("product.id", productID),
    )

    // HTTP call with context propagation
    resp, err := h.inventoryClient.CheckStock(ctx, productID, qty)
    if err != nil {
        span.RecordError(err)
        return false, err
    }

    return resp.Available, nil
}

Common Mistakes and How to Avoid Them

Losing context. The most common mistake is passing context.Background() instead of the ctx from the calling function. Always propagate context through every layer.
Missing propagation in HTTP clients. When manually creating HTTP requests, call otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header)), otherwise downstream services will not receive the trace context.
Excessive sampling in production. Tracing 100% of requests on a high-load service creates significant overhead. Use TraceIDRatioBased(0.01–0.1) and increase the rate during incident investigations.
Unclosed spans. Always call defer span.End() immediately after tracer.Start(). Unclosed spans are never exported and cause memory leaks.
Overly detailed attributes containing PII. Do not record personal user data (email, phone, card numbers) in span attributes. This violates GDPR and creates security risks.
Ignoring exporter errors. In production, set up alerting on the otelcol_exporter_send_failed_spans metric in the Collector to detect trace delivery issues in time.
A single tracer for the entire application. Create named tracers via otel.Tracer("package-name") for each package — this simplifies filtering in Jaeger.

Production Tips

A few recommendations for running in a production environment:

Use the OpenTelemetry Collector as an intermediate layer between your applications and Jaeger — this allows you to change backends without recompiling services, add data enrichment, and enable tail-based sampling.
Configure Jaeger with Elasticsearch to store traces for longer than 48 hours. Index by service.name, span.kind, and http.status_code.
Add Span Events for important business events within a span: span.AddEvent("cache.miss", trace.WithAttributes(attribute.String("key", cacheKey))).
Integrate Jaeger with Grafana via the datasource plugin for a unified dashboard with metrics, logs, and traces (Grafana Tempo as a Jaeger alternative offers native integration).

Conclusion

Distributed tracing with OpenTelemetry, Jaeger, and Go is not merely a debugging tool — it is the foundation of a modern observability strategy. Implementation requires a one-time effort: configuring the TracerProvider, adding middleware for HTTP and gRPC, and instrumenting PostgreSQL and Redis. But the payoff far exceeds the investment: incident diagnosis time drops from hours to minutes.

The key principles for successful adoption are: strict context propagation discipline, smart sampling in production, correlation of traces with logs and metrics, and using the OpenTelemetry Collector as a buffer between applications and the backend. Start by instrumenting the critical path of your application, and you will quickly see the real value of end-to-end request tracing.

Distributed Tracing in Microservices: End-to-End Request Tracing with OpenTelemetry, Jaeger, and Go