Monitoring and Observability for Go Services: Metrics, Tracing, and Logs in a Unified Ecosystem

Introduction: The Three Pillars of Observability

Observability is the ability to understand the internal state of a system based on its external outputs. For production Go services, this means three essential components:

• Metrics — numerical indicators of system state over time (CPU, RPS, latency, error rate).
• Tracing — tracking the path of a request through all services and components.
• Logs — structured event records with context.

Without each of these pillars, you're operating blind. Metrics tell you "something is broken," logs tell you "why," and tracing tells you "exactly where." In 2026, OpenTelemetry has become the de facto standard, and the combination of Prometheus + Grafana + Loki + Tempo covers all three areas in a unified ecosystem. In this article, we'll walk through the entire journey from code instrumentation to Kubernetes deployment.

Instrumenting a Go Application: Prometheus Client and Custom Metrics

The first step is to add metrics directly to your service code. The official Prometheus client for Go lets you export counters, histograms, gauges, and summaries.

// go get github.com/prometheus/client_golang/prometheus\n// go get github.com/prometheus/client_golang/prometheus/promhttp\n\npackage main\n\nimport (\n    \"net/http\"\n    \"time\"\n\n    \"github.com/prometheus/client_golang/prometheus\"\n    \"github.com/prometheus/client_golang/prometheus/promhttp\"\n)\n\nvar (\n    httpRequestsTotal = prometheus.NewCounterVec(\n        prometheus.CounterOpts{\n            Name: \"http_requests_total\",\n            Help: \"Total number of HTTP requests\",\n        },\n        []string{\"method\", \"path\", \"status\"},\n    )\n\n    httpRequestDuration = prometheus.NewHistogramVec(\n        prometheus.HistogramOpts{\n            Name:    \"http_request_duration_seconds\",\n            Help:    \"HTTP request duration in seconds\",\n            Buckets: prometheus.DefBuckets,\n        },\n        []string{\"method\", \"path\"},\n    )\n)\n\nfunc init() {\n    prometheus.MustRegister(httpRequestsTotal)\n    prometheus.MustRegister(httpRequestDuration)\n}\n\nfunc metricsMiddleware(next http.Handler) http.Handler {\n    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n        start := time.Now()\n        rw := &responseWriter{w, http.StatusOK}\n        next.ServeHTTP(rw, r)\n        duration := time.Since(start).Seconds()\n\n        statusStr := http.StatusText(rw.status)\n        httpRequestsTotal.WithLabelValues(r.Method, r.URL.Path, statusStr).Inc()\n        httpRequestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(duration)\n    })\n}\n\nfunc main() {\n    mux := http.NewServeMux()\n    mux.Handle(\"/metrics\", promhttp.Handler())\n    mux.Handle(\"/api/v1/orders\", metricsMiddleware(http.HandlerFunc(ordersHandler)))\n    http.ListenAndServe(\":8080\", mux)\n}

The /metrics endpoint is the standard scrape target for Prometheus. Use labels carefully: high cardinality (e.g., user_id as a label) will kill Prometheus performance.

Distributed Tracing with OpenTelemetry in Go — 2026 Best Practices

OpenTelemetry (OTel) has become the unified standard for tracing and metrics. By 2026, the SDK has stabilized, and integrating it into Go services is straightforward.

// go get go.opentelemetry.io/otel\n// go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp\n// go get go.opentelemetry.io/otel/sdk/trace\n\npackage telemetry\n\nimport (\n    \"context\"\n\n    \"go.opentelemetry.io/otel\"\n    \"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp\"\n    \"go.opentelemetry.io/otel/sdk/resource\"\n    sdktrace \"go.opentelemetry.io/otel/sdk/trace\"\n    semconv \"go.opentelemetry.io/otel/semconv/v1.21.0\"\n)\n\nfunc InitTracer(serviceName, otlpEndpoint string) (*sdktrace.TracerProvider, error) {\n    exporter, err := otlptracehttp.New(\n        context.Background(),\n        otlptracehttp.WithEndpoint(otlpEndpoint),\n        otlptracehttp.WithInsecure(),\n    )\n    if err != nil {\n        return nil, err\n    }\n\n    res := resource.NewWithAttributes(\n        semconv.SchemaURL,\n        semconv.ServiceName(serviceName),\n        semconv.ServiceVersion(\"1.0.0\"),\n    )\n\n    tp := sdktrace.NewTracerProvider(\n        sdktrace.WithBatcher(exporter),\n        sdktrace.WithResource(res),\n        sdktrace.WithSampler(sdktrace.AlwaysSample()),\n    )\n    otel.SetTracerProvider(tp)\n    return tp, nil\n}\n\n// Usage in a handler:\nfunc ordersHandler(w http.ResponseWriter, r *http.Request) {\n    tracer := otel.Tracer(\"orders-service\")\n    ctx, span := tracer.Start(r.Context(), \"GetOrders\")\n    defer span.End()\n\n    orders, err := db.GetOrders(ctx)\n    if err != nil {\n        span.RecordError(err)\n        http.Error(w, \"internal error\", http.StatusInternalServerError)\n        return\n    }\n    // ...\n}

Traces are sent to Grafana Tempo via an OTLP collector. It's critical to propagate context.Context through all calls — this is the foundation of distributed tracing in Go. For automatic instrumentation of HTTP clients and databases, use the official OTel contrib packages.

Structured Logging: slog, zerolog, and Loki Integration

Since Go 1.21, the standard library includes log/slog — a structured logger. For high-load services, zerolog is preferred due to its zero-allocation design.

// zerolog\npackage main\n\nimport (\n    \"os\"\n    \"github.com/rs/zerolog\"\n    \"github.com/rs/zerolog/log\"\n)\n\nfunc main() {\n    zerolog.TimeFieldFormat = zerolog.TimeFormatUnix\n    log.Logger = zerolog.New(os.Stdout).With().\n        Timestamp().\n        Str(\"service\", \"orders-service\").\n        Str(\"env\", \"production\").\n        Logger()\n\n    log.Info().\n        Str(\"method\", \"GET\").\n        Str(\"path\", \"/api/v1/orders\").\n        Int(\"status\", 200).\n        Dur(\"duration\", duration).\n        Msg(\"request completed\")\n}

JSON-formatted logs from stdout are collected by Promtail (or Grafana Alloy in 2026) and sent to Grafana Loki. Loki indexes logs by labels (service, env, level), while full-text search operates on log content. Important: avoid creating unnecessary labels in Loki — the same high-cardinality problem applies as with Prometheus.

To correlate logs with traces, add trace_id and span_id to every log entry:

import \"go.opentelemetry.io/otel/trace\"\n\nfunc logWithTrace(ctx context.Context, msg string) {\n    span := trace.SpanFromContext(ctx)\n    sc := span.SpanContext()\n    log.Info().\n        Str(\"trace_id\", sc.TraceID().String()).\n        Str(\"span_id\", sc.SpanID().String()).\n        Msg(msg)\n}

Storing Metrics with Redis for Real-Time Aggregation

Prometheus handles long-term storage well, but for real-time aggregation — such as rate limiting, leaderboards, or sliding window counters — Redis is a natural fit. The Go + Redis combination allows you to count metrics directly in memory and periodically flush them to Prometheus via a custom collector.

// go get github.com/redis/go-redis/v9\n\npackage metrics\n\nimport (\n    \"context\"\n    \"github.com/redis/go-redis/v9\"\n    \"github.com/prometheus/client_golang/prometheus\"\n)\n\ntype RedisMetricsCollector struct {\n    rdb     *redis.Client\n    counter *prometheus.Desc\n}\n\nfunc NewRedisMetricsCollector(rdb *redis.Client) *RedisMetricsCollector {\n    return &RedisMetricsCollector{\n        rdb: rdb,\n        counter: prometheus.NewDesc(\n            \"realtime_orders_processed_total\",\n            \"Orders processed (from Redis counter)\",\n            []string{\"region\"}, nil,\n        ),\n    }\n}\n\nfunc (c *RedisMetricsCollector) Describe(ch chan<- *prometheus.Desc) {\n    ch <- c.counter\n}\n\nfunc (c *RedisMetricsCollector) Collect(ch chan<- prometheus.Metric) {\n    ctx := context.Background()\n    val, _ := c.rdb.Get(ctx, \"orders:processed:eu\").Float64()\n    ch <- prometheus.MustNewConstMetric(c.counter, prometheus.CounterValue, val, \"eu\")\n}

Redis is also used for caching results of expensive Prometheus queries on dashboards and for storing temporary aggregates between service restarts.

Deploying the Monitoring Stack in Kubernetes: Prometheus Operator, Grafana, Tempo

In a production Kubernetes environment, the recommended approach is to use kube-prometheus-stack (a Helm chart) that deploys Prometheus Operator, Grafana, and a set of pre-configured alerts.

# values.yaml for kube-prometheus-stack\nprometheus:\n  prometheusSpec:\n    retention: 30d\n    storageSpec:\n      volumeClaimTemplate:\n        spec:\n          storageClassName: fast-ssd\n          resources:\n            requests:\n              storage: 100Gi\n\ngrafana:\n  enabled: true\n  adminPassword: \"changeme\"\n  additionalDataSources:\n    - name: Loki\n      type: loki\n      url: http://loki:3100\n    - name: Tempo\n      type: tempo\n      url: http://tempo:3200\n\nalertmanager:\n  enabled: true

For automatic service discovery, use a ServiceMonitor — a Prometheus Operator CRD:

apiVersion: monitoring.coreos.com/v1\nkind: ServiceMonitor\nmetadata:\n  name: orders-service\n  namespace: production\nspec:\n  selector:\n    matchLabels:\n      app: orders-service\n  endpoints:\n    - port: http\n      path: /metrics\n      interval: 15s

Grafana Tempo is deployed separately and receives traces via OTLP. In Grafana, you configure correlations: from a metrics dashboard you can jump to logs in Loki, and from logs you can navigate to a trace in Tempo using trace_id. This is the unified observability ecosystem in action.

Alerting: Configuring Rules and Integrating with Messaging Platforms

Alertmanager routes alerts to Slack, Telegram, PagerDuty, and other systems. Example PrometheusRule:

apiVersion: monitoring.coreos.com/v1\nkind: PrometheusRule\nmetadata:\n  name: orders-service-alerts\nspec:\n  groups:\n    - name: orders.rules\n      rules:\n        - alert: HighErrorRate\n          expr: |\n            rate(http_requests_total{status=\"Internal Server Error\"}[5m])\n            / rate(http_requests_total[5m]) > 0.05\n          for: 2m\n          labels:\n            severity: critical\n          annotations:\n            summary: \"High error rate on {{ $labels.path }}\"\n            description: \"Error rate is {{ $value | humanizePercentage }}\"\n\n        - alert: SlowResponseTime\n          expr: |\n            histogram_quantile(0.99,\n              rate(http_request_duration_seconds_bucket[5m])\n            ) > 1.0\n          for: 5m\n          labels:\n            severity: warning\n          annotations:\n            summary: \"P99 latency above 1s\"

Alertmanager config for Telegram:

route:\n  group_by: ['alertname', 'severity']\n  receiver: 'telegram'\n\nreceivers:\n  - name: 'telegram'\n    telegram_configs:\n      - bot_token: '${TELEGRAM_BOT_TOKEN}'\n        chat_id: -1001234567890\n        message: |\n          🚨 *{{ .GroupLabels.alertname }}*\n          {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}

Practical Example: A Complete Observability Stack for a Go Microservice

Below is a docker-compose setup for local development that brings up the entire stack with a single command.

version: '3.8'\n\nservices:\n  orders-service:\n    build: ./services/orders\n    ports:\n      - \"8080:8080\"\n    environment:\n      - OTLP_ENDPOINT=otel-collector:4318\n      - REDIS_URL=redis:6379\n    depends_on:\n      - redis\n      - otel-collector\n\n  redis:\n    image: redis:7-alpine\n    ports:\n      - \"6379:6379\"\n\n  otel-collector:\n    image: otel/opentelemetry-collector-contrib:latest\n    volumes:\n      - ./otel-config.yaml:/etc/otel/config.yaml\n    command: [\"--config=/etc/otel/config.yaml\"]\n    ports:\n      - \"4318:4318\"\n\n  prometheus:\n    image: prom/prometheus:latest\n    volumes:\n      - ./prometheus.yml:/etc/prometheus/prometheus.yml\n    ports:\n      - \"9090:9090\"\n\n  grafana:\n    image: grafana/grafana:latest\n    ports:\n      - \"3000:3000\"\n    environment:\n      - GF_AUTH_ANONYMOUS_ENABLED=true\n    volumes:\n      - ./grafana/provisioning:/etc/grafana/provisioning\n\n  loki:\n    image: grafana/loki:latest\n    ports:\n      - \"3100:3100\"\n\n  tempo:\n    image: grafana/tempo:latest\n    command: [\"-config.file=/etc/tempo.yaml\"]\n    volumes:\n      - ./tempo.yaml:/etc/tempo.yaml\n    ports:\n      - \"3200:3200\"\n\n  promtail:\n    image: grafana/promtail:latest\n    volumes:\n      - /var/lib/docker/containers:/var/lib/docker/containers:ro\n      - ./promtail-config.yaml:/etc/promtail/config.yaml\n    command: -config.file=/etc/promtail/config.yaml

OTel Collector config (otel-config.yaml):

receivers:\n  otlp:\n    protocols:\n      http:\n        endpoint: 0.0.0.0:4318\n\nexporters:\n  otlp/tempo:\n    endpoint: tempo:4317\n    tls:\n      insecure: true\n  prometheus:\n    endpoint: 0.0.0.0:8889\n\nservice:\n  pipelines:\n    traces:\n      receivers: [otlp]\n      exporters: [otlp/tempo]\n    metrics:\n      receivers: [otlp]\n      exporters: [prometheus]

After running docker compose up -d, open Grafana at localhost:3000. Connect Prometheus, Loki, and Tempo as data sources, configure correlations — and you have a fully functional observability stack ready for development.

Conclusion

Building observability for Go microservices is easier than it might seem: OpenTelemetry unifies tracing and metrics, zerolog/slog handle structured logging, and the Prometheus + Grafana + Loki + Tempo stack provides a single pane of glass for analysis. Redis naturally complements the stack for real-time aggregation. In Kubernetes, Prometheus Operator and ServiceMonitor automate service discovery and metrics collection. Start with a local docker-compose setup, build good instrumentation habits, then promote the stack to production — and your Go services will become truly transparent.