DevOps

Canary Deployment on Kubernetes with Automatic Rollback via CI/CD and Prometheus Metrics

Ruslan Ismailov Published 12 min read
C

Introduction: What Is Canary Deployment and Why You Need It

Canary deployment is a release strategy where a small percentage of real traffic is routed to the new version of an application while the majority of the load stays on the stable version. The name comes from the old mining practice of bringing a canary into a coal mine — if the bird died, there was dangerous gas in the air. Similarly, if the new version of a service degrades on 5% of traffic, it never reaches 100% of users.

How canary differs from other deployment strategies:

  • Rolling update — gradually replaces old pods with new ones, distributing traffic as replacements happen. There is no precise control over the percentage of traffic hitting the new version.
  • Blue-green — maintains two full copies of the environment and switches all traffic at once. Expensive in terms of resources, with no gradual transition.
  • Canary — gives precise control over the traffic split, allows you to validate metrics before a full rollout, and automatically rolls back if issues arise.

Canary deployment is especially relevant for microservices architectures, where services are deployed independently and the cost of a production error is high. Combined with Kubernetes, CI/CD pipelines, and Prometheus, it becomes a fully automated, safe deployment process.

Canary Deployment Architecture in Kubernetes

The standard architecture consists of three components: two Deployment objects (stable and canary), one Service, and an Ingress with weighted routing.

Deployments: Stable and Canary

Two Deployments are run with different labels (track: stable and track: canary) but the same application label. The Service selects pods by the shared label, while traffic weight distribution is configured at the Ingress level.

Weighted Routing with NGINX Ingress

NGINX Ingress Controller supports canary routing through annotations. A separate Ingress resource is marked as canary and assigned a weight (e.g., 10%). The Gateway API (the new Kubernetes standard) offers a more expressive approach via the HTTPRoute resource with explicit backend weights.

Step-by-Step Canary Setup with Kubernetes Manifests

Step 1: Stable Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-service-stable
  labels:
    app: go-service
    track: stable
spec:
  replicas: 4
  selector:
    matchLabels:
      app: go-service
      track: stable
  template:
    metadata:
      labels:
        app: go-service
        track: stable
    spec:
      containers:
      - name: go-service
        image: registry.example.com/go-service:v1.4.2
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Step 2: Canary Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-service-canary
  labels:
    app: go-service
    track: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-service
      track: canary
  template:
    metadata:
      labels:
        app: go-service
        track: canary
    spec:
      containers:
      - name: go-service
        image: registry.example.com/go-service:v1.5.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Step 3: Service

apiVersion: v1
kind: Service
metadata:
  name: go-service
spec:
  selector:
    app: go-service
  ports:
  - port: 80
    targetPort: 8080

The Service selects pods by the label app: go-service, meaning both stable and canary pods are included. With 4 stable + 1 canary replica, load will be distributed roughly 80/20. For precise control, use Ingress.

Step 4: Ingress with Weighted Routing

# Main Ingress (stable)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-service-stable
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: go-service-stable-svc
            port:
              number: 80
---
# Canary Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-service-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: go-service-canary-svc
            port:
              number: 80

The annotation nginx.ingress.kubernetes.io/canary-weight: "10" routes 10% of traffic to the canary version. The value can be updated dynamically via kubectl annotate without recreating the resource.

CI/CD Pipeline Integration: GitHub Actions

Below is an example GitHub Actions workflow that implements canary deployment with gradual traffic increases and metric checks at each stage.

name: Canary Deploy

on:
  push:
    branches: [main]

env:
  IMAGE: registry.example.com/go-service
  NAMESPACE: production

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and push image
        run: |
          docker build -t $IMAGE:${{ github.sha }} .
          docker push $IMAGE:${{ github.sha }}

  canary-deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/setup-kubectl@v3

      - name: Deploy canary (10%)
        run: |
          kubectl set image deployment/go-service-canary \
            go-service=$IMAGE:${{ github.sha }} \
            -n $NAMESPACE
          kubectl annotate ingress go-service-canary \
            nginx.ingress.kubernetes.io/canary-weight=10 \
            --overwrite -n $NAMESPACE
          kubectl rollout status deployment/go-service-canary -n $NAMESPACE

      - name: Wait and check metrics (10%)
        run: bash scripts/check_metrics.sh 10

      - name: Increase to 30%
        run: |
          kubectl annotate ingress go-service-canary \
            nginx.ingress.kubernetes.io/canary-weight=30 \
            --overwrite -n $NAMESPACE

      - name: Wait and check metrics (30%)
        run: bash scripts/check_metrics.sh 30

      - name: Full rollout (100%)
        run: |
          kubectl set image deployment/go-service-stable \
            go-service=$IMAGE:${{ github.sha }} \
            -n $NAMESPACE
          kubectl rollout status deployment/go-service-stable -n $NAMESPACE
          kubectl annotate ingress go-service-canary \
            nginx.ingress.kubernetes.io/canary-weight=0 \
            --overwrite -n $NAMESPACE
          kubectl scale deployment/go-service-canary --replicas=0 -n $NAMESPACE

Monitoring Metrics with Prometheus

Prometheus is the key tool for evaluating canary release quality. We define two primary success criteria: error rate and latency (p99).

Configuring ServiceMonitor for Prometheus Operator

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: go-service-canary
  namespace: production
spec:
  selector:
    matchLabels:
      track: canary
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

PromQL Queries for Quality Assessment

Error rate (5xx over the last 5 minutes):

sum(rate(http_requests_total{job="go-service-canary",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="go-service-canary"}[5m]))

P99 latency:

histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{job="go-service-canary"}[5m]))
  by (le)
)

Automatic Rollback When Error Thresholds Are Exceeded

The check_metrics.sh script queries the Prometheus API and triggers a rollback if metrics exceed acceptable limits.

#!/bin/bash
# check_metrics.sh
# Argument: current canary weight (for logging)
CANARY_WEIGHT=$1
PROMETHEUS_URL="http://prometheus.monitoring.svc.cluster.local:9090"
ERROR_THRESHOLD=0.02   # 2% error rate
LATENCY_THRESHOLD=0.5  # 500ms p99
WAIT_SECONDS=120

echo "Waiting ${WAIT_SECONDS}s for metrics to accumulate at canary weight=${CANARY_WEIGHT}%..."
sleep $WAIT_SECONDS

# Query error rate
ERROR_RATE=$(curl -sf "${PROMETHEUS_URL}/api/v1/query" \
  --data-urlencode 'query=sum(rate(http_requests_total{job="go-service-canary",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="go-service-canary"}[5m]))' \
  | jq -r '.data.result[0].value[1] // "0"')

# Query p99 latency
LATENCY=$(curl -sf "${PROMETHEUS_URL}/api/v1/query" \
  --data-urlencode 'query=histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="go-service-canary"}[5m])) by (le))' \
  | jq -r '.data.result[0].value[1] // "0"')

echo "Error rate: ${ERROR_RATE} (threshold: ${ERROR_THRESHOLD})"
echo "P99 latency: ${LATENCY}s (threshold: ${LATENCY_THRESHOLD}s)"

# Compare against thresholds using awk
ERROR_EXCEEDED=$(awk "BEGIN {print (${ERROR_RATE} > ${ERROR_THRESHOLD}) ? 1 : 0}")
LATENCY_EXCEEDED=$(awk "BEGIN {print (${LATENCY} > ${LATENCY_THRESHOLD}) ? 1 : 0}")

if [ "$ERROR_EXCEEDED" = "1" ] || [ "$LATENCY_EXCEEDED" = "1" ]; then
  echo "CRITICAL: metrics exceeded acceptable thresholds. Initiating rollback..."
  kubectl annotate ingress go-service-canary \
    nginx.ingress.kubernetes.io/canary-weight=0 \
    --overwrite -n production
  kubectl scale deployment/go-service-canary --replicas=0 -n production
  echo "Rollback complete. Canary deployment aborted."
  exit 1
fi

echo "Metrics are within normal range. Continuing deployment."
exit 0

The script is called from the pipeline at each stage before traffic is increased. When it exits with code 1, GitHub Actions stops the workflow and subsequent steps are not executed — the rollback has already taken place.

Real-World Case: Zero-Downtime Canary Deployment of a Go Service

Consider a typical scenario: a Go-based payment processing service handling ~500 RPS in production. The team is releasing a new version with SQL query optimizations. The cost of failure is critical — even 1% of 5xx errors on a payment service is unacceptable.

  1. Image build: GitHub Actions builds the Docker image of the Go service and pushes it to the registry tagged with the commit SHA.
  2. Deploy canary at 5%: One new pod is launched, and Ingress is configured to route 5% of traffic to it. We wait 2 minutes.
  3. Check metrics: Prometheus records an error rate of 0.1% (normal) and p99 latency of 120ms (normal, threshold ≤500ms). The pipeline continues.
  4. Increase to 20%: Replicas are added and the weight is updated. A new check runs after 3 minutes — everything is within range.
  5. Full rollout: The stable Deployment is updated with the new image via rolling update, the canary weight is removed, and the canary is scaled to 0. Downtime — 0 seconds.

The entire process took about 15 minutes, fully automated, with no manual intervention. The GitOps approach (manifests in Git, changes only through the pipeline) ensures an audit trail for every change.

Common Mistakes and How to Avoid Them

  • Observation window is too short: 30 seconds is not enough for statistically significant metrics at low RPS. A minimum of 2–5 minutes is recommended depending on traffic volume.
  • Ignoring JVM/Go runtime warm-up: In the first seconds after a pod starts, latency will be higher than normal. Use readinessProbe and startupProbe to ensure traffic only reaches a ready pod.
  • Canary without sticky session isolation: If users need to stay on one version (e.g., for an A/B test), use nginx.ingress.kubernetes.io/canary-by-cookie or canary-by-header.
  • No alerts for a stuck canary: If the pipeline fails but the canary Ingress remains with a weight of 20%, that is an abnormal state. Add an Alertmanager rule for a canary Ingress that has been active for too long.
  • Single Service for both Deployments: If precise weighted routing is required, use separate Services for stable and canary, and manage traffic distribution exclusively through Ingress or Gateway API.
  • No resource limits on canary: A canary pod without limits can consume resources needed by neighboring pods. Always set both requests and limits.

Conclusion and Checklist

Canary deployment on Kubernetes is a mature approach to safe releases that, when configured correctly, completely eliminates the need for manual oversight. The combination of Kubernetes + NGINX Ingress + Prometheus + CI/CD delivers a full cycle: deploy → observe → automated decision (proceed or roll back).

Safe deployment is not about an on-call engineer heroically fixing things at 3 a.m. — it's about well-configured automation that catches problems before users ever notice them.

Checklist for implementing canary deployment:

  • Separate Deployments created for stable and canary versions
  • Ingress configured with canary-weight annotations
  • ServiceMonitor or pod annotations set up for Prometheus metric collection
  • Error rate and latency thresholds defined (e.g., <2% errors, p99 <500ms)
  • Metrics check script integrated into the CI/CD pipeline
  • Rollback implemented as an explicit pipeline step with exit code 1
  • Readiness and liveness probes configured on the canary pod
  • Alert configured for a stuck canary Ingress
  • Manifests stored in Git (GitOps approach)
  • A test run performed with a deliberate error to verify rollback behavior

Technologies

Tags

Ruslan Ismailov

Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →