DevOps

Dynamic Microservice Configuration in Kubernetes: ConfigMaps, Hot Reload, and a Centralized Config Server

Ruslan Ismailov Published 14 min read
D

Introduction: Why Configuration Management Is a Pain Point for Microservices

When a monolith splits into dozens of microservices, the first pain felt by any DevOps team is configuration. Each service has its own database settings, addresses of neighboring services, feature flags, timeouts, and limits. Environments multiply: dev, staging, production. And with that comes risk: a single misconfigured config file can bring down the entire cluster.

In practice, it looks like this: a developer changes a third-party API endpoint, pushes the change to a ConfigMap, and the service has no idea — because environment variables were already injected at pod startup. You end up doing a manual rolling restart. In a multi-service Kubernetes environment, this becomes a systemic problem that demands an architectural solution.

In this article, we'll cover the full stack: from basic ConfigMap and Secret objects to the Sidecar-Watcher pattern, a centralized Config Server in Go backed by PostgreSQL and Redis, configuration versioning, and safe CI/CD pipeline integration.

The Basics: ConfigMaps and Secrets in Kubernetes

A ConfigMap is a standard Kubernetes object for storing non-sensitive configuration data as key-value pairs. A Secret is used for sensitive data (passwords, tokens), stored in base64 and optionally encrypted at rest via KMS.

Mounting a ConfigMap as Environment Variables

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  DATABASE_HOST: "postgres.production.svc.cluster.local"
  DATABASE_PORT: "5432"
  CACHE_TTL: "300"
  LOG_LEVEL: "info"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: payment-service
          image: payment-service:1.4.2
          envFrom:
            - configMapRef:
                name: app-config

Mounting a ConfigMap as a File (Volume)

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    worker_processes auto;
    events { worker_connections 1024; }
    http {
      server {
        listen 80;
        location /health { return 200 'ok'; }
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-gateway
spec:
  template:
    spec:
      volumes:
        - name: nginx-conf
          configMap:
            name: nginx-config
      containers:
        - name: nginx
          image: nginx:1.25
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf

Key limitation: when mounted via envFrom, environment variables are injected once at container startup. Changing the ConfigMap will not update them without restarting the pod. When mounted via a Volume, kubelet updates the file automatically (with a delay of up to 2 minutes), but the application must watch the file for changes itself.

The Hot Reload Problem: Why ConfigMap Doesn't Restart Pods

Kubernetes ConfigMap hot reload is one of the most common questions among DevOps engineers. The answer is simple: Kubernetes is not responsible for application logic. The platform updates the mounted file in the pod (via symlink), but does not send any signal to the application.

There are three strategies to address this:

  1. Manual rolling restart or via annotation — simple, but causes brief downtime if PodDisruptionBudget is not configured properly.
  2. The application reads the file on a timer or via inotify — works only with Volume mounts and requires changes to the service code.
  3. A sidecar controller or external operator — the most elegant solution for Kubernetes environments.

Forced rolling restart via annotation (adding the ConfigMap hash to the Deployment annotation):

# Command to force a restart
kubectl rollout restart deployment/payment-service -n production

# Or via annotation in a CI/CD pipeline:
CONFIG_HASH=$(kubectl get configmap app-config -o jsonpath='{.data}' | sha256sum | cut -d' ' -f1)
kubectl patch deployment payment-service \
  -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"configmap-hash\":\"$CONFIG_HASH\"}}}}}"

The Sidecar Pattern and Stakater Reloader

Stakater Reloader is an open-source Kubernetes operator that watches for changes to ConfigMaps and Secrets and automatically performs rolling restarts of associated Deployments, StatefulSets, and DaemonSets. It is the classic implementation of zero-restart Kubernetes configuration updates — or more precisely, restarts that are minimal and fully controlled.

Installing Stakater Reloader via Helm

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install reloader stakater/reloader \
  --namespace reloader \
  --create-namespace \
  --set reloader.watchGlobally=false

Annotating a Deployment for Automatic Reload

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  annotations:
    reloader.stakater.com/auto: "true"
    # Or explicitly specify which ConfigMap to watch:
    configmap.reloader.stakater.com/reload: "app-config,feature-flags"
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: payment-service
          image: payment-service:1.4.2
          envFrom:
            - configMapRef:
                name: app-config

Building Your Own Config Watcher in Go

If you need full control, implement a configuration file watcher directly in your Go application. Use the fsnotify package:

package config

import (
    "encoding/json"
    "log"
    "os"
    "sync"

    "github.com/fsnotify/fsnotify"
)

type AppConfig struct {
    DatabaseHost string `json:"DATABASE_HOST"`
    CacheTTL     int    `json:"CACHE_TTL"`
    LogLevel     string `json:"LOG_LEVEL"`
}

type ConfigWatcher struct {
    mu       sync.RWMutex
    config   *AppConfig
    filePath string
    watcher  *fsnotify.Watcher
}

func NewConfigWatcher(filePath string) (*ConfigWatcher, error) {
    watcher, err := fsnotify.NewWatcher()
    if err != nil {
        return nil, err
    }

    cw := &ConfigWatcher{
        filePath: filePath,
        watcher:  watcher,
    }

    if err := cw.load(); err != nil {
        return nil, err
    }

    if err := watcher.Add(filePath); err != nil {
        return nil, err
    }

    go cw.watch()
    return cw, nil
}

func (cw *ConfigWatcher) load() error {
    data, err := os.ReadFile(cw.filePath)
    if err != nil {
        return err
    }
    cfg := &AppConfig{}
    if err := json.Unmarshal(data, cfg); err != nil {
        return err
    }
    cw.mu.Lock()
    cw.config = cfg
    cw.mu.Unlock()
    log.Printf("[ConfigWatcher] Config reloaded from %s", cw.filePath)
    return nil
}

func (cw *ConfigWatcher) watch() {
    for {
        select {
        case event, ok := <-cw.watcher.Events:
            if !ok {
                return
            }
            // Kubernetes updates the file via symlink-swap
            if event.Has(fsnotify.Create) || event.Has(fsnotify.Write) {
                if err := cw.load(); err != nil {
                    log.Printf("[ConfigWatcher] Error reloading config: %v", err)
                }
            }
        case err, ok := <-cw.watcher.Errors:
            if !ok {
                return
            }
            log.Printf("[ConfigWatcher] Watcher error: %v", err)
        }
    }
}

func (cw *ConfigWatcher) Get() *AppConfig {
    cw.mu.RLock()
    defer cw.mu.RUnlock()
    return cw.config
}

Important detail: when Kubernetes updates a Volume, it does not modify the file directly — it swaps the symlink in the ..data directory. That's why you need to watch for Create events, not just Write.

Centralized Config Server: Architecture and Implementation

ConfigMaps are fine for simple scenarios. When you have 20+ services, configurations spread across multiple namespaces, and you need versioning, change auditing, and dynamic updates without any restarts — you need a centralized Config Server.

When You Need an External Config Server

  • You need to track the history of configuration changes (audit log)
  • Feature flags that must be applied instantly without restarting the service
  • Dynamic settings: rate limits, circuit breaker thresholds, A/B parameters
  • Configuration is shared between services owned by different teams
  • You need fine-grained authorization: which service can read which key

Config Server Architecture

The Config Server stores configuration in PostgreSQL (for reliability and versioning) and caches hot data in Redis. Services subscribe to updates via long polling or Server-Sent Events. When a configuration changes, the Config Server publishes an event to Redis Pub/Sub, and all subscribers receive a notification and update their local cache.

-- PostgreSQL schema for Config Server
CREATE TABLE config_entries (
    id          BIGSERIAL PRIMARY KEY,
    service     VARCHAR(100) NOT NULL,
    environment VARCHAR(50)  NOT NULL,
    key         VARCHAR(255) NOT NULL,
    value       TEXT         NOT NULL,
    version     INT          NOT NULL DEFAULT 1,
    created_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    created_by  VARCHAR(100) NOT NULL,
    is_deleted  BOOLEAN      NOT NULL DEFAULT FALSE
);

CREATE UNIQUE INDEX idx_config_service_env_key 
    ON config_entries(service, environment, key) 
    WHERE NOT is_deleted;

CREATE TABLE config_history (
    id         BIGSERIAL PRIMARY KEY,
    entry_id   BIGINT       NOT NULL REFERENCES config_entries(id),
    old_value  TEXT,
    new_value  TEXT         NOT NULL,
    version    INT          NOT NULL,
    changed_at TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    changed_by VARCHAR(100) NOT NULL
);

Config Server REST API in Go

package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "time"

    "github.com/go-chi/chi/v5"
    "github.com/jackc/pgx/v5/pgxpool"
    "github.com/redis/go-redis/v9"
)

type ConfigServer struct {
    db    *pgxpool.Pool
    redis *redis.Client
}

type ConfigEntry struct {
    Key     string `json:"key"`
    Value   string `json:"value"`
    Version int    `json:"version"`
}

// GET /config/{service}/{environment}
func (cs *ConfigServer) GetConfig(w http.ResponseWriter, r *http.Request) {
    service := chi.URLParam(r, "service")
    env     := chi.URLParam(r, "environment")
    ctx     := r.Context()

    cacheKey := "config:" + service + ":" + env
    cached, err := cs.redis.Get(ctx, cacheKey).Result()
    if err == nil {
        w.Header().Set("Content-Type", "application/json")
        w.Header().Set("X-Config-Source", "cache")
        w.Write([]byte(cached))
        return
    }

    rows, err := cs.db.Query(ctx,
        `SELECT key, value, version FROM config_entries
         WHERE service=$1 AND environment=$2 AND NOT is_deleted
         ORDER BY key`,
        service, env,
    )
    if err != nil {
        http.Error(w, "db error", http.StatusInternalServerError)
        return
    }
    defer rows.Close()

    entries := map[string]ConfigEntry{}
    for rows.Next() {
        var e ConfigEntry
        rows.Scan(&e.Key, &e.Value, &e.Version)
        entries[e.Key] = e
    }

    data, _ := json.Marshal(entries)
    cs.redis.Set(ctx, cacheKey, data, 5*time.Minute)

    w.Header().Set("Content-Type", "application/json")
    w.Write(data)
}

// POST /config/{service}/{environment} — update with cache invalidation
func (cs *ConfigServer) UpdateConfig(w http.ResponseWriter, r *http.Request) {
    service := chi.URLParam(r, "service")
    env     := chi.URLParam(r, "environment")
    ctx     := r.Context()

    var entry ConfigEntry
    if err := json.NewDecoder(r.Body).Decode(&entry); err != nil {
        http.Error(w, "invalid body", http.StatusBadRequest)
        return
    }

    _, err := cs.db.Exec(ctx,
        `INSERT INTO config_entries(service, environment, key, value, created_by)
         VALUES($1,$2,$3,$4,$5)
         ON CONFLICT DO UPDATE SET value=$4, version=version+1`,
        service, env, entry.Key, entry.Value, "api",
    )
    if err != nil {
        http.Error(w, "db error", http.StatusInternalServerError)
        return
    }

    // Invalidate cache and publish event
    cacheKey := "config:" + service + ":" + env
    cs.redis.Del(ctx, cacheKey)
    cs.redis.Publish(ctx, "config-updates",
        service+":"+env+":"+entry.Key)

    w.WriteHeader(http.StatusNoContent)
}

func main() {
    r := chi.NewRouter()
    // ... initialize db and redis ...
    cs := &ConfigServer{}
    r.Get("/config/{service}/{environment}", cs.GetConfig)
    r.Post("/config/{service}/{environment}", cs.UpdateConfig)
    log.Fatal(http.ListenAndServe(":8080", r))
}

Configuration Versioning and Rollback

Versioning is a key capability that native ConfigMaps lack. In the schema above, the config_history table stores the full changelog. Rolling back to a previous version is done with a simple query:

-- Roll back a key to its previous version
WITH prev AS (
  SELECT old_value, version - 1 as prev_version
  FROM config_history
  WHERE entry_id = $1
  ORDER BY version DESC
  LIMIT 1
)
UPDATE config_entries
SET value = prev.old_value, version = prev.prev_version
FROM prev
WHERE id = $1;

For ConfigMaps in Kubernetes, versioning can be achieved via GitOps: store all manifests in Git and use ArgoCD or Flux. In that case, a configuration rollback is simply a git revert followed by an automatic deploy.

Per-Environment Configuration: Kustomize vs Helm

Kubernetes configuration management in 2026 almost always comes down to choosing between Kustomize and Helm for managing environment-specific configurations.

Structure with Kustomize

config/
├── base/
│   ├── configmap.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/
│   │   ├── configmap-patch.yaml
│   │   └── kustomization.yaml
│   ├── staging/
│   │   ├── configmap-patch.yaml
│   │   └── kustomization.yaml
│   └── production/
│       ├── configmap-patch.yaml
│       └── kustomization.yaml

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patchesStrategicMerge:
  - configmap-patch.yaml

# overlays/production/configmap-patch.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: "postgres-prod.internal"
  LOG_LEVEL: "warn"
  CACHE_TTL: "600"

Helm is a better fit when configuration is heavily parameterized and you need a full templating engine. Kustomize is preferable for straightforward patch strategies without an additional DSL.

CI/CD Integration: Updating Configuration Without Downtime

In a CI/CD pipeline, configuration updates should happen atomically and safely. The general approach: first update the ConfigMap, then trigger a rolling restart only for the services that depend on the changed keys.

# .github/workflows/deploy-config.yml
name: Deploy Configuration
on:
  push:
    paths:
      - 'config/**'
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup kubectl
        uses: azure/setup-kubectl@v3

      - name: Apply ConfigMaps
        run: |
          kubectl apply -k config/overlays/production

      - name: Compute config hash and trigger restart
        run: |
          CONFIG_HASH=$(kubectl get configmap app-config \
            -n production \
            -o jsonpath='{.data}' | sha256sum | cut -d' ' -f1)

          kubectl patch deployment payment-service \
            -n production \
            -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"config-hash\":\"$CONFIG_HASH\"}}}}}" 

          # Wait for successful rollout
          kubectl rollout status deployment/payment-service \
            -n production \
            --timeout=5m

When using Stakater Reloader, the forced restart step is unnecessary — the operator will perform the rolling restart automatically after the new ConfigMap is applied.

Configuration Security: Secrets Separate from Configuration

The golden rule: never store secrets (passwords, API keys, TLS certificates) in a ConfigMap. Use Kubernetes Secrets with at-rest encryption via your cloud provider's KMS (AWS KMS, GCP CKMS).

The optimal stack for production environments:

  • HashiCorp Vault — centralized secrets storage, dynamic secrets, fine-grained access policies, audit log
  • External Secrets Operator — syncs secrets from Vault, AWS Secrets Manager, or GCP Secret Manager into Kubernetes Secrets
  • Sealed Secrets — encrypts Kubernetes Secrets for storage in Git (GitOps approach)
# ExternalSecret — sync from Vault into a Kubernetes Secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: payment-db-secret
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_PASSWORD
      remoteRef:
        key: secret/production/payment-service
        property: db_password

For sensitive configuration data in the Config Server, use application-level encryption before writing to PostgreSQL — AES-256-GCM with a key sourced from Vault.

Decision Matrix: When to Use What

Let's wrap up with a practical decision matrix for choosing the right configuration management tool:

  • Kubernetes ConfigMap — up to 10 services, static configuration, small DevOps team, no audit requirements. Simple to maintain, native to Kubernetes.
  • ConfigMap + Stakater Reloader — when you need automatic hot reload without manual restarts. Minimal overhead, no external dependencies.
  • Centralized Config Server — 20+ services, need for audit logs, versioning, dynamic feature flags, and fine-grained access control. Requires infrastructure support (PostgreSQL + Redis).
  • HashiCorp Vault — always use for production secrets. Complements ConfigMaps or a Config Server, does not replace them. Mandatory when compliance requirements apply (PCI DSS, SOC 2).
  • Spring Cloud Config / Consul — justified as a replacement for a custom Config Server if you're already invested in these ecosystems.

Conclusion

Managing microservice configuration in Kubernetes is an evolutionary journey. Start with native ConfigMaps: they are simple, reliable, and well-integrated into the ecosystem. Add Stakater Reloader for automatic rolling restarts on configuration changes — this solves 80% of use cases without unnecessary complexity.

When your architecture grows to dozens of microservices with requirements for dynamic updates, auditing, and versioning — build a centralized Config Server. Go is an excellent choice for this task: high performance, straightforward concurrency, and a rich ecosystem of libraries for PostgreSQL and Redis.

Secrets are always separate. The External Secrets Operator with Vault is the production standard in 2026. Separating configuration from secrets is not optional — it is a fundamental requirement of a secure architecture.

A good configuration management system is invisible in production and very visible during an incident — that's exactly when you'll appreciate the audit log, instant rollback, and centralized control.

Technologies

Tags

Ruslan Ismailov

Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →