DevOps

Stateful Applications in Kubernetes: StatefulSets, Persistent Volumes, and PostgreSQL in a Cluster

Ruslan Ismailov Published 18 min read
S

Introduction: stateful vs stateless — what's the difference and why it matters

Kubernetes was originally designed as a platform for stateless services: if a container crashes, a new one spins up and everything keeps running. But the real world works differently. Databases, message queues, caches with persistence — these are all stateful applications that store data and require a stable identity across restarts.

Key differences between stateful and stateless in the context of Kubernetes:

  • Pod identity. Stateless pods are interchangeable: pod-abc and pod-xyz do the same thing. Stateful pods have stable names (e.g., postgres-0, postgres-1) and DNS records that persist across restarts.
  • Persistent storage. A stateless container doesn't need to save data to disk. PostgreSQL without a persistent volume will lose all data on restart.
  • Startup and shutdown order. In a PostgreSQL cluster, it's important that the primary starts before the replicas. StatefulSet guarantees ordered creation and deletion of pods.

The StatefulSet object in Kubernetes was introduced specifically to address these challenges. Let's explore it in detail.

StatefulSets: how they work and how they differ from Deployments

StatefulSet is a Kubernetes controller specifically designed for managing stateful applications. Unlike a Deployment, it provides three guarantees:

  1. Stable network identifiers. Each pod gets a predictable DNS name in the format <pod-name>.<service-name>.<namespace>.svc.cluster.local. For PostgreSQL, this means postgres-0.postgres-headless.default.svc.cluster.local always points to the primary node.
  2. Stable persistent storage. Each pod gets its own PersistentVolumeClaim, which is not deleted when the pod restarts — data is preserved.
  3. Ordered deployment and scaling. Pods are created in order (0, 1, 2...) and deleted in reverse order.

StatefulSet vs Deployment: a practical comparison

Deployment is suitable for API servers, frontends, and stateless worker processes. StatefulSet is required for PostgreSQL, Redis Cluster, Elasticsearch, Kafka — anywhere each instance is unique and stores data. When updating a Deployment, all pods can be replaced simultaneously (with RollingUpdate and zero delays). StatefulSet updates pods sequentially, which is critical for database clusters.

Minimal StatefulSet example for PostgreSQL

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: default
spec:
  serviceName: postgres-headless
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: mydb
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: postgres-data
              mountPath: /var/lib/postgresql/data
          resources:
            requests:
              memory: "512Mi"
              cpu: "500m"
            limits:
              memory: "2Gi"
              cpu: "2"
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "$(POSTGRES_USER)", "-d", "$(POSTGRES_DB)"]
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            exec:
              command: ["pg_isready", "-U", "$(POSTGRES_USER)"]
            initialDelaySeconds: 30
            periodSeconds: 10
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 50Gi

Pay attention to the volumeClaimTemplates section — this is the key feature of StatefulSet. For each pod, a separate PVC is automatically created with the name postgres-data-postgres-0, postgres-data-postgres-1, and so on.

Headless Service for StatefulSet

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
  namespace: default
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432

A Headless Service (with clusterIP: None) does not create a single virtual IP, but instead registers individual DNS records for each pod. This is the foundation for stable addressing of PostgreSQL cluster nodes.

Persistent Volumes and Persistent Volume Claims: configuring storage

Storage in Kubernetes is organized through three layers of abstraction:

  • PersistentVolume (PV) — the actual storage resource: a cloud disk, NFS share, or local SSD. Created by a cluster administrator or dynamically via a StorageClass.
  • PersistentVolumeClaim (PVC) — a storage request from a pod. Describes the required size, access mode, and StorageClass.
  • StorageClass — a template for dynamic PV creation. Defines the disk type (SSD, HDD), reclaim policy, and provisioner.

StorageClass for production PostgreSQL

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Key parameters for production:

  • reclaimPolicy: Retain — when a PVC is deleted, the data on disk is preserved. Never use Delete for production databases.
  • volumeBindingMode: WaitForFirstConsumer — the disk is created in the same availability zone as the pod. Critical for multi-AZ clusters.
  • allowVolumeExpansion: true — allows increasing the PVC size without recreation.
  • encrypted: "true" — disk-level data encryption.

Expanding PVC size

To increase storage capacity, simply change the storage field in an existing PVC:

kubectl patch pvc postgres-data-postgres-0 \
  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

Kubernetes will automatically expand the volume if the provisioner supports it and the StorageClass has allowVolumeExpansion: true.

Running PostgreSQL in Kubernetes: step by step

Let's walk through the complete process of deploying PostgreSQL in Kubernetes from scratch. We'll create a Secret, ConfigMap, Headless Service, external access Service, and StatefulSet.

Step 1: Secret with credentials

apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
  namespace: default
type: Opaque
stringData:
  username: pgadmin
  password: "S3cur3P@ssw0rd!"
  replication-password: "R3pl1c@P@ss!"

In production, it is recommended to use external secret managers: HashiCorp Vault with the vault-secrets-operator, AWS Secrets Manager, or Sealed Secrets.

Step 2: ConfigMap with postgresql.conf

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  namespace: default
data:
  postgresql.conf: |
    max_connections = 200
    shared_buffers = 512MB
    effective_cache_size = 1536MB
    maintenance_work_mem = 128MB
    checkpoint_completion_target = 0.9
    wal_buffers = 16MB
    default_statistics_target = 100
    random_page_cost = 1.1
    effective_io_concurrency = 200
    work_mem = 2621kB
    min_wal_size = 1GB
    max_wal_size = 4GB
    max_worker_processes = 4
    max_parallel_workers_per_gather = 2
    max_parallel_workers = 4
    wal_level = replica
    archive_mode = on
    archive_command = 'test ! -f /archive/%f && cp %p /archive/%f'
    hot_standby = on
    log_timezone = 'UTC'
    datestyle = 'iso, mdy'
    timezone = 'UTC'
    log_statement = 'ddl'
    log_min_duration_statement = 1000

Step 3: Service for external access

apiVersion: v1
kind: Service
metadata:
  name: postgres-primary
  namespace: default
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  selector:
    app: postgres
    role: primary
  ports:
    - port: 5432
      targetPort: 5432
  type: ClusterIP

Step 4: Applying manifests and verifying

# Apply all manifests
kubectl apply -f postgres-secret.yaml
kubectl apply -f postgres-config.yaml
kubectl apply -f postgres-headless-svc.yaml
kubectl apply -f postgres-statefulset.yaml

# Check status
kubectl get statefulsets
kubectl get pods -l app=postgres
kubectl get pvc

# Connect to PostgreSQL
kubectl exec -it postgres-0 -- psql -U pgadmin -d mydb

# Check replication status
kubectl exec -it postgres-0 -- psql -U pgadmin -c "SELECT * FROM pg_stat_replication;"

PostgreSQL Operator (CloudNativePG) in 2026: automating cluster management

Manual StatefulSet management is fine for learning, but in production, using an operator is strongly recommended. CloudNativePG is the de facto standard for running PostgreSQL in Kubernetes as of 2026. It is a CNCF project actively maintained by the EDB (EnterpriseDB) team.

What CloudNativePG provides

  • Automatic management of primary/replica topology with failover.
  • Built-in support for streaming replication and replication slots.
  • Native integration with pg_basebackup, Barman, and WAL-G for backups.
  • User and database management via CRDs.
  • Automatic application of PostgreSQL configuration changes without downtime.
  • Support for scheduled backups and point-in-time recovery (PITR).
  • Out-of-the-box integration with Prometheus and Grafana.

Installing CloudNativePG

# Install via kubectl
kubectl apply -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.25/releases/cnpg-1.25.0.yaml

# Or via Helm
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm upgrade --install cnpg \
  --namespace cnpg-system \
  --create-namespace \
  cnpg/cloudnative-pg

# Verify installation
kubectl get pods -n cnpg-system

Creating a PostgreSQL cluster with CloudNativePG

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-cluster
  namespace: production
spec:
  instances: 3

  imageName: ghcr.io/cloudnative-pg/postgresql:16.3

  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "512MB"
      effective_cache_size: "1536MB"
      log_statement: "ddl"
      log_min_duration_statement: "1000"
    pg_hba:
      - host all all 10.0.0.0/8 scram-sha-256

  bootstrap:
    initdb:
      database: myapp
      owner: myapp_user
      secret:
        name: myapp-db-credentials

  storage:
    size: 100Gi
    storageClass: fast-ssd

  walStorage:
    size: 20Gi
    storageClass: fast-ssd

  backup:
    barmanObjectStore:
      destinationPath: s3://my-postgres-backups/cnpg
      s3Credentials:
        accessKeyId:
          name: s3-credentials
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: s3-credentials
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
        maxParallel: 8
    retentionPolicy: "30d"

  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "4Gi"
      cpu: "4"

  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname

  monitoring:
    enablePodMonitor: true

This manifest creates a three-node PostgreSQL 16 cluster with automatic replication, S3 backups, and built-in monitoring. CloudNativePG automatically assigns primary and replica roles, manages failover, and handles node updates.

Checking cluster status via the kubectl plugin

# Install the cnpg plugin
kubectl krew install cnpg

# Cluster status
kubectl cnpg status postgres-cluster -n production

# Manual primary switchover
kubectl cnpg promote postgres-cluster postgres-cluster-2 -n production

# View logs
kubectl cnpg logs cluster postgres-cluster -n production

Backup and restore for PostgreSQL in Kubernetes

Backups are a critical part of production operations. CloudNativePG supports two modes: physical backups via pg_basebackup/Barman and WAL archiving for PITR.

Scheduled Backup

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: postgres-daily-backup
  namespace: production
spec:
  schedule: "0 2 * * *"  # Every day at 02:00 UTC
  backupOwnerReference: self
  cluster:
    name: postgres-cluster
  method: barmanObjectStore
  immediate: true

Point-in-Time Recovery (PITR)

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-restored
  namespace: production
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.3

  bootstrap:
    recovery:
      source: postgres-cluster
      recoveryTarget:
        targetTime: "2026-03-15 14:30:00"

  externalClusters:
    - name: postgres-cluster
      barmanObjectStore:
        destinationPath: s3://my-postgres-backups/cnpg
        s3Credentials:
          accessKeyId:
            name: s3-credentials
            key: ACCESS_KEY_ID
          secretAccessKey:
            name: s3-credentials
            key: ACCESS_SECRET_KEY

  storage:
    size: 100Gi
    storageClass: fast-ssd

PITR allows you to restore the database to any point in time for which WAL segments are available. This is invaluable in the event of an accidental DROP TABLE or an application-level logical error.

Manual backup with Velero

For an additional layer of protection, Velero can be used to back up PVCs at the Kubernetes level. However, for PostgreSQL, application-consistent backups via Barman or pg_dump are preferred, since Velero takes filesystem-level snapshots, which can result in an inconsistent state if the snapshot is taken while the database is running.

Network policies and security for stateful services

Security for stateful services in Kubernetes requires extra attention: a data breach from a database can be catastrophic. Apply the principle of least privilege.

NetworkPolicy for PostgreSQL

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-network-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      cnpg.io/cluster: postgres-cluster
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow traffic only from the application
    - from:
        - namespaceSelector:
            matchLabels:
              name: production
          podSelector:
            matchLabels:
              app: myapp
      ports:
        - protocol: TCP
          port: 5432
    # Allow intra-cluster replication
    - from:
        - podSelector:
            matchLabels:
              cnpg.io/cluster: postgres-cluster
      ports:
        - protocol: TCP
          port: 5432
  egress:
    # Allow replication between nodes
    - to:
        - podSelector:
            matchLabels:
              cnpg.io/cluster: postgres-cluster
      ports:
        - protocol: TCP
          port: 5432
    # Allow DNS
    - ports:
        - protocol: UDP
          port: 53
    # Allow backups to S3
    - ports:
        - protocol: TCP
          port: 443

Pod Security and RBAC

Additional security measures for production:

  • Use PodSecurityAdmission with the restricted policy for the namespace containing PostgreSQL.
  • Configure SecurityContext with runAsNonRoot: true, readOnlyRootFilesystem: true, and allowPrivilegeEscalation: false.
  • Restrict RBAC: the PostgreSQL ServiceAccount should not have access to secrets in other namespaces.
  • Use TLS for connections between replicas (CloudNativePG enables this by default).
  • Enable etcd encryption to protect Secrets at the Kubernetes cluster level.
  • Rotate passwords via an external vault and use short-lived credentials.

PodDisruptionBudget for high availability

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: postgres-pdb
  namespace: production
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      cnpg.io/cluster: postgres-cluster

A PDB ensures that Kubernetes will not delete more than one pod at a time during a node drain or cluster upgrade, preserving the quorum required for replication.

Monitoring PostgreSQL in Kubernetes: pg_exporter and Grafana

Without comprehensive monitoring, managing production PostgreSQL is impossible. CloudNativePG includes a built-in Prometheus-compatible endpoint, but for detailed monitoring, using postgres_exporter is recommended.

Deploying postgres_exporter

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-exporter
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-exporter
  template:
    metadata:
      labels:
        app: postgres-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9187"
    spec:
      containers:
        - name: postgres-exporter
          image: prometheuscommunity/postgres-exporter:v0.15.0
          ports:
            - containerPort: 9187
          env:
            - name: DATA_SOURCE_NAME
              valueFrom:
                secretKeyRef:
                  name: postgres-exporter-secret
                  key: datasource
          args:
            - --collector.stat_statements
            - --collector.replication
            - --collector.replication_slot
            - --collector.long_running_transactions
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "200m"

PodMonitor for CloudNativePG

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: postgres-cluster-monitor
  namespace: production
spec:
  selector:
    matchLabels:
      cnpg.io/cluster: postgres-cluster
  podMetricsEndpoints:
    - port: metrics
      interval: 30s
      scrapeTimeout: 25s

Key metrics for a Grafana dashboard

Set up alerts and dashboards for the following metrics:

  • pg_stat_database_tup_fetched — number of fetch operations per database.
  • pg_stat_replication_pg_wal_lsn_diff — replication lag in bytes. Alert when it exceeds 100MB.
  • pg_locks_count — number of locks. An increase indicates query issues.
  • pg_stat_bgwriter_buffers_alloc — pressure on the buffer cache.
  • cnpg_collector_pg_postmaster_start_time — time of the last PostgreSQL restart.
  • pg_database_size_bytes — database sizes. Used for storage expansion planning.
  • pg_stat_statements_mean_exec_time_seconds — average query execution time.

Grafana Dashboard

Use the ready-made CloudNativePG dashboard for Grafana (ID: 20417) — it includes all critical cluster metrics and individual panels for each instance. For postgres_exporter, use dashboard ID 9628.

Conclusion and recommendations

Running PostgreSQL in Kubernetes is no longer exotic — in 2026, it is a mature practice with a rich ecosystem of tools. Here is a summary of key production recommendations:

  • Use CloudNativePG instead of manually managing StatefulSets. The operator handles failover, replication management, backups, and upgrades.
  • Always configure StorageClass with reclaimPolicy: Retain. Data loss due to accidental PVC deletion is unacceptable.
  • Place replicas in different availability zones using affinity rules and topologySpreadConstraints.
  • Set up WAL archiving from the very beginning. PITR is a lifesaver for logical errors that physical backups cannot cover.
  • Use NetworkPolicy to restrict access to PostgreSQL to authorized services only.
  • Don't run PostgreSQL on shared nodes with resource-intensive applications. Use a dedicated node pool with taints and tolerations for isolation.
  • Regularly test your restore process. A backup that has never been tested is not a backup.
  • Monitor replication lag and WAL size. These metrics are the first to signal performance problems.

The Kubernetes ecosystem continues to evolve, and stateful applications in Kubernetes are becoming increasingly reliable. A properly configured PostgreSQL cluster in Kubernetes with CloudNativePG, solid monitoring, and automated backups can meet the demands of even the most heavily loaded production systems.

Technologies

Tags

Ruslan Ismailov

Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →