Stateful Applications in Kubernetes: StatefulSets, Persistent Volumes, and PostgreSQL in a Cluster
Introduction: stateful vs stateless — what's the difference and why it matters
Kubernetes was originally designed as a platform for stateless services: if a container crashes, a new one spins up and everything keeps running. But the real world works differently. Databases, message queues, caches with persistence — these are all stateful applications that store data and require a stable identity across restarts.
Key differences between stateful and stateless in the context of Kubernetes:
- Pod identity. Stateless pods are interchangeable: pod-abc and pod-xyz do the same thing. Stateful pods have stable names (e.g.,
postgres-0,postgres-1) and DNS records that persist across restarts. - Persistent storage. A stateless container doesn't need to save data to disk. PostgreSQL without a persistent volume will lose all data on restart.
- Startup and shutdown order. In a PostgreSQL cluster, it's important that the primary starts before the replicas. StatefulSet guarantees ordered creation and deletion of pods.
The StatefulSet object in Kubernetes was introduced specifically to address these challenges. Let's explore it in detail.
StatefulSets: how they work and how they differ from Deployments
StatefulSet is a Kubernetes controller specifically designed for managing stateful applications. Unlike a Deployment, it provides three guarantees:
- Stable network identifiers. Each pod gets a predictable DNS name in the format
<pod-name>.<service-name>.<namespace>.svc.cluster.local. For PostgreSQL, this meanspostgres-0.postgres-headless.default.svc.cluster.localalways points to the primary node. - Stable persistent storage. Each pod gets its own
PersistentVolumeClaim, which is not deleted when the pod restarts — data is preserved. - Ordered deployment and scaling. Pods are created in order (0, 1, 2...) and deleted in reverse order.
StatefulSet vs Deployment: a practical comparison
Deployment is suitable for API servers, frontends, and stateless worker processes. StatefulSet is required for PostgreSQL, Redis Cluster, Elasticsearch, Kafka — anywhere each instance is unique and stores data. When updating a Deployment, all pods can be replaced simultaneously (with RollingUpdate and zero delays). StatefulSet updates pods sequentially, which is critical for database clusters.
Minimal StatefulSet example for PostgreSQL
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: default
spec:
serviceName: postgres-headless
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: mydb
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-secret
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "2"
readinessProbe:
exec:
command: ["pg_isready", "-U", "$(POSTGRES_USER)", "-d", "$(POSTGRES_DB)"]
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
exec:
command: ["pg_isready", "-U", "$(POSTGRES_USER)"]
initialDelaySeconds: 30
periodSeconds: 10
volumeClaimTemplates:
- metadata:
name: postgres-data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 50Gi
Pay attention to the volumeClaimTemplates section — this is the key feature of StatefulSet. For each pod, a separate PVC is automatically created with the name postgres-data-postgres-0, postgres-data-postgres-1, and so on.
Headless Service for StatefulSet
apiVersion: v1
kind: Service
metadata:
name: postgres-headless
namespace: default
spec:
clusterIP: None
selector:
app: postgres
ports:
- port: 5432
targetPort: 5432
A Headless Service (with clusterIP: None) does not create a single virtual IP, but instead registers individual DNS records for each pod. This is the foundation for stable addressing of PostgreSQL cluster nodes.
Persistent Volumes and Persistent Volume Claims: configuring storage
Storage in Kubernetes is organized through three layers of abstraction:
- PersistentVolume (PV) — the actual storage resource: a cloud disk, NFS share, or local SSD. Created by a cluster administrator or dynamically via a StorageClass.
- PersistentVolumeClaim (PVC) — a storage request from a pod. Describes the required size, access mode, and StorageClass.
- StorageClass — a template for dynamic PV creation. Defines the disk type (SSD, HDD), reclaim policy, and provisioner.
StorageClass for production PostgreSQL
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
type: gp3
iops: "3000"
throughput: "125"
encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
Key parameters for production:
reclaimPolicy: Retain— when a PVC is deleted, the data on disk is preserved. Never useDeletefor production databases.volumeBindingMode: WaitForFirstConsumer— the disk is created in the same availability zone as the pod. Critical for multi-AZ clusters.allowVolumeExpansion: true— allows increasing the PVC size without recreation.encrypted: "true"— disk-level data encryption.
Expanding PVC size
To increase storage capacity, simply change the storage field in an existing PVC:
kubectl patch pvc postgres-data-postgres-0 \
-p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
Kubernetes will automatically expand the volume if the provisioner supports it and the StorageClass has allowVolumeExpansion: true.
Running PostgreSQL in Kubernetes: step by step
Let's walk through the complete process of deploying PostgreSQL in Kubernetes from scratch. We'll create a Secret, ConfigMap, Headless Service, external access Service, and StatefulSet.
Step 1: Secret with credentials
apiVersion: v1
kind: Secret
metadata:
name: postgres-secret
namespace: default
type: Opaque
stringData:
username: pgadmin
password: "S3cur3P@ssw0rd!"
replication-password: "R3pl1c@P@ss!"
In production, it is recommended to use external secret managers: HashiCorp Vault with the vault-secrets-operator, AWS Secrets Manager, or Sealed Secrets.
Step 2: ConfigMap with postgresql.conf
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-config
namespace: default
data:
postgresql.conf: |
max_connections = 200
shared_buffers = 512MB
effective_cache_size = 1536MB
maintenance_work_mem = 128MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 2621kB
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 4
max_parallel_workers_per_gather = 2
max_parallel_workers = 4
wal_level = replica
archive_mode = on
archive_command = 'test ! -f /archive/%f && cp %p /archive/%f'
hot_standby = on
log_timezone = 'UTC'
datestyle = 'iso, mdy'
timezone = 'UTC'
log_statement = 'ddl'
log_min_duration_statement = 1000
Step 3: Service for external access
apiVersion: v1
kind: Service
metadata:
name: postgres-primary
namespace: default
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
selector:
app: postgres
role: primary
ports:
- port: 5432
targetPort: 5432
type: ClusterIP
Step 4: Applying manifests and verifying
# Apply all manifests
kubectl apply -f postgres-secret.yaml
kubectl apply -f postgres-config.yaml
kubectl apply -f postgres-headless-svc.yaml
kubectl apply -f postgres-statefulset.yaml
# Check status
kubectl get statefulsets
kubectl get pods -l app=postgres
kubectl get pvc
# Connect to PostgreSQL
kubectl exec -it postgres-0 -- psql -U pgadmin -d mydb
# Check replication status
kubectl exec -it postgres-0 -- psql -U pgadmin -c "SELECT * FROM pg_stat_replication;"
PostgreSQL Operator (CloudNativePG) in 2026: automating cluster management
Manual StatefulSet management is fine for learning, but in production, using an operator is strongly recommended. CloudNativePG is the de facto standard for running PostgreSQL in Kubernetes as of 2026. It is a CNCF project actively maintained by the EDB (EnterpriseDB) team.
What CloudNativePG provides
- Automatic management of primary/replica topology with failover.
- Built-in support for streaming replication and replication slots.
- Native integration with pg_basebackup, Barman, and WAL-G for backups.
- User and database management via CRDs.
- Automatic application of PostgreSQL configuration changes without downtime.
- Support for scheduled backups and point-in-time recovery (PITR).
- Out-of-the-box integration with Prometheus and Grafana.
Installing CloudNativePG
# Install via kubectl
kubectl apply -f \
https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.25/releases/cnpg-1.25.0.yaml
# Or via Helm
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm upgrade --install cnpg \
--namespace cnpg-system \
--create-namespace \
cnpg/cloudnative-pg
# Verify installation
kubectl get pods -n cnpg-system
Creating a PostgreSQL cluster with CloudNativePG
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-cluster
namespace: production
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.3
postgresql:
parameters:
max_connections: "200"
shared_buffers: "512MB"
effective_cache_size: "1536MB"
log_statement: "ddl"
log_min_duration_statement: "1000"
pg_hba:
- host all all 10.0.0.0/8 scram-sha-256
bootstrap:
initdb:
database: myapp
owner: myapp_user
secret:
name: myapp-db-credentials
storage:
size: 100Gi
storageClass: fast-ssd
walStorage:
size: 20Gi
storageClass: fast-ssd
backup:
barmanObjectStore:
destinationPath: s3://my-postgres-backups/cnpg
s3Credentials:
accessKeyId:
name: s3-credentials
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-credentials
key: ACCESS_SECRET_KEY
wal:
compression: gzip
maxParallel: 8
retentionPolicy: "30d"
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "4Gi"
cpu: "4"
affinity:
enablePodAntiAffinity: true
topologyKey: kubernetes.io/hostname
monitoring:
enablePodMonitor: true
This manifest creates a three-node PostgreSQL 16 cluster with automatic replication, S3 backups, and built-in monitoring. CloudNativePG automatically assigns primary and replica roles, manages failover, and handles node updates.
Checking cluster status via the kubectl plugin
# Install the cnpg plugin
kubectl krew install cnpg
# Cluster status
kubectl cnpg status postgres-cluster -n production
# Manual primary switchover
kubectl cnpg promote postgres-cluster postgres-cluster-2 -n production
# View logs
kubectl cnpg logs cluster postgres-cluster -n production
Backup and restore for PostgreSQL in Kubernetes
Backups are a critical part of production operations. CloudNativePG supports two modes: physical backups via pg_basebackup/Barman and WAL archiving for PITR.
Scheduled Backup
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: postgres-daily-backup
namespace: production
spec:
schedule: "0 2 * * *" # Every day at 02:00 UTC
backupOwnerReference: self
cluster:
name: postgres-cluster
method: barmanObjectStore
immediate: true
Point-in-Time Recovery (PITR)
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-restored
namespace: production
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.3
bootstrap:
recovery:
source: postgres-cluster
recoveryTarget:
targetTime: "2026-03-15 14:30:00"
externalClusters:
- name: postgres-cluster
barmanObjectStore:
destinationPath: s3://my-postgres-backups/cnpg
s3Credentials:
accessKeyId:
name: s3-credentials
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-credentials
key: ACCESS_SECRET_KEY
storage:
size: 100Gi
storageClass: fast-ssd
PITR allows you to restore the database to any point in time for which WAL segments are available. This is invaluable in the event of an accidental DROP TABLE or an application-level logical error.
Manual backup with Velero
For an additional layer of protection, Velero can be used to back up PVCs at the Kubernetes level. However, for PostgreSQL, application-consistent backups via Barman or pg_dump are preferred, since Velero takes filesystem-level snapshots, which can result in an inconsistent state if the snapshot is taken while the database is running.
Network policies and security for stateful services
Security for stateful services in Kubernetes requires extra attention: a data breach from a database can be catastrophic. Apply the principle of least privilege.
NetworkPolicy for PostgreSQL
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: postgres-network-policy
namespace: production
spec:
podSelector:
matchLabels:
cnpg.io/cluster: postgres-cluster
policyTypes:
- Ingress
- Egress
ingress:
# Allow traffic only from the application
- from:
- namespaceSelector:
matchLabels:
name: production
podSelector:
matchLabels:
app: myapp
ports:
- protocol: TCP
port: 5432
# Allow intra-cluster replication
- from:
- podSelector:
matchLabels:
cnpg.io/cluster: postgres-cluster
ports:
- protocol: TCP
port: 5432
egress:
# Allow replication between nodes
- to:
- podSelector:
matchLabels:
cnpg.io/cluster: postgres-cluster
ports:
- protocol: TCP
port: 5432
# Allow DNS
- ports:
- protocol: UDP
port: 53
# Allow backups to S3
- ports:
- protocol: TCP
port: 443
Pod Security and RBAC
Additional security measures for production:
- Use
PodSecurityAdmissionwith therestrictedpolicy for the namespace containing PostgreSQL. - Configure
SecurityContextwithrunAsNonRoot: true,readOnlyRootFilesystem: true, andallowPrivilegeEscalation: false. - Restrict RBAC: the PostgreSQL ServiceAccount should not have access to secrets in other namespaces.
- Use TLS for connections between replicas (CloudNativePG enables this by default).
- Enable etcd encryption to protect Secrets at the Kubernetes cluster level.
- Rotate passwords via an external vault and use short-lived credentials.
PodDisruptionBudget for high availability
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: postgres-pdb
namespace: production
spec:
maxUnavailable: 1
selector:
matchLabels:
cnpg.io/cluster: postgres-cluster
A PDB ensures that Kubernetes will not delete more than one pod at a time during a node drain or cluster upgrade, preserving the quorum required for replication.
Monitoring PostgreSQL in Kubernetes: pg_exporter and Grafana
Without comprehensive monitoring, managing production PostgreSQL is impossible. CloudNativePG includes a built-in Prometheus-compatible endpoint, but for detailed monitoring, using postgres_exporter is recommended.
Deploying postgres_exporter
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-exporter
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: postgres-exporter
template:
metadata:
labels:
app: postgres-exporter
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9187"
spec:
containers:
- name: postgres-exporter
image: prometheuscommunity/postgres-exporter:v0.15.0
ports:
- containerPort: 9187
env:
- name: DATA_SOURCE_NAME
valueFrom:
secretKeyRef:
name: postgres-exporter-secret
key: datasource
args:
- --collector.stat_statements
- --collector.replication
- --collector.replication_slot
- --collector.long_running_transactions
resources:
requests:
memory: "64Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "200m"
PodMonitor for CloudNativePG
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: postgres-cluster-monitor
namespace: production
spec:
selector:
matchLabels:
cnpg.io/cluster: postgres-cluster
podMetricsEndpoints:
- port: metrics
interval: 30s
scrapeTimeout: 25s
Key metrics for a Grafana dashboard
Set up alerts and dashboards for the following metrics:
pg_stat_database_tup_fetched— number of fetch operations per database.pg_stat_replication_pg_wal_lsn_diff— replication lag in bytes. Alert when it exceeds 100MB.pg_locks_count— number of locks. An increase indicates query issues.pg_stat_bgwriter_buffers_alloc— pressure on the buffer cache.cnpg_collector_pg_postmaster_start_time— time of the last PostgreSQL restart.pg_database_size_bytes— database sizes. Used for storage expansion planning.pg_stat_statements_mean_exec_time_seconds— average query execution time.
Grafana Dashboard
Use the ready-made CloudNativePG dashboard for Grafana (ID: 20417) — it includes all critical cluster metrics and individual panels for each instance. For postgres_exporter, use dashboard ID 9628.
Conclusion and recommendations
Running PostgreSQL in Kubernetes is no longer exotic — in 2026, it is a mature practice with a rich ecosystem of tools. Here is a summary of key production recommendations:
- Use CloudNativePG instead of manually managing StatefulSets. The operator handles failover, replication management, backups, and upgrades.
- Always configure StorageClass with
reclaimPolicy: Retain. Data loss due to accidental PVC deletion is unacceptable. - Place replicas in different availability zones using affinity rules and
topologySpreadConstraints. - Set up WAL archiving from the very beginning. PITR is a lifesaver for logical errors that physical backups cannot cover.
- Use NetworkPolicy to restrict access to PostgreSQL to authorized services only.
- Don't run PostgreSQL on shared nodes with resource-intensive applications. Use a dedicated node pool with taints and tolerations for isolation.
- Regularly test your restore process. A backup that has never been tested is not a backup.
- Monitor replication lag and WAL size. These metrics are the first to signal performance problems.
The Kubernetes ecosystem continues to evolve, and stateful applications in Kubernetes are becoming increasingly reliable. A properly configured PostgreSQL cluster in Kubernetes with CloudNativePG, solid monitoring, and automated backups can meet the demands of even the most heavily loaded production systems.
Technologies
Tags
Ruslan Ismailov
Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →