DevOps

Kubernetes Operators in Go: Automating Custom Resource Management in 2026

Ruslan Ismailov Published 18 min read
K

Introduction: What Is a Kubernetes Operator and Why Do You Need One

Kubernetes provides a powerful declarative API for infrastructure management, but standard abstractions — Deployment, StatefulSet, Service — are often insufficient for managing complex stateful applications. Helm charts solve the problem of manifest templating but cannot react to cluster events or make real-time decisions. Kubernetes Operators fill this gap by encoding operational knowledge (runbooks) directly into a controller running inside the cluster.

An Operator is a Kubernetes extension pattern consisting of two parts: a Custom Resource Definition (CRD) that describes the desired state, and a controller that reconciles the actual cluster state with the desired one. Unlike Helm, an Operator can perform scheduled backups, automatically recover a PostgreSQL cluster after a failure, rotate secrets, and execute rolling upgrades with custom business logic.

By 2026, the Operator pattern has become the de facto standard for platform teams: OperatorHub.io lists over 400 ready-made solutions, and the Go toolchain — kubebuilder and controller-runtime — has reached a high level of maturity. In this article, we'll walk through everything from core concepts to a working Operator for managing PostgreSQL instances.

Key Concepts: CRD, Controller, and Reconciliation Loop

Custom Resource Definition

A CRD is an extension of the Kubernetes API schema. Once a CRD manifest is applied, the cluster starts accepting objects of the new kind — for example, PostgreSQLCluster. Users interact with it via kubectl just as they would with a Deployment.

apiVersion: apiextensions.k8s.io/v1\nkind: CustomResourceDefinition\nmetadata:\n  name: postgresqlclusters.db.example.com\nspec:\n  group: db.example.com\n  versions:\n    - name: v1alpha1\n      served: true\n      storage: true\n      schema:\n        openAPIV3Schema:\n          type: object\n          properties:\n            spec:\n              type: object\n              properties:\n                replicas:\n                  type: integer\n                  minimum: 1\n                version:\n                  type: string\n            status:\n              type: object\n              properties:\n                phase:\n                  type: string\n  scope: Namespaced\n  names:\n    plural: postgresqlclusters\n    singular: postgresqlcluster\n    kind: PostgreSQLCluster\n

Reconciliation Loop

The heart of any Operator is the reconciliation loop. The controller subscribes to events (object creation, modification, deletion) and calls the Reconcile function on each event. The function fetches the current state from the cluster and decides what actions are needed to bring the actual state in line with the desired state described in spec. The most important property of a reconciler is idempotency: calling it repeatedly with the same state must not produce any side effects.

"Don't think about events — think about state. A reconciler always answers one question: what needs to happen right now to reach the desired state?"

Tooling: kubebuilder vs operator-sdk in 2026

The two main scaffolding tools for Go operators are kubebuilder (maintained by sig-controller-tools) and operator-sdk (Red Hat). In 2026, the difference between them is minimal: operator-sdk uses kubebuilder as its foundation and adds OLM (Operator Lifecycle Manager) integration along with plugins for Ansible/Helm operators.

  • kubebuilder — the choice for teams that want minimal overhead and full control over the codebase.
  • operator-sdk — the choice for teams planning to publish on OperatorHub or working within the OpenShift ecosystem.
  • Both use controller-runtime — a Go library that abstracts the low-level details of working with the Kubernetes API.

In this article, we use kubebuilder v4, which is current as of 2026.

Step-by-Step Operator Creation in Go

Step 1: Project Scaffolding

Install kubebuilder and initialize the project:

# Install kubebuilder\ncurl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)\nchmod +x kubebuilder && mv kubebuilder /usr/local/bin/\n\n# Initialize the project\nmkdir postgres-operator && cd postgres-operator\nkubebuilder init --domain example.com --repo github.com/example/postgres-operator\n\n# Create API and controller\nkubebuilder create api --group db --version v1alpha1 --kind PostgreSQLCluster\n

After running these commands, kubebuilder generates the project structure: api/v1alpha1/ contains Go types for the CRD, internal/controller/ holds the reconciler scaffold, and config/ contains kustomize manifests.

Step 2: Defining CRD Types in Go

// api/v1alpha1/postgresqlcluster_types.go
package v1alpha1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

// PostgreSQLClusterSpec describes the desired state of the cluster
type PostgreSQLClusterSpec struct {
    // Replicas — number of replicas (1 primary + N standbys)
    Replicas int32 `json:"replicas"`
    // Version — PostgreSQL version, e.g. "16.2"
    Version string `json:"version"`
    // StorageSize — PVC size for each instance
    StorageSize string `json:"storageSize"`
    // BackupSchedule — cron schedule for backups
    BackupSchedule string `json:"backupSchedule,omitempty"`
}

// PostgreSQLClusterStatus reflects the actual state
type PostgreSQLClusterStatus struct {
    Phase      string `json:"phase,omitempty"`
    ReadyNodes int32  `json:"readyNodes,omitempty"`
    Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase"
// +kubebuilder:printcolumn:name="Ready",type="integer",JSONPath=".status.readyNodes"
type PostgreSQLCluster struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec   PostgreSQLClusterSpec   `json:"spec,omitempty"`
    Status PostgreSQLClusterStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true
type PostgreSQLClusterList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items []PostgreSQLCluster `json:"items"`
}

func init() {
    SchemeBuilder.Register(&PostgreSQLCluster{}, &PostgreSQLClusterList{})
}

After modifying the types, run make generate manifests — controller-gen will automatically update the CRD YAML and generate DeepCopy methods.

Step 3: Implementing the Reconciler

// internal/controller/postgresqlcluster_controller.go
package controller

import (
    "context"
    "fmt"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    dbv1alpha1 "github.com/example/postgres-operator/api/v1alpha1"
)

type PostgreSQLClusterReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=db.example.com,resources=postgresqlclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=db.example.com,resources=postgresqlclusters/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete

func (r *PostgreSQLClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // 1. Fetch the object from the cluster
    cluster := &dbv1alpha1.PostgreSQLCluster{}
    if err := r.Get(ctx, req.NamespacedName, cluster); err != nil {
        if errors.IsNotFound(err) {
            // Object deleted — nothing to do
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, fmt.Errorf("failed to get PostgreSQLCluster: %w", err)
    }

    logger.Info("Reconciling", "name", cluster.Name, "phase", cluster.Status.Phase)

    // 2. Ensure the StatefulSet exists and matches the spec
    if err := r.reconcileStatefulSet(ctx, cluster); err != nil {
        return ctrl.Result{}, err
    }

    // 3. Ensure the Service exists
    if err := r.reconcileService(ctx, cluster); err != nil {
        return ctrl.Result{}, err
    }

    // 4. Update status
    if err := r.updateStatus(ctx, cluster); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

func (r *PostgreSQLClusterReconciler) reconcileStatefulSet(
    ctx context.Context,
    cluster *dbv1alpha1.PostgreSQLCluster,
) error {
    desired := r.buildStatefulSet(cluster)

    // Set owner reference for garbage collection
    if err := ctrl.SetControllerReference(cluster, desired, r.Scheme); err != nil {
        return err
    }

    existing := &appsv1.StatefulSet{}
    err := r.Get(ctx, client.ObjectKeyFromObject(desired), existing)
    if errors.IsNotFound(err) {
        return r.Create(ctx, desired)
    }
    if err != nil {
        return err
    }

    // Update only if replicas or image changed
    existing.Spec.Replicas = desired.Spec.Replicas
    existing.Spec.Template = desired.Spec.Template
    return r.Update(ctx, existing)
}

func (r *PostgreSQLClusterReconciler) buildStatefulSet(
    cluster *dbv1alpha1.PostgreSQLCluster,
) *appsv1.StatefulSet {
    image := fmt.Sprintf("postgres:%s", cluster.Spec.Version)
    replicas := cluster.Spec.Replicas

    return &appsv1.StatefulSet{
        ObjectMeta: metav1.ObjectMeta{
            Name:      cluster.Name,
            Namespace: cluster.Namespace,
        },
        Spec: appsv1.StatefulSetSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{"app": cluster.Name},
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{"app": cluster.Name},
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{
                        {
                            Name:  "postgres",
                            Image: image,
                            Env: []corev1.EnvVar{
                                {
                                    Name:  "POSTGRES_PASSWORD",
                                    Value: "changeme", // in production, use a Secret
                                },
                            },
                        },
                    },
                },
            },
        },
    }
}

func (r *PostgreSQLClusterReconciler) reconcileService(
    ctx context.Context,
    cluster *dbv1alpha1.PostgreSQLCluster,
) error {
    svc := &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      cluster.Name + "-svc",
            Namespace: cluster.Namespace,
        },
        Spec: corev1.ServiceSpec{
            Selector: map[string]string{"app": cluster.Name},
            Ports: []corev1.ServicePort{
                {Port: 5432},
            },
        },
    }
    if err := ctrl.SetControllerReference(cluster, svc, r.Scheme); err != nil {
        return err
    }
    existing := &corev1.Service{}
    if err := r.Get(ctx, client.ObjectKeyFromObject(svc), existing); errors.IsNotFound(err) {
        return r.Create(ctx, svc)
    } else {
        return err
    }
}

func (r *PostgreSQLClusterReconciler) updateStatus(
    ctx context.Context,
    cluster *dbv1alpha1.PostgreSQLCluster,
) error {
    cluster.Status.Phase = "Running"
    cluster.Status.ReadyNodes = cluster.Spec.Replicas
    return r.Status().Update(ctx, cluster)
}

func (r *PostgreSQLClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&dbv1alpha1.PostgreSQLCluster{}).
        Owns(&appsv1.StatefulSet{}).
        Owns(&corev1.Service{}).
        Complete(r)
}

Practical Example: PostgreSQL Backup and Restore

Let's extend the Operator with backup functionality. We'll add a PostgreSQLBackup CRD and a reconciler that creates a Kubernetes Job to run pg_dump.

// Backup reconciler fragment
func (r *PostgreSQLBackupReconciler) Reconcile(
    ctx context.Context,
    req ctrl.Request,
) (ctrl.Result, error) {
    backup := &dbv1alpha1.PostgreSQLBackup{}
    if err := r.Get(ctx, req.NamespacedName, backup); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Idempotency: skip if Job already exists
    job := &batchv1.Job{}
    jobName := fmt.Sprintf("%s-backup-job", backup.Name)
    err := r.Get(ctx, types.NamespacedName{
        Name:      jobName,
        Namespace: backup.Namespace,
    }, job)

    if errors.IsNotFound(err) {
        newJob := r.buildBackupJob(backup, jobName)
        if err := ctrl.SetControllerReference(backup, newJob, r.Scheme); err != nil {
            return ctrl.Result{}, err
        }
        if err := r.Create(ctx, newJob); err != nil {
            return ctrl.Result{}, fmt.Errorf("failed to create backup job: %w", err)
        }
        backup.Status.Phase = "Running"
        return ctrl.Result{RequeueAfter: 30 * time.Second},
            r.Status().Update(ctx, backup)
    }

    // Check Job result
    if job.Status.Succeeded > 0 {
        backup.Status.Phase = "Completed"
        backup.Status.CompletedAt = &metav1.Time{Time: time.Now()}
    } else if job.Status.Failed > 3 {
        backup.Status.Phase = "Failed"
    } else {
        // Job still running
        return ctrl.Result{RequeueAfter: 15 * time.Second}, nil
    }

    return ctrl.Result{}, r.Status().Update(ctx, backup)
}

func (r *PostgreSQLBackupReconciler) buildBackupJob(
    backup *dbv1alpha1.PostgreSQLBackup,
    name string,
) *batchv1.Job {
    clusterSvc := backup.Spec.ClusterName + "-svc"
    return &batchv1.Job{
        ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: backup.Namespace},
        Spec: batchv1.JobSpec{
            Template: corev1.PodTemplateSpec{
                Spec: corev1.PodSpec{
                    RestartPolicy: corev1.RestartPolicyOnFailure,
                    Containers: []corev1.Container{
                        {
                            Name:  "pg-dump",
                            Image: "postgres:16",
                            Command: []string{
                                "pg_dump",
                                "-h", clusterSvc,
                                "-U", "postgres",
                                "-F", "c",
                                "-f", "/backup/dump.pgc",
                                backup.Spec.Database,
                            },
                        },
                    },
                },
            },
        },
    }
}

Error Handling and Idempotency

The reconciliation loop must be resilient to network failures, version conflicts (optimistic locking), and repeated invocations. Key rules:

  • Always re-fetch the object at the start of Reconcile — don't rely on the cached state from the event.
  • Use ctrl.Result{RequeueAfter: ...} for deferred checks instead of a blocking sleep.
  • Wrap errors using fmt.Errorf("...: %w", err) for proper error tracing.
  • Handle conflicts: when errors.IsConflict(err) occurs, simply return the error — the controller will automatically retry the reconcile.
  • Use finalizers for cleanup logic when an object is deleted: add the finalizer on creation, remove it after cleanup is complete.
// Finalizer usage example
const finalizer = "db.example.com/cleanup"

if cluster.DeletionTimestamp.IsZero() {
    // Object is not being deleted — add finalizer
    if !controllerutil.ContainsFinalizer(cluster, finalizer) {
        controllerutil.AddFinalizer(cluster, finalizer)
        return ctrl.Result{}, r.Update(ctx, cluster)
    }
} else {
    // Object is being deleted — perform cleanup
    if controllerutil.ContainsFinalizer(cluster, finalizer) {
        if err := r.cleanupExternalResources(ctx, cluster); err != nil {
            return ctrl.Result{}, err
        }
        controllerutil.RemoveFinalizer(cluster, finalizer)
        return ctrl.Result{}, r.Update(ctx, cluster)
    }
}

Testing the Operator: envtest and Integration Tests

controller-runtime ships with the envtest package, which runs a real Kubernetes API server and etcd locally — without a full cluster. This enables writing integration tests that verify the complete reconciliation cycle.

// internal/controller/suite_test.go
package controller_test

import (
    "testing"
    "path/filepath"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

var (
    testEnv *envtest.Environment
    k8sClient client.Client
)

func TestControllers(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Controller Suite")
}

var _ = BeforeSuite(func() {
    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")},
    }
    cfg, err := testEnv.Start()
    Expect(err).NotTo(HaveOccurred())

    k8sClient, err = client.New(cfg, client.Options{})
    Expect(err).NotTo(HaveOccurred())
})

var _ = AfterSuite(func() {
    Expect(testEnv.Stop()).To(Succeed())
})

// internal/controller/postgresqlcluster_controller_test.go
var _ = Describe("PostgreSQLCluster Controller", func() {
    It("should create a StatefulSet for a new cluster", func() {
        ctx := context.Background()
        cluster := &dbv1alpha1.PostgreSQLCluster{
            ObjectMeta: metav1.ObjectMeta{
                Name:      "test-cluster",
                Namespace: "default",
            },
            Spec: dbv1alpha1.PostgreSQLClusterSpec{
                Replicas:    2,
                Version:     "16.2",
                StorageSize: "10Gi",
            },
        }
        Expect(k8sClient.Create(ctx, cluster)).To(Succeed())

        sts := &appsv1.StatefulSet{}
        Eventually(func() error {
            return k8sClient.Get(ctx, types.NamespacedName{
                Name:      "test-cluster",
                Namespace: "default",
            }, sts)
        }, "10s", "1s").Should(Succeed())

        Expect(*sts.Spec.Replicas).To(Equal(int32(2)))
    })
})

Deploying the Operator: kustomize, Helm, and OLM

kubebuilder generates a ready-to-use config/ directory with kustomize manifests. Deployment is straightforward:

# Build and push the image
make docker-build docker-push IMG=ghcr.io/example/postgres-operator:v0.1.0

# Deploy with kustomize
make deploy IMG=ghcr.io/example/postgres-operator:v0.1.0

# Or directly
kubectl apply -k config/default

For production deployments, it is recommended to package the Operator as a Helm chart with configurable parameters via values.yaml. If you plan to publish on OperatorHub, use operator-sdk to generate an OLM bundle:

operator-sdk generate bundle \
  --package postgres-operator \
  --version 0.1.0 \
  --channels stable

operator-sdk bundle validate ./bundle

Security: RBAC and the Principle of Least Privilege

The Operator runs with the permissions of a ServiceAccount in the cluster. The // +kubebuilder:rbac: annotations in the controller code automatically generate a ClusterRole when you run make manifests. Follow the principle of least privilege:

  • Grant access only to the resources and namespaces that are actually needed.
  • Where possible, use a namespace-scoped Role instead of a ClusterRole.
  • Never grant the Operator cluster-admin privileges.
  • Use Admission Webhooks (ValidatingWebhookConfiguration) to validate CRDs at the API server level.
  • Store secrets (database passwords, S3 keys) in Kubernetes Secrets or External Secrets Operator — not in the CRD spec.
# Example of a generated ClusterRole (excerpt)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: postgres-operator-manager-role
rules:
- apiGroups: ["db.example.com"]
  resources: ["postgresqlclusters"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["db.example.com"]
  resources: ["postgresqlclusters/status"]
  verbs: ["get", "update", "patch"]
- apiGroups: ["apps"]
  resources: ["statefulsets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["services", "persistentvolumeclaims"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

Conclusion: When to Write an Operator vs. Using Standard Tools

A Kubernetes Operator is a powerful tool, but not a silver bullet. Here are the criteria for making the right decision:

  • Write an Operator when you need to automate operational procedures (backup/restore, failover, rolling upgrades with health checks) that require real-time reaction to cluster events.
  • Write an Operator when you have a complex stateful service (database, message queue, broker) with non-trivial scaling logic.
  • Use Helm or kustomize when the task comes down to manifest templating and configuration management without complex business logic.
  • Use an existing Operator (CloudNativePG for PostgreSQL, Strimzi for Kafka) if it meets your requirements — don't reinvent the wheel.

In 2026, the Kubernetes Operators ecosystem in Go has reached maturity: kubebuilder v4 + controller-runtime provide a solid foundation, and patterns like idempotent reconciliation and envtest-based testing have become industry standards. For platform engineers and Go developers, mastering this stack unlocks the ability to build truly self-managing infrastructure.

Technologies

Tags

Ruslan Ismailov

Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →