ontroller-Runtime Framework:** Use kubebuilder which wraps controller-runtime. This provides robust caching, rate-limiting queues, and leader election out of the box. 3. Status Subresource: Always separate spec (desired state) from status (observed state). This prevents infinite reconcile loops caused by status updates triggering events. 4. Finalizers: Implement finalizers to ensure cleanup logic runs before resource deletion, preventing orphaned external resources (e.g., cloud volumes, DNS records).

Step-by-Step Implementation

We will build a DatabaseCluster operator using Go and Kubebuilder. This operator manages PostgreSQL clusters, handling deployment, service creation, and status reporting.

1. Environment Setup

Initialize the project structure.

kubebuilder init --domain example.com --repo github.com/myorg/db-operator
kubebuilder create api --group apps --version v1 --kind DatabaseCluster

2. Define the CRD Schema

Edit api/v1/databasecluster_types.go. Define the spec and status structures.

type DatabaseClusterSpec struct {
    // Size is the number of replicas for the database cluster.
    // +kubebuilder:validation:Minimum=1
    Size int32 `json:"size"`
    
    // Version specifies the database engine version.
    Version string `json:"version"`
    
    // StorageSize defines the persistent volume claim size.
    StorageSize string `json:"storageSize"`
}

type DatabaseClusterStatus struct {
    // Phase indicates the current phase of the cluster.
    Phase ClusterPhase `json:"phase"`
    
    // Conditions represent the latest available observations of the cluster's state.
    Conditions []metav1.Condition `json:"conditions,omitempty"`
    
    // Endpoint is the connection string for the cluster.
    Endpoint string `json:"endpoint,omitempty"`
}

type ClusterPhase string

const (
    PhaseCreating   ClusterPhase = "Creating"
    PhaseRunning    ClusterPhase = "Running"
    PhaseScaling    ClusterPhase = "Scaling"
    PhaseFailed     ClusterPhase = "Failed"
)

3. Implement the Reconcile Loop

Edit controllers/databasecluster_controller.go. The reconcile function is the heart of the operator.

func (r *DatabaseClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := r.Log.WithValues("databasecluster", req.NamespacedName)

    var dbCluster appsv1.DatabaseCluster
    if err := r.Get(ctx, req.NamespacedName, &dbCluster); err != nil {
        if apierrors.IsNotFound(err) {
            // Object not found, likely deleted. Finalizers handle cleanup.
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    // 1. Define Desired State
    desiredDeployment := r.buildDeployment(&dbCluster)
    desiredService := r.buildService(&dbCluster)

    // 2. Reconcile Deployment
    if err := r.reconcileResource(ctx, &dbCluster, desiredDeployment); err != nil {
        return r.updateStatus(ctx, &dbCluster, appsv1.PhaseFailed, err)
    }

    // 3. Reconcile Service
    if err := r.reconcileResource(ctx, &dbCluster, desiredService); err != nil {
        return r.updateStatus(ctx, &dbCluster, appsv1.PhaseFailed, err)
    }

    // 4. Update Status to Running
    return r.updateStatus(ctx, &dbCluster, appsv1.PhaseRunning, nil)
}

4. Idempotent Resource Management

The reconcileResource helper ensures the actual state matches the desired state.

func (r *DatabaseClusterReconciler) reconcileResource(ctx context.Context, owner metav1.Object, desired runtime.Object) error {
    // Create or Update logic
    // Use controllerutil.SetControllerReference to set owner references
    // This ensures child resources are garbage collected when the CR is deleted.
    
    mutateFn := func() (controllerutil.MutateResult, error) {
        // Merge desired spec into existing object
        existing := desired.DeepCopyObject().(runtime.Object)
        if err := r.Client.Get(ctx, client.ObjectKeyFromObject(desired), existing); err != nil {
            if apierrors.IsNotFound(err) {
                return controllerutil.MutateResultCreated, r.Client.Create(ctx, desired)
            }
            return controllerutil.MutateResultOther, err
        }
        
        // Update fields based on desired state
        // ... merge logic ...
        return controllerutil.MutateResultUpdated, r.Client.Update(ctx, desired)
    }

    _, err := controllerutil.CreateOrUpdate(ctx, r.Client, desired.(client.Object), mutateFn)
    return err
}

5. Status Updates

Always update status using Status().Update() to avoid triggering unnecessary reconciles.

func (r *DatabaseClusterReconciler) updateStatus(ctx context.Context, dbCluster *appsv1.DatabaseCluster, phase appsv1.ClusterPhase, err error) (ctrl.Result, error) {
    // Patch status
    patch := client.MergeFrom(dbCluster.DeepCopy())
    dbCluster.Status.Phase = phase
    if err != nil {
        meta.SetStatusCondition(&dbCluster.Status.Conditions, metav1.Condition{
            Type:    "Ready",
            Status:  metav1.ConditionFalse,
            Reason:  "ReconcileError",
            Message: err.Error(),
        })
    } else {
        meta.SetStatusCondition(&dbCluster.Status.Conditions, metav1.Condition{
            Type:    "Ready",
            Status:  metav1.ConditionTrue,
            Reason:  "Reconciled",
        })
    }
    
    if updateErr := r.Client.Status().Patch(ctx, dbCluster, patch); updateErr != nil {
        log.Error(updateErr, "Failed to update status")
        return ctrl.Result{}, updateErr
    }
    
    return ctrl.Result{}, nil
}

Pitfall Guide

1. Edge-Driven Logic

Mistake: Writing logic that depends on the type of event (e.g., if event.Type == Update). Impact: Breaks idempotency. If the controller restarts, it misses events and fails to reconcile. Best Practice: Never check event types in the reconcile loop. Always fetch the current CR state and drive the cluster to the desired state.

2. Neglecting the Status Subresource

Mistake: Storing observed state in the spec or updating spec during reconciliation. Impact: Causes infinite reconcile loops. The controller updates the object, triggering a new event, which triggers another update. Best Practice: Use status subresource for all observed data. Enable +kubebuilder:subresource:status in the types file.

3. Finalizer Deadlocks

Mistake: Implementing finalizers that block deletion indefinitely due to external API failures or lack of timeout handling. Impact: The CR remains in Terminating state, blocking namespace deletion and causing resource leaks. Best Practice: Implement retry logic with exponential backoff for finalizer cleanup. Log errors clearly and consider a timeout mechanism for non-critical cleanup steps.

4. RBAC Over-Provisioning

Mistake: Granting the operator controller cluster-admin or broad permissions to speed up development. Impact: Security vulnerability. If the operator is compromised, the attacker gains full cluster access. Best Practice: Follow the principle of least privilege. Define specific RBAC rules using +kubebuilder:rbac markers. Only grant permissions for the exact resources the operator manages.

5. Ignoring Schema Versioning

Mistake: Modifying the CRD schema without implementing conversion webhooks or versioning strategies. Impact: Existing clusters become unreadable or corrupted after operator upgrades. Best Practice: Use multiple API versions (v1, v2) and implement conversion.Webhook to handle translations between versions. Never break backward compatibility in a minor release.

6. Missing Immutability Checks

Mistake: Allowing changes to immutable fields (e.g., changing the database engine version from Postgres to MySQL) via spec updates. Impact: The operator may attempt impossible transitions, leading to data loss or crash loops. Best Practice: Validate spec changes in the reconcile loop. If an immutable field changes, update the status to Failed with a descriptive error and halt reconciliation.

7. Lack of Unit Testing with EnvTest

Mistake: Testing operators only in a live cluster. Impact: Slow feedback loops, flaky tests due to cluster state, and inability to test edge cases safely. Best Practice: Use envtest from controller-runtime to run unit tests against a real API server and etcd instance running locally. This allows testing of CRD validation, controller logic, and race conditions without a full cluster.

Production Bundle

Action Checklist

• Define CRD Validation: Add +kubebuilder:validation markers for all spec fields to prevent invalid resources at the API level.
• Implement Idempotent Reconcile: Ensure the reconcile function produces the same result regardless of execution frequency or order.
• Enable Status Subresource: Configure status subresource in the CRD and controller to separate desired and observed state.
• Configure RBAC Limits: Review and restrict RBAC permissions to the minimum required for the operator's scope.
• Add Prometheus Metrics: Expose metrics for reconcile duration, errors, and custom resource states using controller-runtime metrics.
• Test with EnvTest: Write comprehensive unit tests covering normal flows, error scenarios, and finalizer logic using envtest.
• Handle Finalizers: Implement finalizers for all external resources or critical cleanup tasks, ensuring graceful deletion.
• Version the API: Plan for schema evolution; use API versioning strategies to support rolling upgrades of the operator.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Stateless Microservices	Helm / Kustomize	Standard, low overhead, sufficient for declarative deployment.	Low
Stateful Database with Backups	Kubernetes Operator	Requires lifecycle logic, backups, and state-aware scaling.	Medium
Multi-Cluster Sync	Kubernetes Operator	Complex coordination logic impossible with static manifests.	High
Internal Tooling / Prototypes	Custom Scripting	Speed of implementation outweighs long-term maintenance needs.	Low
Regulated Data Workloads	Kubernetes Operator	Auditability, consistent state enforcement, and automated compliance checks.	Medium

Configuration Template

CRD Definition with Validation:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databaseclusters.apps.example.com
spec:
  group: apps.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["size", "version"]
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 10
                  description: "Number of replicas"
                version:
                  type: string
                  pattern: "^[0-9]+\\.[0-9]+\\.[0-9]+$"
                  description: "Semantic version of the database"
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: ["Creating", "Running", "Scaling", "Failed"]
                endpoint:
                  type: string
  scope: Namespaced
  names:
    plural: databaseclusters
    singular: databasecluster
    kind: DatabaseCluster
    shortNames:
      - dbc

Controller RBAC Markers:

// +kubebuilder:rbac:groups=apps.example.com,resources=databaseclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.example.com,resources=databaseclusters/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps.example.com,resources=databaseclusters/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete

Quick Start Guide

1. Install Kubebuilder:

   curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
   chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/
   

2. Initialize Project:

   mkdir my-operator && cd my-operator
   kubebuilder init --domain mycompany.com --repo github.com/myorg/my-operator
   

3. Create API:

   kubebuilder create api --group mygroup --version v1 --kind MyResource
   

4. Implement Logic: Edit api/v1/myresource_types.go to add spec/status. Edit controllers/myresource_controller.go to implement the Reconcile function. Run make generate and make manifests.

5. Deploy and Test:

   # Install CRDs
   make install
   
   # Run controller locally
   make run
   
   # Create a test resource
   kubectl apply -f config/samples/mygroup_v1_myresource.yaml
   

Result: The operator is running, watching for MyResource instances, and reconciling state automatically. Use kubectl get myresource to observe status updates.