Kubernetes operators guide

cas == nil || *sts.Spec.Replicas != int32(mydb.Spec.Size) { sts.Spec.Replicas = &[]int32{int32(mydb.Spec.Size)}[0] logger.Info("Updating StatefulSet replicas", "replicas", mydb.Spec.Size) return ctrl.Result{}, r.Update(ctx, sts) }

// 4. Update Status
// Reflect actual state back to the CR
if mydb.Status.Phase != "Running" {
	mydb.Status.Phase = "Running"
	mydb.Status.Nodes = []string{"node-0", "node-1"} // Example
	if err := r.Status().Update(ctx, &mydb); err != nil {
		return ctrl.Result{}, err
	}
}

return ctrl.Result{}, nil

}

func (r *MyDatabaseReconciler) statefulSetForCR(cr *examplev1.MyDatabase) *appsv1.StatefulSet { replicas := int32(cr.Spec.Size) return &appsv1.StatefulSet{ ObjectMeta: metav1.ObjectMeta{ Name: cr.Name, Namespace: cr.Namespace, }, Spec: appsv1.StatefulSetSpec{ Replicas: &replicas, // ... container spec, volume claims, etc. }, } }

func (r *MyDatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&examplev1.MyDatabase{}). Owns(&appsv1.StatefulSet{}). Owns(&corev1.Service{}). Complete(r) }

#### Key Implementation Patterns

1.  **Idempotency:** The `Reconcile` function must be safe to run multiple times. It should not assume the current state; it must fetch and compare.
2.  **Owner References:** Use `ctrl.SetControllerReference` to link child resources to the CR. This enables automatic garbage collection when the CR is deleted.
3.  **Finalizers:** Implement finalizers to handle cleanup logic (e.g., deleting persistent volumes or external cloud resources) before the CR is removed.
4.  **Status Updates:** Always update the `status` subresource. This provides observability into the operator's view of the system.

### Pitfall Guide

Production operators fail due to subtle implementation errors. The following pitfalls are derived from real-world operator maintenance experience.

1.  **Non-Idempotent Reconcile Loops:**
    *   *Mistake:* Modifying resources based on assumptions or performing actions that change state without checking current state first.
    *   *Impact:* Resource thrashing, excessive API server load, and inconsistent cluster state.
    *   *Fix:* Always `Get` the resource before `Update`. Compare the desired state with the retrieved state.

2.  **Blocking the Reconcile Loop:**
    *   *Mistake:* Performing long-running operations (e.g., waiting for a backup to complete, sleeping) inside `Reconcile`.
    *   *Impact:* The controller becomes unresponsive to other events. Other CRs are starved.
    *   *Fix:* Use `ctrl.Result{RequeueAfter: 5 * time.Minute}` for async tasks. Return immediately and let the loop re-trigger.

3.  **Ignoring RBAC Scopes:**
    *   *Mistake:* Granting the operator `cluster-admin` or wildcard permissions.
    *   *Impact:* Security vulnerabilities. If the operator is compromised, the attacker gains full cluster access.
    *   *Fix:* Use minimal RBAC. Grant permissions only for the specific resources the operator manages. Use `kubebuilder:rbac` markers to generate precise roles.

4.  **Missing Finalizers for Cleanup:**
    *   *Mistake:* Deleting the CR leaves orphaned resources (PVCs, external load balancers, cloud instances).
    *   *Impact:* Resource leaks, billing costs, and "zombie" infrastructure.
    *   *Fix:* Add a finalizer to the CR. When a delete timestamp is detected, execute cleanup logic, then remove the finalizer to allow garbage collection.

5.  **Coupling Operator Logic to Specific Versions:**
    *   *Mistake:* Hardcoding logic that only works for version 1.0 of the managed application.
    *   *Impact:* Operator breaks during upgrades or requires frequent operator releases.
    *   *Fix:* Design the CRD schema to be version-agnostic where possible. Implement upgrade logic that inspects the `spec.version` and applies migration steps dynamically.

6.  **Lack of Integration Testing:**
    *   *Mistake:* Testing only with `kubectl apply` in a live cluster.
    *   *Impact:* Flaky behavior in production. Race conditions are hard to reproduce manually.
    *   *Fix:* Use `envtest` from controller-runtime. This spins up a local etcd and API server for fast, deterministic unit and integration tests.

7.  **Status Blindness:**
    *   *Mistake:* The operator updates resources but never updates the CR status.
    *   *Impact:* Users cannot see the state of their application. `kubectl get mydb` shows no useful information.
    *   *Fix:* Implement a status writer. Update conditions and phases based on the health of child resources.

### Production Bundle

#### Action Checklist

- [ ] **Schema Validation:** Define strict `openAPIV3Schema` in the CRD to prevent invalid configurations from reaching the controller.
- [ ] **Finalizers:** Implement finalizers for all external resources or persistent data that requires cleanup.
- [ ] **RBAC Minimization:** Review RBAC markers. Ensure the operator only has permissions for resources it creates or manages.
- [ ] **Leader Election:** Enable leader election for HA deployments. This prevents multiple operator pods from reconciling simultaneously.
- [ ] **Metrics Integration:** Expose Prometheus metrics for reconciliation duration, error counts, and custom application metrics.
- [ ] **Integration Tests:** Write tests using `envtest` covering create, update, delete, and error scenarios.
- [ ] **Documentation:** Document the CRD spec, including all fields, defaults, and status conditions for end-users.

#### Decision Matrix

Use this matrix to determine if an Operator is the right tool for your workload.

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Stateless Microservice | Deployment + Helm | Operators add unnecessary complexity for stateless apps. Helm handles templating and upgrades sufficiently. | Low |
| Complex Stateful App (DB/Queue) | Kubernetes Operator | Requires automated backups, scaling, and self-healing. Operators encode this logic reliably. | Medium (Dev time) / Low (Ops time) |
| Multi-Cluster Management | Cluster API / Fleet Manager | Operators manage single-cluster scope. Multi-cluster requires federation or GitOps tools. | High |
| Legacy Migration | Operator + Sidecar | Wrap legacy binaries in containers and use an Operator to manage lifecycle if stateful logic is complex. | High |
| Configuration Management | GitOps (ArgoCD/Flux) | Operators are for runtime logic. GitOps is for declarative state synchronization. Use GitOps to deploy Operators. | Low |

#### Configuration Template

A production-ready CRD snippet with validation and subresources.

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mydatabases.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {} # Enables status subresource
        scale:
          specReplicasPath: .spec.size
          statusReplicasPath: .status.replicas
      additionalPrinterColumns:
        - name: Phase
          type: string
          description: Current phase
          jsonPath: .status.phase
        - name: Size
          type: integer
          description: Cluster size
          jsonPath: .spec.size
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["size", "version"]
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 50
                  description: Number of nodes in the cluster.
                version:
                  type: string
                  description: Application version.
                storage:
                  type: object
                  properties:
                    size:
                      type: string
                      pattern: '^\d+(Gi|Ti)$'
                    class:
                      type: string
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: ["Creating", "Running", "Scaling", "Failed"]
                replicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
                      reason:
                        type: string
  scope: Namespaced
  names:
    plural: mydatabases
    singular: mydatabase
    kind: MyDatabase
    shortNames:
      - mdb

Quick Start Guide

Get a basic operator running in under 5 minutes using the Operator SDK.

1. Initialize Project:

   operator-sdk init --domain example.com --repo github.com/myorg/myoperator
   

2. Create API:

   operator-sdk create api --group example --version v1 --kind MyDatabase --resource --controller
   

3. Edit Controller: Open internal/controller/mydatabase_controller.go. Implement the Reconcile logic to create a Deployment based on the CR spec. Add RBAC markers at the top of the file.

4. Run Locally:

   make install
   make run
   

   The operator runs locally, connecting to your active kubeconfig. This allows rapid iteration.

5. Deploy Sample CR:

   kubectl apply -f config/samples/example_v1_mydatabase.yaml
   

   Verify the operator creates the managed resources and updates the status.

Kubernetes Operators represent the maturation of cloud-native operations. By encoding domain knowledge into the control plane, teams achieve autonomy, reliability, and scalability that static manifests cannot provide. The initial investment in operator development yields compounding returns through reduced operational toil and increased system resilience.