Kubernetes Security Misconceptions and Architectural Solutions for Cloud-Native Infrastructure

lize Secrets Native Kubernetes secrets are base64-encoded and stored in etcd. Without encryption at rest, they are plaintext to anyone with etcd access. Use the External Secrets Operator (ESO) to sync credentials from cloud KMS, HashiCorp Vault, or AWS Secrets Manager.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-role"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: production/db
      property: username
  - secretKey: password
    remoteRef:
      key: production/db
      property: password

Step 5: Verify Images Before Admission

Vulnerability scanning alone is insufficient. Supply chain attacks bypass CVE databases by injecting malicious layers into trusted images. Use Sigstore/Cosign to verify image signatures at the admission layer. OPA/Gatekeeper or Kyverno can enforce signature checks.

// policy-validation.ts
// TypeScript webhook client for validating image signatures via Cosign CLI
import { execSync } from 'child_process';
import type { AdmissionReview, AdmissionRequest } from 'kubernetes-client-types';

export async function validateImageSignature(admissionReview: AdmissionReview): Promise<AdmissionReview> {
  const request = admissionReview.request as AdmissionRequest;
  const containers = request.object.spec.containers;
  const allowed = [];

  for (const container of containers) {
    try {
      execSync(`cosign verify --key env://COSIGN_PUBLIC_KEY ${container.image}`, { stdio: 'pipe' });
      allowed.push(container.name);
    } catch {
      return {
        apiVersion: 'admission.k8s.io/v1',
        kind: 'AdmissionReview',
        response: {
          uid: request.uid,
          allowed: false,
          status: {
            message: `Image ${container.image} failed signature verification`,
            code: 403
          }
        }
      };
    }
  }

  return {
    apiVersion: 'admission.k8s.io/v1',
    kind: 'AdmissionReview',
    response: {
      uid: request.uid,
      allowed: true,
      patchType: 'JSONPatch',
      patch: Buffer.from(JSON.stringify([
        { op: 'add', path: '/metadata/annotations/verified-by', value: allowed.join(',') }
      ])).toString('base64')
    }
  };
}

Architecture Decisions & Rationale

• PSA over PSP: PSP was deprecated in 1.21 and removed in 1.25 due to security flaws and maintenance burden. PSA is built into the kube-apiserver, requires no external controllers, and aligns with Kubernetes' declarative model.
• ESO over native secrets: Native secrets lack rotation, audit trails, and KMS integration. ESO decouples secret lifecycle from cluster state, enabling automatic rotation and centralized policy enforcement.
• Cosign over static scanning: CVE databases are reactive. Signature verification establishes cryptographic provenance, ensuring only approved build pipelines deploy to production.
• eBPF for runtime: Traditional runtime security relies on cgroups and syscalls, which are easily bypassed. eBPF attaches to kernel hooks, providing low-overhead, tamper-resistant monitoring for process execution, network activity, and filesystem changes.

Pitfall Guide

1. Granting cluster-admin to developers or CI/CD pipelines Cluster-admin bypasses namespace boundaries and grants full control over etcd, RBAC, and admission controllers. Use namespace-scoped roles and delegate cluster provisioning to a dedicated platform team.

2. Assuming NetworkPolicies default to deny Kubernetes permits all traffic unless explicitly restricted. Deploy a default-deny policy per namespace before adding allow rules. Missing this step leaves lateral movement paths open.

3. Storing secrets in Git or Kubernetes Secrets without encryption at rest Base64 encoding is not encryption. Enable etcd encryption providers and route sensitive data through external secret managers. Git history retains plaintext secrets indefinitely.

4. Relying solely on vulnerability scanning Scanners detect known CVEs but miss zero-days, misconfigurations, and supply chain compromises. Pair scanning with image signing, runtime monitoring, and behavioral baselines.

5. Misconfiguring PSA modes (enforce vs audit vs warn) Enforce blocks non-compliant pods. Audit logs violations. Warn displays warnings to users. Mixing modes across namespaces creates inconsistent security postures. Standardize on enforce for production, audit for staging.

6. Leaving service account tokens auto-mounted Every pod receives a JWT token by default. Attackers use these tokens to query the Kubernetes API and escalate privileges. Set automountServiceAccountToken: false and mount only when API access is required.

7. Treating compliance as security Passing a CIS benchmark scan does not guarantee runtime safety. Compliance checks are static snapshots. Security requires continuous policy enforcement, drift detection, and anomaly response.

Best Practices from Production

• Treat security manifests as code: version control, review, and apply via GitOps.
• Use policy engines (Kyverno, OPA) for custom rules that PSA/RBAC cannot express.
• Implement runtime eBPF agents (Falco, Tetragon) with alert routing to SIEM.
• Rotate credentials automatically and invalidate tokens on pod termination.
• Validate network policies with tools like kubectl-np-viewer or Cilium's policy visualization.

Production Bundle

Action Checklist

• Enable Pod Security Admission at namespace level with restricted baseline
• Disable auto-mounted service account tokens cluster-wide
• Deploy default-deny NetworkPolicy in every production namespace
• Migrate secrets to External Secrets Operator with KMS backing
• Enforce image signature verification at admission layer
• Deploy eBPF runtime agent with process and network monitoring
• Integrate policy validation into CI/CD pipeline before apply
• Schedule quarterly RBAC and NetworkPolicy audits

Decision Matrix

Configuration Template

# security-baseline.yaml
# Apply to production namespace to enforce zero-trust baseline
apiVersion: v1
kind: Namespace
metadata:
  name: prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: prod
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deny-cluster-admin-to-devs
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: workload-sa
  namespace: prod
automountServiceAccountToken: false

Quick Start Guide

1. Apply namespace security labels: Run kubectl label namespace <ns> pod-security.kubernetes.io/enforce=restricted --overwrite to activate PSA enforcement.
2. Deploy default-deny policy: Apply the NetworkPolicy manifest above to block all pod-to-pod traffic until explicitly allowed.
3. Install External Secrets Operator: helm install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets --namespace external-secrets --create-namespace. Configure your cloud KMS provider in the SecretStore CRD.
4. Verify enforcement: Deploy a test pod with securityContext.privileged: true. The API server will reject it with a PSA violation. Check audit logs with kubectl get events -n <ns> to confirm policy interception.