Check eBPF support and map sizes

pf lb list

**4. Implement Zero Trust Network Policies**
Create a default-deny policy for the namespace, followed by specific allow rules.

```yaml
# default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# allow-frontend-to-backend.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

5. Enable L7 Policy Enforcement (eBPF Specific) Leverage eBPF to enforce HTTP-level policies, restricting access based on path and method.

# l7-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l7-frontend-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/api/v1/data"

Pitfall Guide

Production networking failures often stem from subtle misconfigurations or misunderstandings of the underlying Linux primitives.

1. MTU Mismatch and Fragmentation

   • Issue: Overlay networks add headers (VXLAN adds 50 bytes). If the physical MTU is 1500 and pod MTU remains 1500, packets fragment or drop, causing TCP performance degradation.
   • Resolution: Calculate effective MTU: Physical MTU - Encapsulation Header. Configure CNI to set pod MTU accordingly. For VXLAN on 1500 MTU, set pod MTU to 1450. Verify with ping -M do -s 1472 <target>.

2. IPAM Exhaustion

   • Issue: Default pod CIDR sizes (e.g., /24) limit nodes to 254 pods. In high-density clusters, this causes scheduling failures.
   • Resolution: Align Pod CIDR size with node count and expected pod density. Use /16 for large clusters or configure IPAM to allocate smaller blocks per node (e.g., /27) to maximize address utilization.

3. Conntrack Table Saturation

   • Issue: In iptables or IPVS modes, high connection rates fill the nf_conntrack table, causing nf_conntrack: table full, dropping packet errors.
   • Resolution: Increase net.netfilter.nf_conntrack_max. However, the superior fix is migrating to eBPF, which bypasses conntrack for service routing, eliminating this bottleneck.

4. DNS Resolution Failures

   • Issue: Pods cannot resolve internal services due to CoreDNS resource limits, incorrect search domains, or network policies blocking UDP port 53.
   • Resolution: Ensure NetworkPolicies allow egress to CoreDNS. Tune CoreDNS resources (resources.limits.memory). Verify ndots configuration; high ndots values cause excessive DNS queries and latency.

5. Service IP Collision

   • Issue: Kubernetes Service CIDR overlaps with an external network reachable via the node, causing routing loops or unreachable services.
   • Resolution: Audit external routing tables. Ensure service-cluster-ip-range is disjoint from all external subnets. Use ip route to verify no overlaps exist on worker nodes.

6. Assumption of Policy Enforcement

   • Issue: Applying NetworkPolicy resources without a CNI that supports them results in no enforcement. Flannel, for example, does not enforce policies.
   • Resolution: Verify CNI capabilities. Use kubectl get networkpolicy and test connectivity. Ensure the CNI is actively watching and translating policies into iptables/eBPF rules.

7. NodePort vs. LoadBalancer Confusion

   • Issue: Exposing services via NodePort without an external load balancer exposes services to the public internet if security groups are misconfigured.
   • Resolution: Use Ingress controllers or Cloud LoadBalancers for external traffic. Restrict NodePort access via cloud provider security groups/firewalls to trusted sources only.

Production Bundle

Action Checklist

• Audit CNI Capabilities: Verify the CNI supports NetworkPolicy, eBPF, and required routing modes.
• Enforce Default Deny: Apply default-deny NetworkPolicies to all production namespaces immediately.
• Validate MTU Configuration: Run MTU discovery scripts on all nodes and configure CNI to match physical constraints.
• Migrate to eBPF: For clusters with >500 services, switch to eBPF data plane to eliminate conntrack and iptables overhead.
• Configure IPAM Sizing: Review Pod CIDR allocation and adjust to prevent exhaustion based on growth projections.
• Deploy Observability: Install Hubble or equivalent flow visibility tool to map traffic patterns and debug policies.
• Test DNS Resilience: Simulate CoreDNS pod failures and verify service discovery recovery.
• Review External Routing: Ensure Service CIDR does not overlap with external networks and verify SNAT configuration.

Decision Matrix

Configuration Template

Cilium Helm Values for Production Hardening:

# production-cilium-values.yaml
kubeProxyReplacement: true
k8sServiceHost: <control-plane-ip>
k8sServicePort: 6443

bpf:
  masquerade: true
  # Optimize for high throughput
  lb:
    algorithm: maglev
  # Disable conntrack for service routing where possible
  conntrack:
    enabled: false

# Security: Enable strict default deny
policyEnforceMode: "always"

# Observability
hubble:
  enabled: true
  relay:
    enabled: true
    replicas: 2
  ui:
    enabled: true
  metrics:
    enabled:
    - dns
    - drop
    - tcp
    - flow
    - port-distribution
    - icmp
    - http

# Resource Limits
resources:
  limits:
    cpu: "1"
    memory: "1Gi"
  requests:
    cpu: "200m"
    memory: "256Mi"

# IPAM Configuration
ipam:
  mode: "kubernetes"

Quick Start Guide

1. Install Cilium CLI:

   curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
   sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
   sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
   rm cilium-linux-amd64.tar.gz{,.sha256sum}
   

2. Deploy Cilium:

   cilium install --version v1.14.0
   

3. Verify Installation:

   cilium status --wait
   # Expected output: All pods running, kube-proxy replaced.
   

4. Apply Default Deny:

   kubectl apply -f - <<EOF
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: default-deny
     namespace: default
   spec:
     podSelector: {}
     policyTypes:
     - Ingress
     - Egress
   EOF
   

5. Test Connectivity:

   cilium connectivity test
   # Verifies pod-to-pod, node-to-pod, and policy enforcement.