Kubernetes Networking: The Hidden Complexity Behind Service Abstractions and Traffic Flow Management

matchLabels: app: api-backend policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: app: api-gateway ports: - port: 8080 protocol: TCP egress: - to: - podSelector: matchLabels: app: postgres ports: - port: 5432 protocol: TCP - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system podSelector: matchLabels: k8s-app: kube-dns ports: - port: 53 protocol: UDP

### Step 5: DNS and Service Discovery Tuning
CoreDNS must be tuned for high query volumes. Default configurations cache aggressively but lack upstream timeout handling. Adjust `forward`, `cache`, and `loop` plugins to prevent resolution stalls.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
            policy sequential
        }
        cache 30
        loop
        reload
        loadbalance
    }

Architecture Decisions and Rationale

• eBPF over iptables: Eliminates O(N) rule scanning, reduces CPU overhead by ~70%, and enables L7 policy enforcement without sidecars.
• Default-deny NetworkPolicy: Enforces zero-trust networking at the cluster level. Additive policies prevent accidental exposure.
• Ingress decoupling: Separates routing, TLS termination, and rate limiting from service logic, enabling independent scaling and security patching.
• CoreDNS tuning: Prevents resolution bottlenecks during scaling events. Sequential forwarding and connection limits avoid upstream DNS overload.

Pitfall Guide

1. Assuming default CNI covers security requirements Most default CNIs (Flannel, Calico in BGP mode) lack L7 visibility and enforce policies at the iptables layer. Without explicit NetworkPolicies, all pod-to-pod traffic flows unrestricted. Best practice: Deploy a CNI with eBPF enforcement and apply default-deny policies immediately after cluster bootstrap.

2. Ignoring conntrack table exhaustion Linux connection tracking maintains state for every TCP/UDP flow. The default nf_conntrack_max is often 65,536, which exhausts under high connection rates, causing silent packet drops. Best practice: Tune net.netfilter.nf_conntrack_max to 1,048,576+ on nodes, monitor /proc/net/nf_conntrack_count, and use eBPF to bypass conntrack for pod-to-pod traffic.

3. Overusing NodePort and LoadBalancer internally NodePort exposes services on every node IP, breaking network segmentation and causing IP conflicts in multi-cloud environments. LoadBalancer creates external cloud resources for internal traffic, increasing cost and latency. Best practice: Use ClusterIP for internal communication, Ingress for north-south routing, and cloud load balancers only for public endpoints.

4. Treating DNS as infinite and stateless CoreDNS query rates scale with pod count and service discovery patterns. Unbounded caching, missing TTL controls, and recursive search paths (search: default.svc.cluster.local svc.cluster.local cluster.local) cause resolution delays and stale endpoints. Best practice: Limit search paths, set explicit TTLs, monitor coredns_dns_request_duration_seconds, and use headless services for stateful workloads requiring direct Pod IP resolution.

5. Misunderstanding Service IP vs Pod IP routing Service IPs are virtual and never assigned to network interfaces. Routing directly to a Service IP from outside the cluster fails because kube-proxy/eBPF only intercepts traffic within the cluster CIDR. Best practice: Always route through Ingress or cloud load balancers for external traffic. Use kubectl get endpoints to verify backend health before debugging routing.

6. Ignoring east-west vs north-south traffic patterns East-west traffic (pod-to-pod) requires low latency and high throughput. North-south traffic (client-to-cluster) requires TLS termination, rate limiting, and WAF capabilities. Applying the same routing logic to both causes asymmetric routing, certificate mismatch errors, and performance degradation. Best practice: Use eBPF for east-west optimization and dedicated Ingress controllers for north-south termination.

7. Failing to validate traffic flows before deployment NetworkPolicies and routing rules are declarative but not self-validating. Deploying without connectivity testing leads to silent failures. Best practice: Use kubectl run debug --image=nicolaka/netshoot -it --rm -- bash to simulate traffic, verify DNS resolution, test TCP handshakes, and validate policy enforcement with cilium policy trace.

Production Bundle

Action Checklist

• Verify CNI installation and eBPF mode: kubectl get pods -n kube-system -l app.kubernetes.io/name=cilium
• Apply default-deny NetworkPolicy to all namespaces: kubectl apply -f default-deny.yaml
• Tune CoreDNS ConfigMap for query volume and upstream timeout handling
• Configure Ingress controller with TLS termination and path-based routing
• Set conntrack limits and monitor table utilization: sysctl net.netfilter.nf_conntrack_max
• Validate east-west connectivity with synthetic traffic tests
• Enable Hubble or equivalent observability for L3/L4/L7 flow visibility
• Document traffic boundaries and update runbooks for routing failures

Decision Matrix

Configuration Template

# default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# cilium-values.yaml (production baseline)
kubeProxyReplacement: true
bpf:
  masquerade: true
  tproxy: true
  conntrack:
    enabled: true
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
ipam:
  mode: kubernetes
operator:
  replicas: 2
---
# coredns-tuned.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health { lameduck 5s }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
            policy sequential
        }
        cache 30
        loop
        reload
        loadbalance
    }

Quick Start Guide

1. Install Cilium with eBPF and Hubble

   helm repo add cilium https://helm.cilium.io/
   helm install cilium cilium/cilium --version 1.14.0 \
     --set kubeProxyReplacement=true \
     --set hubble.enabled=true \
     --set hubble.relay.enabled=true \
     --set hubble.ui.enabled=true \
     -n kube-system
   

2. Apply default-deny NetworkPolicy

   kubectl apply -f - <<EOF
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: default-deny
     namespace: default
   spec:
     podSelector: {}
     policyTypes: [Ingress, Egress]
   EOF
   

3. Deploy test workload and service

   kubectl create deployment nginx --image=nginx:latest --replicas=2
   kubectl expose deployment nginx --port=80 --target-port=80 --name=nginx-svc
   

4. Validate connectivity and policy enforcement

   kubectl run curl --image=nicolaka/netshoot -it --rm --restart=Never -- curl -s http://nginx-svc.default.svc.cluster.local
   # Expected: HTML response
   # Verify policy: kubectl get networkpolicy
   # Check flow: cilium status && hubble observe
   

5. Tune node networking parameters

   sysctl -w net.netfilter.nf_conntrack_max=1048576
   sysctl -w net.ipv4.tcp_tw_reuse=1
   echo "net.netfilter.nf_conntrack_max=1048576" >> /etc/sysctl.conf
   

Kubernetes networking is not a configuration task; it is an architecture decision. Treat traffic flow as a first-class design constraint, enforce policies at the kernel layer, and validate routing before scaling. The overhead of upfront networking design pays exponential dividends in stability, observability, and incident response time.