es execution to the Agent Development Kit, which manages conversation state, tool routing, and provider communication. 2. State Storage: Conversation history and agent metadata persist in PostgreSQL. The default installation bundles a lightweight instance; production deployments should externalize this to a managed database or external PostgreSQL cluster. 3. Tool Server: An MCP-compliant tool server exposes Kubernetes operations as callable functions. Agents invoke these tools through structured requests, eliminating the need for custom API wrappers. 4. Provider Decoupling: The Agent resource references a ModelConfig by name. The controller resolves the provider endpoint at runtime, making cloud and local inference interchangeable without agent redefinition.

Step 1: Controller Installation

The control plane installs via two Helm charts: one for CRD registration, one for the runtime components. The tool server bundles automatically in recent releases.

helm install ai-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --namespace ai-orchestrator \
  --create-namespace

helm install ai-runtime oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --namespace ai-orchestrator \
  --set providers.default=anthropic \
  --set providers.anthropic.apiKey="${INFRA_AI_KEY}"

Verify controller readiness:

kubectl get pods -n ai-orchestrator -l app.kubernetes.io/name=kagent

Step 2: Cloud Inference Configuration

Cloud providers require credential isolation. Store the API key in a Kubernetes Secret, then define a ModelConfig that references it. The controller reads the secret at runtime and never persists credentials in etcd.

apiVersion: v1
kind: Secret
metadata:
  name: cloud-inference-creds
  namespace: ai-orchestrator
type: Opaque
stringData:
  PROVIDER_TOKEN: "${INFRA_AI_KEY}"
---
apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: cloud-sonnet-v4
  namespace: ai-orchestrator
spec:
  provider: Anthropic
  model: claude-sonnet-4-5
  apiKeySecret: cloud-inference-creds
  apiKeySecretKey: PROVIDER_TOKEN
  anthropic: {}

Apply the manifest:

kubectl apply -f cloud-model.yaml

Step 3: Local Inference Runtime

On-premises inference requires persistent model storage and stable service discovery. Ollama serves as the runtime, but the deployment must account for model caching and resource allocation.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-model-cache
  namespace: ai-orchestrator
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 25Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: local-inference-engine
  namespace: ai-orchestrator
spec:
  replicas: 1
  selector:
    matchLabels:
      component: inference
  template:
    metadata:
      labels:
        component: inference
    spec:
      containers:
        - name: runtime
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
              protocol: TCP
          resources:
            requests:
              cpu: "2"
              memory: 10Gi
            limits:
              memory: 14Gi
          volumeMounts:
            - name: cache
              mountPath: /root/.ollama
      volumes:
        - name: cache
          persistentVolumeClaim:
            claimName: local-model-cache
---
apiVersion: v1
kind: Service
metadata:
  name: local-inference-svc
  namespace: ai-orchestrator
spec:
  selector:
    component: inference
  ports:
    - port: 80
      targetPort: 11434
      protocol: TCP

Deploy and cache the target model:

kubectl apply -f local-runtime.yaml
kubectl wait --for=condition=ready pod -l component=inference -n ai-orchestrator --timeout=120s
kubectl exec -n ai-orchestrator deploy/local-inference-engine -- ollama pull llama3.1

Register the local provider. Ollama does not require authentication, but the CRD schema mandates a secret reference. Create a placeholder:

apiVersion: v1
kind: Secret
metadata:
  name: local-inference-creds
  namespace: ai-orchestrator
type: Opaque
stringData:
  DUMMY_KEY: "placeholder"
---
apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: local-llama-v31
  namespace: ai-orchestrator
spec:
  provider: Ollama
  model: llama3.1
  apiKeySecret: local-inference-creds
  apiKeySecretKey: DUMMY_KEY
  ollama:
    host: http://local-inference-svc.ai-orchestrator.svc.cluster.local

Step 4: Agent Definition & Tool Wiring

Agents declare behavior through system prompts, tool references, and model bindings. The built-in MCP tool server exposes Kubernetes operations without custom code.

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: ops-assistant
  namespace: ai-orchestrator
spec:
  description: Infrastructure diagnostic agent with provider-agnostic routing
  type: Declarative
  declarative:
    modelConfig: cloud-sonnet-v4
    systemMessage: |
      You are an infrastructure diagnostic assistant.
      - Query cluster state using available tools before drawing conclusions.
      - Request clarification when resource names or namespaces are ambiguous.
      - Format all outputs as structured Markdown with execution summaries.
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          apiGroup: kagent.dev
          toolNames:
            - k8s_get_resources
            - k8s_describe_resource
            - k8s_get_available_api_resources

Switching providers requires only updating the modelConfig field to local-llama-v31. The controller resolves the new endpoint on the next reconciliation cycle. No agent redeployment or code changes are necessary.

Pitfall Guide

1. Function Calling Incompatibility

Explanation: Ollama serves many models, but only those trained with tool-use capabilities support structured function calling. Models like llama3 lack this training and will fail to parse MCP tool requests. Fix: Verify tool support before deployment. Use llama3.1, qwen2.5, or newer variants. Validate with ollama run <model> --verbose and inspect tool-call parsing logs.

2. Secret Key Reference Mismatch

Explanation: The apiKeySecretKey field must exactly match the key name in the referenced Secret. Typos or case mismatches cause silent authentication failures. Fix: Use kubectl get secret <name> -o jsonpath='{.data}' to verify key names. Add a validation webhook or CI lint step to catch mismatches before apply.

3. PVC Storage Class Incompatibility

Explanation: Local model caching requires persistent storage. If the cluster lacks a default StorageClass or uses a read-only provisioner, the Ollama pod enters Pending or CrashLoopBackOff. Fix: Explicitly set storageClassName in the PVC manifest. Verify provisioner readiness with kubectl get sc. For ephemeral testing, use emptyDir but expect model re-downloads on rescheduling.

4. Unrestricted Kubernetes Tool Access

Explanation: The MCP tool server inherits the controller's RBAC permissions. If the controller runs with cluster-admin privileges, agents can execute destructive operations across all namespaces. Fix: Apply least-privilege RBAC to the controller ServiceAccount. Restrict tool access via namespace-scoped roles and audit policies. Implement human approval gates for write operations using admission webhooks.

5. Model Alias Volatility

Explanation: Provider aliases like claude-sonnet-4-5 resolve to the latest dated snapshot. Upstream updates can alter model behavior, breaking deterministic agent responses. Fix: Pin to dated versions (e.g., claude-sonnet-4-5-20251001) in production. Use aliases only for development. Maintain a version registry and test agent prompts against new snapshots before promotion.

6. State Backend Bottlenecks

Explanation: The bundled PostgreSQL instance handles conversation state and agent metadata. Under concurrent load, connection pooling limits and disk I/O degrade response latency. Fix: Externalize state storage to a managed PostgreSQL cluster or cloud database. Configure connection pooling via kagent Helm values. Monitor query latency and implement read replicas for high-throughput environments.

7. Network Policy Isolation Failures

Explanation: Ollama and the tool server communicate over cluster networking. Default policies may block intra-namespace traffic or expose inference endpoints to untrusted workloads. Fix: Apply namespace-scoped NetworkPolicies. Restrict Ollama ingress to the ai-orchestrator namespace only. Egress rules should limit cloud API calls to provider endpoints. Validate with kubectl exec connectivity tests.

Production Bundle

Action Checklist

• Verify StorageClass availability and PVC provisioning before deploying local inference runtimes
• Pin model versions to dated snapshots for production agent definitions
• Apply least-privilege RBAC to the controller ServiceAccount and audit tool execution logs
• Externalize PostgreSQL state storage and configure connection pooling for concurrent workloads
• Implement NetworkPolicies to restrict inference endpoint access to authorized namespaces
• Validate function calling support for all local models before registering ModelConfig resources
• Establish CI/CD validation for Secret key references and CRD schema compliance
• Configure human approval gates or admission webhooks for destructive Kubernetes tool operations

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Sensitive data processing	Local Ollama runtime	Zero egress, full data sovereignty	Higher infrastructure cost, predictable
Burst compute demand	Cloud Anthropic API	Elastic scaling, no GPU provisioning	Variable token costs, egress fees
Multi-tenant isolation	Namespace-scoped ModelConfig + RBAC	Prevents cross-tenant tool access	Minimal, leverages existing K8s controls
Deterministic compliance	Pdated model versions + external Postgres	Reproducible responses, auditable state	Moderate storage cost, operational overhead

Configuration Template

# ai-orchestrator-stack.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ai-orchestrator
---
apiVersion: v1
kind: Secret
metadata:
  name: prod-inference-creds
  namespace: ai-orchestrator
type: Opaque
stringData:
  CLOUD_TOKEN: "${PROD_INFRA_KEY}"
---
apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: prod-cloud-model
  namespace: ai-orchestrator
spec:
  provider: Anthropic
  model: claude-sonnet-4-5-20251001
  apiKeySecret: prod-inference-creds
  apiKeySecretKey: CLOUD_TOKEN
  anthropic: {}
---
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: prod-ops-agent
  namespace: ai-orchestrator
spec:
  description: Production infrastructure assistant with strict safety boundaries
  type: Declarative
  declarative:
    modelConfig: prod-cloud-model
    systemMessage: |
      You are a production infrastructure assistant.
      - Never modify resources without explicit confirmation.
      - Always verify namespace and resource existence before querying.
      - Output structured Markdown with execution trace and risk assessment.
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          apiGroup: kagent.dev
          toolNames:
            - k8s_get_resources
            - k8s_describe_resource

Quick Start Guide

1. Initialize Control Plane: Run the two Helm install commands to register CRDs and deploy the controller. Verify pod readiness with kubectl get pods -n ai-orchestrator.
2. Configure Provider: Create a Secret containing your cloud API key, then apply a ModelConfig manifest referencing it. Validate registration with kubectl get modelconfigs -n ai-orchestrator.
3. Deploy Local Runtime: Apply the Ollama Deployment, PVC, and Service manifests. Pull a function-calling-capable model using ollama pull. Register a second ModelConfig pointing to the internal Service.
4. Launch Agent: Apply the Agent CRD with your system prompt and tool references. Test execution via the kagent UI or CLI. Switch providers by updating the modelConfig field and reapplying.