9 Ways AI Coding Agents Break in Production (May 2026)

s.includes(schema)) { return { allowed: false, reason: Guardrail violation: Mutation on protected schema '${schema}'. }; } }

// 3. Deterministic Guardrail: Parameter Sanitization
if (action.tool === 'shell_exec') {
  const cmd = String(action.parameters['command']);
  if (cmd.includes('rm -rf /') || cmd.includes('DROP DATABASE')) {
    return { allowed: false, reason: 'Guardrail violation: Destructive shell command detected.' };
  }
}

return { allowed: true };

}

/**

• Executes a validated action, updating metrics and respecting dry-run mode. */ public async executeAction(action: ToolAction): Promise<void> { const validation = this.validateAction(action); if (!validation.allowed) { throw new Error(Execution blocked: ${validation.reason}); }

if (this.blastRadius.dryRun) {

  console.log(`[DRY RUN] Would execute: ${action.tool} with params`, action.parameters);
  return;
}

// Update metrics based on action type
if (action.tool === 'git_commit') this.currentMetrics.commits++;
if (action.tool === 'db_delete') {
  const rows = Number(action.parameters['row_count'] || 0);
  this.currentMetrics.dbRows += rows;
}

// Delegate to actual tool runner
await this.runTool(action);

}

private async runTool(action: ToolAction): Promise<void> { // Implementation of actual tool execution // In production, this would interface with the specific tool provider console.log(Executing ${action.tool}...); } }

// usage.ts // Example instantiation with production-safe configuration const orchestrator = new ResilientAgentOrchestrator({ maxCommits: 5, maxDatabaseRows: 10, dryRun: true, // Start with dry-run enabled for safety });

const context: AgentContext = { runtimeState: new Map([['DB_HOST', 'prod-db.internal']]), schemaConstraints: ['users', 'payments'], // Protected schemas authHeaders: { 'X-API-Key': 'injected-secret' }, };

// Agent proposes an action const proposedAction: ToolAction = { tool: 'db_delete', parameters: { table: 'logs', row_count: 50 }, context: context, };

// Orchestration enforces limits orchestrator.executeAction(proposedAction) .then(() => console.log('Action executed safely.')) .catch(err => console.error('Action blocked:', err.message));

### Rationale

*   **Blast Radius Configuration:** Hard limits on commits and database rows prevent runaway loops. The `maxCommits: 5` and `maxDatabaseRows: 10` defaults ensure that even if the agent enters a failure loop, the damage is contained.
*   **Deterministic Validation:** The `validateAction` method uses direct checks against schema constraints and command patterns. This avoids the latency and cost of LLM-based validation while providing stronger guarantees for known risk patterns.
*   **Dry-Run Mode:** The `dryRun` flag allows teams to test agent behavior without side effects. This is essential for validating scaffold reliability before enabling live execution.
*   **Context Injection:** The `AgentContext` structure forces explicit provision of runtime state. This prevents the agent from hallucinating environment variables or missing critical schema constraints.

## Pitfall Guide

### 1. The Benchmark Mirage
**Explanation:** Selecting models based solely on leaderboard scores (e.g., SmolLM3 3B at 93.3%) without validating scaffold compatibility. Benchmarks measure task completion on static data, not resilience to production variance.
**Fix:** Validate models on a live harness that includes hidden state and non-deterministic tool responses. Prioritize scaffold robustness over raw benchmark metrics.

### 2. Environmental Overtrust
**Explanation:** Agents treating files, logs, and API responses as authoritative without verification. A stale README or poisoned config file can lead to incorrect tool calls or deployment plans.
**Fix:** Implement source validation in the scaffold. Verify file freshness, checksum integrity, and API response schemas before injecting context into the agent's working memory.

### 3. Guardrail Latency Tax
**Explanation:** Using stacked LLM validators to check agent actions. Each LLM-on-LLM check adds significant round-trip latency and does not eliminate non-determinism.
**Fix:** Replace LLM validators with deterministic checks (regex, AST analysis, policy engines). Deterministic guards provide faster, more predictable validation for known risk patterns.

### 4. Hidden Runtime State
**Explanation:** Agents writing code that runs locally but fails in production due to missing environment variables, database schemas, or upstream headers. The agent lacks visibility into the live environment.
**Fix:** Use explicit state injection. The scaffold must query and inject all required runtime context (env vars, schemas, auth tokens) before the agent begins tool execution.

### 5. Unbounded Tool Execution
**Explanation:** Allowing agents to execute tools without transactional limits. This can lead to blast radius events, such as 30 erroneous commits or mass database deletions in a single run.
**Fix:** Enforce blast radius limits at the tool layer. Cap the number of commits, database rows affected, and API calls per session. Use dry-run modes for initial validation.

### 6. Non-Deterministic Trace Blindness
**Explanation:** Traditional observability tools fail to capture agentic workflows because identical prompts can produce different tool sequences. Traces branch through planning, memory retrieval, and retries.
**Fix:** Implement agentic tracing that propagates unique trace IDs across all tool calls and retries. Instrument the scaffold to log decision points, tool inputs/outputs, and validation results for full replayability.

### 7. Tool Rotation Burn
**Explanation:** Underestimating the cost of switching between AI coding tools. Retrospectives indicate rotation costs can reach hundreds of dollars per developer over 1.5 years due to retraining, workflow adaptation, and license fees.
**Fix:** Standardize on a scaffold architecture that abstracts the underlying model. This allows model swapping without rewriting integration logic. Budget for rotation costs and evaluate total cost of ownership, not just per-token pricing.

## Production Bundle

### Action Checklist

- [ ] **Inject Runtime State:** Ensure all environment variables, database schemas, and auth headers are explicitly injected into the agent context before execution.
- [ ] **Set Blast Radius Limits:** Configure hard limits on commits, database rows, and API calls per session. Start with conservative defaults.
- [ ] **Implement Deterministic Guardrails:** Replace LLM validators with regex, AST, and policy-based checks for tool validation.
- [ ] **Enable Dry-Run Mode:** Run all agent workflows in dry-run mode initially to validate scaffold behavior without side effects.
- [ ] **Instrument Agentic Tracing:** Deploy tracing that captures tool sequences, validation results, and decision points for full observability.
- [ ] **Validate on Live Harness:** Test models against a production-like harness that includes hidden state and non-deterministic tool responses.
- [ ] **Budget for Rotation:** Account for tool rotation costs in long-term planning. Abstract model dependencies to minimize switching friction.

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
| :--- | :--- | :--- | :--- |
| **Solo Developer** | Use cost-effective model + strict scaffold | Scaffold reliability matters more than model size for side projects. | Low |
| **Team (5-20 devs)** | Standardize scaffold + budget rotation | Consistency and reduced friction outweigh minor benchmark gains. | Medium |
| **Latency-Critical App** | Deterministic guardrails only | LLM validators introduce unacceptable latency; deterministic checks are faster. | Low |
| **Production Data Access** | Blast radius caps + dry-run validation | Safety is paramount; limits prevent catastrophic data loss. | N/A |
| **Cost-Sensitive Batch** | Small open models (e.g., SmolLM3, Qwen) | Benchmarks show small models can compete; validate on live harness first. | Low |

### Configuration Template

```yaml
# agent-scaffold-config.yaml
orchestrator:
  blast_radius:
    max_commits: 5
    max_database_rows: 10
    dry_run: true
  guardrails:
    type: deterministic
    rules:
      - pattern: "rm -rf /"
        action: block
      - pattern: "DROP DATABASE"
        action: block
  state_injection:
    sources:
      - type: env_var
        prefix: "APP_"
      - type: db_schema
        tables: ["users", "orders"]
      - type: auth_header
        key: "X-API-Key"
  observability:
    tracing:
      enabled: true
      propagate_trace_id: true
    logging:
      level: debug
      include_tool_io: true

Quick Start Guide

1. Define State Schema: Identify all runtime dependencies (env vars, schemas, headers) required by your application. Configure the scaffold to inject these explicitly.
2. Set Safety Limits: Initialize the orchestrator with conservative blast radius limits. Enable dry-run mode to test behavior safely.
3. Add Deterministic Checks: Implement guardrails for known risk patterns (destructive commands, protected schemas). Avoid LLM-based validation for performance.
4. Run Validation: Execute a test workflow in dry-run mode. Verify that state injection works, guardrails trigger correctly, and blast radius limits are enforced.
5. Deploy with Tracing: Enable agentic tracing and deploy to a staging environment. Monitor tool sequences and validation results to ensure scaffold reliability before production rollout.