9 Ways AI Coding Agents Break in Production (May 2026)

lls must use deterministic checks (regex, AST analysis, policy engines) rather than secondary LLM calls. Stacking LLM validators introduces unacceptable latency and does not eliminate non-determinism. 3. Transactional Tool Bounds: Every tool invocation must be wrapped in a transactional context with hard limits on side effects. This prevents loop-induced blast radius expansion.

Implementation: Resilient Agent Orchestrator

The following TypeScript implementation demonstrates a scaffold that enforces these principles. It defines an orchestrator that manages state injection, validates actions deterministically, and caps execution impact.

// types.ts
export interface ToolAction {
  tool: string;
  parameters: Record<string, unknown>;
  context: AgentContext;
}

export interface ValidationResult {
  allowed: boolean;
  reason?: string;
}

export interface BlastRadiusConfig {
  maxCommits: number;
  maxDatabaseRows: number;
  dryRun: boolean;
}

export interface AgentContext {
  runtimeState: Map<string, string>;
  schemaConstraints: string[];
  authHeaders: Record<string, string>;
}

// orchestrator.ts
import { ToolAction, ValidationResult, BlastRadiusConfig, AgentContext } from './types';

export class ResilientAgentOrchestrator {
  private blastRadius: BlastRadiusConfig;
  private currentMetrics: { commits: number; dbRows: number };

  constructor(config: BlastRadiusConfig) {
    this.blastRadius = config;
    this.currentMetrics = { commits: 0, dbRows: 0 };
  }

  /**
   * Validates a tool action against deterministic guardrails and blast radius limits.
   * Returns a ValidationResult indicating if the action is safe to execute.
   */
  public validateAction(action: ToolAction): ValidationResult {
    // 1. Check Blast Radius Limits
    if (action.tool === 'git_commit' && this.currentMetrics.commits >= this.blastRadius.maxCommits) {
      return { allowed: false, reason: 'Blast radius exceeded: Max commits reached.' };
    }
    if (action.tool === 'db_delete' && this.currentMetrics.dbRows >= this.blastRadius.maxDatabaseRows) {
      return { allowed: false, reason: 'Blast radius exceeded: Max DB rows deletion reached.' };
    }

    // 2. Deterministic Guardrail: Schema Constraint Check
    if (action.tool === 'db_query' || action.tool === 'db_mutate') {
      const schema = action.parameters['table'] as string;
      if (action.context.schemaConstraints.includes(schema)) {
        return { allowed: false, reason: `Guardrail violation: Mutation on protected schema '${schema}'.` };
      }
    }

    // 3. Deterministic Guardrail: Parameter Sanitization
    if (action.tool === 'shell_exec') {
      const cmd = String(action.parameters['command']);
      if (cmd.includes('rm -rf /') || cmd.includes('DROP DATABASE')) {
        return { allowed: false, reason: 'Guardrail violation: Destructive shell command detected.' };
      }
    }

    return { allowed: true };
  }

  /**
   * Executes a validated action, updating metrics and respecting dry-run mode.
   */
  public async executeAction(action: ToolAction): Promise<void> {
    const validation = this.validateAction(action);
    if (!validation.allowed) {
      throw new Error(`Execution blocked: ${validation.reason}`);
    }

    if (this.blastRadius.dryRun) {
      console.log(`[DRY RUN] Would execute: ${action.tool} with params`, action.parameters);
      return;
    }

    // Update metrics based on action type
    if (action.tool === 'git_commit') this.currentMetrics.commits++;
    if (action.tool === 'db_delete') {
      const rows = Number(action.parameters['row_count'] || 0);
      this.currentMetrics.dbRows += rows;
    }

    // Delegate to actual tool runner
    await this.runTool(action);
  }

  private async runTool(action: ToolAction): Promise<void> {
    // Implementation of actual tool execution
    // In production, this would interface with the specific tool provider
    console.log(`Executing ${action.tool}...`);
  }
}

// usage.ts
// Example instantiation with production-safe configuration
const orchestrator = new ResilientAgentOrchestrator({
  maxCommits: 5,
  maxDatabaseRows: 10,
  dryRun: true, // Start with dry-run enabled for safety
});

const context: AgentContext = {
  runtimeState: new Map([['DB_HOST', 'prod-db.internal']]),
  schemaConstraints: ['users', 'payments'], // Protected schemas
  authHeaders: { 'X-API-Key': 'injected-secret' },
};

// Agent proposes an action
const proposedAction: ToolAction = {
  tool: 'db_delete',
  parameters: { table: 'logs', row_count: 50 },
  context: context,
};

// Orchestration enforces limits
orchestrator.executeAction(proposedAction)
  .then(() => console.log('Action executed safely.'))
  .catch(err => console.error('Action blocked:', err.message));

Rationale

• Blast Radius Configuration: Hard limits on commits and database rows prevent runaway loops. The maxCommits: 5 and maxDatabaseRows: 10 defaults ensure that even if the agent enters a failure loop, the damage is contained.
• Deterministic Validation: The validateAction method uses direct checks against schema constraints and command patterns. This avoids the latency and cost of LLM-based validation while providing stronger guarantees for known risk patterns.
• Dry-Run Mode: The dryRun flag allows teams to test agent behavior without side effects. This is essential for validating scaffold reliability before enabling live execution.
• Context Injection: The AgentContext structure forces explicit provision of runtime state. This prevents the agent from hallucinating environment variables or missing critical schema constraints.

Pitfall Guide

1. The Benchmark Mirage

Explanation: Selecting models based solely on leaderboard scores (e.g., SmolLM3 3B at 93.3%) without validating scaffold compatibility. Benchmarks measure task completion on static data, not resilience to production variance. Fix: Validate models on a live harness that includes hidden state and non-deterministic tool responses. Prioritize scaffold robustness over raw benchmark metrics.

2. Environmental Overtrust

Explanation: Agents treating files, logs, and API responses as authoritative without verification. A stale README or poisoned config file can lead to incorrect tool calls or deployment plans. Fix: Implement source validation in the scaffold. Verify file freshness, checksum integrity, and API response schemas before injecting context into the agent's working memory.

3. Guardrail Latency Tax

Explanation: Using stacked LLM validators to check agent actions. Each LLM-on-LLM check adds significant round-trip latency and does not eliminate non-determinism. Fix: Replace LLM validators with deterministic checks (regex, AST analysis, policy engines). Deterministic guards provide faster, more predictable validation for known risk patterns.

4. Hidden Runtime State

Explanation: Agents writing code that runs locally but fails in production due to missing environment variables, database schemas, or upstream headers. The agent lacks visibility into the live environment. Fix: Use explicit state injection. The scaffold must query and inject all required runtime context (env vars, schemas, auth tokens) before the agent begins tool execution.

5. Unbounded Tool Execution

Explanation: Allowing agents to execute tools without transactional limits. This can lead to blast radius events, such as 30 erroneous commits or mass database deletions in a single run. Fix: Enforce blast radius limits at the tool layer. Cap the number of commits, database rows affected, and API calls per session. Use dry-run modes for initial validation.

6. Non-Deterministic Trace Blindness

Explanation: Traditional observability tools fail to capture agentic workflows because identical prompts can produce different tool sequences. Traces branch through planning, memory retrieval, and retries. Fix: Implement agentic tracing that propagates unique trace IDs across all tool calls and retries. Instrument the scaffold to log decision points, tool inputs/outputs, and validation results for full replayability.

7. Tool Rotation Burn

Explanation: Underestimating the cost of switching between AI coding tools. Retrospectives indicate rotation costs can reach hundreds of dollars per developer over 1.5 years due to retraining, workflow adaptation, and license fees. Fix: Standardize on a scaffold architecture that abstracts the underlying model. This allows model swapping without rewriting integration logic. Budget for rotation costs and evaluate total cost of ownership, not just per-token pricing.

Production Bundle

Action Checklist

• Inject Runtime State: Ensure all environment variables, database schemas, and auth headers are explicitly injected into the agent context before execution.
• Set Blast Radius Limits: Configure hard limits on commits, database rows, and API calls per session. Start with conservative defaults.
• Implement Deterministic Guardrails: Replace LLM validators with regex, AST, and policy-based checks for tool validation.
• Enable Dry-Run Mode: Run all agent workflows in dry-run mode initially to validate scaffold behavior without side effects.
• Instrument Agentic Tracing: Deploy tracing that captures tool sequences, validation results, and decision points for full observability.
• Validate on Live Harness: Test models against a production-like harness that includes hidden state and non-deterministic tool responses.
• Budget for Rotation: Account for tool rotation costs in long-term planning. Abstract model dependencies to minimize switching friction.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Solo Developer	Use cost-effective model + strict scaffold	Scaffold reliability matters more than model size for side projects.	Low
Team (5-20 devs)	Standardize scaffold + budget rotation	Consistency and reduced friction outweigh minor benchmark gains.	Medium
Latency-Critical App	Deterministic guardrails only	LLM validators introduce unacceptable latency; deterministic checks are faster.	Low
Production Data Access	Blast radius caps + dry-run validation	Safety is paramount; limits prevent catastrophic data loss.	N/A
Cost-Sensitive Batch	Small open models (e.g., SmolLM3, Qwen)	Benchmarks show small models can compete; validate on live harness first.	Low

Configuration Template

# agent-scaffold-config.yaml
orchestrator:
  blast_radius:
    max_commits: 5
    max_database_rows: 10
    dry_run: true
  guardrails:
    type: deterministic
    rules:
      - pattern: "rm -rf /"
        action: block
      - pattern: "DROP DATABASE"
        action: block
  state_injection:
    sources:
      - type: env_var
        prefix: "APP_"
      - type: db_schema
        tables: ["users", "orders"]
      - type: auth_header
        key: "X-API-Key"
  observability:
    tracing:
      enabled: true
      propagate_trace_id: true
    logging:
      level: debug
      include_tool_io: true

Quick Start Guide

1. Define State Schema: Identify all runtime dependencies (env vars, schemas, headers) required by your application. Configure the scaffold to inject these explicitly.
2. Set Safety Limits: Initialize the orchestrator with conservative blast radius limits. Enable dry-run mode to test behavior safely.
3. Add Deterministic Checks: Implement guardrails for known risk patterns (destructive commands, protected schemas). Avoid LLM-based validation for performance.
4. Run Validation: Execute a test workflow in dry-run mode. Verify that state injection works, guardrails trigger correctly, and blast radius limits are enforced.
5. Deploy with Tracing: Enable agentic tracing and deploy to a staging environment. Monitor tool sequences and validation results to ensure scaffold reliability before production rollout.