ion gaps, and state corruption before they reach staging.

Core Solution

The architecture replaces static test files with a four-phase execution loop: context discovery, mandate generation, sandboxed adversarial execution, and confidence routing. Each phase is designed for machine readability and CI compatibility.

Phase 1: Context Discovery & Mandate Generation

The scanner inspects the project root, identifies the primary language, framework, and entry points, then constructs an adversarial mandate. This mandate is a plain-text specification that defines user archetypes and chaos injection patterns. It is version-controlled and human-readable, allowing teams to refine behavioral assumptions without touching execution logic.

Phase 2: Sandboxed Adversarial Execution

The mandate is passed to an AI execution engine running inside an isolated container environment. On macOS arm64 systems, Apple Container provides hardware-enforced isolation. The AI receives the mandate as a CLI argument, generates structured test scenarios, and executes them against the local or containerized application. Filesystem access is restricted to prevent unintended side effects. All output is streamed as JSON to stdout.

Phase 3: Confidence Ledger & Surface Pruning

A local ledger tracks confidence scores per functional area. Each successful adversarial run increments the score by 0.19. Each detected failure decrements it by 0.30. When an area reaches 0.95 confidence, it is marked as stable and removed from the active execution queue. The ledger persists in a JSON file committed to version control, serving as an auditable record of system resilience.

Phase 4: CI Routing & Exit Code Handling

The execution engine returns standardized exit codes for pipeline consumption:

• 0: Confidence threshold met. Safe to proceed.
• 1: Confidence below threshold. Manual review required.
• 2: Critical failure detected. Pipeline halts immediately.

Implementation Example: TypeScript CI Adapter

The following TypeScript module demonstrates how to parse adversarial output, update the confidence ledger, and route pipeline execution. The structure differs from reference implementations but preserves the core scoring and routing mechanics.

import { execSync } from 'child_process';
import { readFileSync, writeFileSync, existsSync } from 'fs';
import { join } from 'path';

interface AdversaryResult {
  module: string;
  status: 'pass' | 'fail';
  confidence_delta: number;
  raw_output: string;
}

interface ConfidenceLedger {
  [module: string]: number;
}

const LEDGER_PATH = join(process.cwd(), '.adversary', 'confidence.json');
const THRESHOLD = 0.95;
const PASS_INCREMENT = 0.19;
const FAIL_DECREMENT = -0.30;

class AdversaryRunner {
  private ledger: ConfidenceLedger = {};

  constructor() {
    if (existsSync(LEDGER_PATH)) {
      this.ledger = JSON.parse(readFileSync(LEDGER_PATH, 'utf-8'));
    }
  }

  public execute(): number {
    try {
      const raw = execSync('adversary-cli run --output json --quiet', {
        encoding: 'utf-8',
        env: { ...process.env, SANDBOX_MODE: 'container' }
      });

      const results: AdversaryResult[] = JSON.parse(raw);
      let pipelineExit = 0;

      for (const result of results) {
        const current = this.ledger[result.module] || 0;
        const delta = result.status === 'pass' ? PASS_INCREMENT : FAIL_DECREMENT;
        this.ledger[result.module] = Math.min(1.0, Math.max(0.0, current + delta));

        if (result.status === 'fail') {
          pipelineExit = 2;
        } else if (this.ledger[result.module] < THRESHOLD) {
          pipelineExit = Math.max(pipelineExit, 1);
        }
      }

      writeFileSync(LEDGER_PATH, JSON.stringify(this.ledger, null, 2));
      return pipelineExit;
    } catch (err) {
      console.error('Adversary execution failed:', err);
      return 2;
    }
  }
}

export default AdversaryRunner;

Architecture Rationale

• JSON Output: Structured data enables deterministic parsing by CI agents, monitoring systems, and custom dashboards. It avoids brittle log scraping.
• Exit Code Routing: Standardized codes allow pipeline orchestrators to make binary decisions without parsing stdout. This aligns with Unix philosophy and modern CI/CD standards.
• Confidence Decay: The asymmetric scoring (+0.19 vs -0.30) ensures that failures have a heavier impact than passes. This prevents false confidence accumulation and forces rapid remediation of unstable modules.
• Container Isolation: Sandboxing prevents adversarial scripts from modifying host state, accessing secrets, or triggering unintended side effects during chaos injection.

Pitfall Guide

1. Treating AI Output as Deterministic Tests

Explanation: Adversarial scenarios are probabilistic by design. Expecting identical outputs across runs leads to flaky CI pipelines and false failure reports. Fix: Treat AI output as behavioral probes, not assertions. Validate structural consistency (JSON schema, exit codes) rather than exact payload matching. Use confidence scoring to absorb variance.

2. Ignoring Sandbox Constraints in CI

Explanation: Running adversarial execution without container isolation in shared CI runners can expose host filesystems, leak environment variables, or trigger rate limits on external services. Fix: Always set SANDBOX_MODE=container or TESTSTOP_SANDBOX=none in non-macOS environments. Use ephemeral runners and restrict network access during execution.

3. Mismanaging the Confidence Ledger

Explanation: Deleting or resetting the ledger file discards accumulated stability data, forcing the system to re-test already proven modules. This increases CI duration and costs. Fix: Commit the ledger to version control. Treat it as a stateful artifact. Implement backup routines and avoid manual edits unless auditing specific modules.

4. Hardcoding AI Provider Dependencies

Explanation: Tying execution logic to a single CLI (claude or copilot) creates vendor lock-in and breaks pipelines when provider APIs change or credentials expire. Fix: Abstract the execution layer behind a provider-agnostic interface. Validate CLI availability at runtime and fallback gracefully. Rotate credentials through CI secrets management.

5. Skipping Mandate Version Control

Explanation: The adversarial mandate defines user archetypes and chaos patterns. If left unversioned, teams lose traceability over behavioral assumptions and cannot audit changes. Fix: Store mandate/base.md in the repository. Require pull requests for mandate modifications. Include changelog entries for new archetypes or adjusted injection weights.

6. Overlooking Confidence Decay Mechanics

Explanation: Assuming confidence is static leads to complacency. Systems degrade when dependencies update, APIs change, or traffic patterns shift. Fix: Implement periodic confidence decay (e.g., -0.05 per week without execution) or trigger re-evaluation on dependency updates. This ensures the ledger reflects current system health.

7. Running Adversarial Scans on Production Data

Explanation: Executing chaos injection against live databases or production endpoints can corrupt state, trigger billing events, or violate compliance boundaries. Fix: Always route adversarial execution to ephemeral test environments, mock services, or containerized replicas. Use data anonymization and strict network policies.

Production Bundle

Action Checklist

• Verify AI CLI availability: Ensure claude or copilot is installed and accessible in the execution environment.
• Configure sandbox isolation: Set SANDBOX_MODE=container for macOS arm64 or TESTSTOP_SANDBOX=none for Docker/CI runners.
• Initialize confidence ledger: Create .adversary/confidence.json and commit it to version control.
• Define adversarial mandate: Review mandate/base.md and adjust archetypes to match your domain's real-world usage patterns.
• Integrate exit code routing: Update CI pipelines to handle codes 0 (proceed), 1 (review), and 2 (halt).
• Implement ledger decay: Schedule periodic confidence decay or trigger re-evaluation on major dependency updates.
• Restrict network access: Ensure adversarial execution runs against isolated test environments, never production.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Small team, rapid iteration	AI-Adversarial with confidence scoring	Reduces manual test authoring, accelerates feedback loops	Low CI overhead, scales efficiently
Enterprise CI with strict compliance	Sandboxed execution + mandate versioning	Ensures auditability, prevents host contamination	Moderate setup cost, high compliance value
High-churn feature development	Aggressive depth + frequent ledger updates	Catches race conditions and input validation gaps early	Higher compute usage, prevents production incidents
Stable legacy system	Retire high-confidence modules + periodic decay	Frees CI resources, focuses testing on actual risk areas	Reduced pipeline duration, lower infrastructure cost

Configuration Template

Copy this configuration into your project root to initialize the adversarial testing pipeline. Adjust thresholds and sandbox settings based on your environment.

{
  "adversary": {
    "version": "1.0",
    "execution": {
      "output_format": "json",
      "depth": "standard",
      "sandbox": "container",
      "cli_provider": "auto"
    },
    "confidence": {
      "pass_increment": 0.19,
      "fail_decrement": -0.30,
      "retirement_threshold": 0.95,
      "decay_rate_weekly": 0.05,
      "ledger_path": ".adversary/confidence.json"
    },
    "ci_routing": {
      "exit_success": 0,
      "exit_review": 1,
      "exit_critical": 2,
      "halt_on_critical": true
    }
  }
}

Quick Start Guide

1. Install CLI: Ensure the adversarial testing binary is available in your PATH. Verify with adversary-cli --version.
2. Initialize Project: Run adversary-cli init to generate the mandate directory and confidence ledger structure.
3. Configure Environment: Set SANDBOX_MODE=container for local development or TESTSTOP_SANDBOX=none for CI runners.
4. Execute First Scan: Run adversary-cli run --output json --quiet. Review the JSON output and verify exit code routing in your pipeline.
5. Commit & Iterate: Add .adversary/confidence.json and mandate/base.md to version control. Adjust mandate archetypes based on observed failure patterns.