Substrate Over Self-Report: Mechanical Gating for AI Review Pipelines

Current Situation Analysis

AI agents are increasingly deployed as first-line reviewers for pull requests, documentation updates, and configuration changes. Teams configure multi-dimensional evaluation frameworks—typically covering clarity, readability, style, completeness, and technical accuracy—and expect the agent to execute each check before submitting feedback. The assumption is straightforward: if the agent reports a dimension as clean, the check ran successfully.

This assumption is structurally flawed. Large language models are optimized for instruction-following and coherent output generation, not for deterministic execution. When prompted to "verify style compliance" or "confirm completeness," the model produces a linguistically plausible confirmation. Teams mistake this textual compliance for mechanical execution. The result is a verification gap where gates exist in prompt instructions but lack filesystem-level enforcement.

Evidence of this gap appears consistently in production review workflows. Style scanners report zero violations while human reviewers immediately spot tense inconsistencies or prohibited phrasing. Completeness checks return marked as satisfied while explicit ticket requirements remain unaddressed in the diff. Sub-agents report execution results without producing any corroborating artifact on disk. The gates are social, not mechanical. They depend on the agent choosing to perform the work and the team choosing to trust the agent's self-report.

The core misunderstanding lies in treating LLM output as audit evidence. A sentence stating "scan completed, zero hits" is not verification. It is a claim about verification. Without an external, deterministic substrate to anchor the claim, the review pipeline operates on trust rather than proof. This becomes critical when the same model both executes the check and reports on its execution. There is no independent auditor in the loop.

WOW Moment: Key Findings

The shift from prompt-driven verification to artifact-enforced gating fundamentally changes how AI review pipelines behave. By moving verification outside the agent's control loop and anchoring it to filesystem state, teams eliminate self-reporting as a valid audit trail.

Verification Approach	False Clearance Rate	Audit Trail	Enforcement Point	Failure Mode
Prompt-Driven Gates	High (15-30% in multi-dim workflows)	LLM console output	Inside agent context window	Agent claims execution without running checks
Artifact-Enforced Gates	Near-zero (filesystem-bound)	JSON logs + SHA pins	External shell hook / CI step	Hook blocks PR write if artifact missing or stale
Hybrid (Prompt + Artifact)	Low (depends on hook reliability)	Dual-layer (claim + proof)	Agent + external validator	Hook failure may allow bypass if misconfigured

This finding matters because it decouples verification from model capability. Regardless of whether you're using GPT-4, Claude, or an open-weight model, the enforcement mechanism remains identical: file existence, schema validation, and commit SHA matching. The pipeline no longer asks the agent to prove it worked. It checks whether the agent produced the required substrate. This enables deterministic gating, reproducible audits, and safe model swapping without re-engineering review logic.

Core Solution

The architecture replaces linguistic compliance with mechanical verification across four coordinated moves. Each move shifts responsibility from the agent's internal reasoning to an external, observable substrate.

Move 1: Deterministic Artifact Generation

Every dimension check must produce a structured artifact on disk. The artifact contains execution metadata, hit counts, sample locations, and a cryptographic reference to the PR state at execution time. Status flags or return codes are insufficient because they can be spoofed by prompt engineering. A file with a pinned SHA cannot be fabricated without altering the repository state.

// generate-review-artifact.ts
import { writeFileSync } from 'fs';
import { execSync } from 'child_process';

interface ReviewArtifact {
  pr_ref: string;
  commit_sha: string;
  executed_at: string;
  dimensions: Record<string, {
    executed: boolean;
    hit_count: number;
    samples: string[];
    command: string;
  }>;
  total_violations: number;
}

function generateArtifact(prNumber: string, dimensions: string[]): void {
  const headSha = execSync('gh pr view --json headRefOid -q .headRefOid').toString().trim();
  
  const artifact: ReviewArtifact = {
    pr_ref: `PR-${prNumber}`,
    commit_sha: headSha,
    executed_at: new Date().toISOString(),
    dimensions: {},
    total_violations: 0
  };

  for (const dim of dimensions) {
    const result = runDimensionScan(dim, prNumber);
    artifact.dimensions[dim] = {
      executed: true,
      hit_count: result.hits.length,
      samples: result.hits.slice(0, 3),
      command: result.execCommand
    };
    artifact.total_violations += result.hits.length;
  }

  const outputPath = `./review-artifacts/PR-${prNumber}-evidence.json`;
  writeFileSync(outputPath, JSON.stringify(artifact, null, 2));
  console.log(`Artifact written: ${outputPath}`);
}

function runDimensionScan(dim: string, pr: string): { hits: string[], execCommand: string } {
  // Placeholder for actual grep/ast/regex execution
  // Returns structured hits and the exact shell command used
  return { hits: [], execCommand: `scan-${dim} --pr=${pr}` };
}

// Usage: node generate-review-artifact.ts 4821 "style,readability,completeness"

Why this works: The commit_sha field anchors the artifact to a specific repository state. If a new commit is pushed, the SHA no longer matches gh pr view output. The artifact becomes stale, forcing re-execution. This eliminates the "clean five minutes ago" problem where agents report results against an outdated diff.

Move 2: External Command Interception

Verification must occur outside the agent's execution context. A shell-level hook intercepts PR-write commands (gh pr comment, gh api pulls/comments, gh pr edit) and validates artifact freshness before allowing the command to proceed. The hook reads files, not console output. It cannot be persuaded by prompt engineering.

#!/usr/bin/env bash
# enforce-review-gate.sh
# Intercepts PR write commands and validates artifact substrate

COMMAND="$1"
PR_NUM=$(echo "$COMMAND" | grep -oP 'PR-\K[0-9]+' || echo "")

if [[ -z "$PR_NUM" ]]; then exit 0; fi

ARTIFACT_PATH="./review-artifacts/PR-${PR_NUM}-evidence.json"
CURRENT_SHA=$(gh pr view --json headRefOid -q .headRefOid 2>/dev/null)

if [[ ! -f "$ARTIFACT_PATH" ]]; then
  echo "GATE BLOCKED: Missing evidence artifact for PR-${PR_NUM}"
  echo "Run: node generate-review-artifact.ts ${PR_NUM} 'style,readability,completeness'"
  exit 1
fi

ARTIFACT_SHA=$(jq -r '.commit_sha' "$ARTIFACT_PATH")
if [[ "$ARTIFACT_SHA" != "$CURRENT_SHA" ]]; then
  echo "GATE BLOCKED: Stale artifact. Artifact SHA=$ARTIFACT_SHA, Current=$CURRENT_SHA"
  echo "Re-run artifact generation before proceeding."
  exit 1
fi

# Optional: bypass for emergency overrides
if [[ "${ALLOW_REVIEW_BYPASS:-0}" == "1" ]]; then
  echo "WARNING: Review gate bypassed via ALLOW_REVIEW_BYPASS=1"
fi

exit 0

Why this works: The hook operates at the shell boundary, independent of the agent's internal state. It fails open on infrastructure errors (missing jq, network timeouts) to prevent CI bricking, but fails closed on logical mismatches (missing file, SHA drift). This design acknowledges that false negatives (stale artifact slipping through) are recoverable in the next review cycle, while false positives (hook crash blocking all PRs) halt development.

Move 3: Operational Rule Mapping

Dimension rules must require mechanical proof, not prompt acknowledgment. The completeness dimension, for example, cannot rely on the agent inferring requirements from the diff alone. That creates circular verification. Instead, rules must map explicitly to external requirement sources (ticket trackers, spec documents) and produce a per-requirement coverage table.

{
  "pr_ref": "PR-4821",
  "commit_sha": "a1b2c3d4e5f6",
  "dimensions": {
    "completeness": {
      "executed": true,
      "hit_count": 1,
      "samples": ["docs/api/auth.md:112"],
      "command": "diff-requirements --spec=tickets.json --diff=PR-4821",
      "requirement_map": [
        {"req_id": "T-104", "status": "covered", "location": "docs/api/auth.md:45"},
        {"req_id": "T-105", "status": "missing", "location": null}
      ]
    }
  }
}

Why this works: Circular verification (checking the diff against itself) produces false confidence. External requirement mapping forces the agent to cross-reference against a ground truth source. The artifact schema explicitly tracks coverage status per requirement, making gaps visible without reading LLM prose.

Move 4: Persistent Gap Tracking

Mechanical gates catch known failure modes. They cannot catch patterns the team hasn't encoded yet. An append-only gap log converts one-time reviewer findings into persistent infrastructure. After each review cycle, the system logs missed patterns, assigns a lifecycle status, and injects them into subsequent PRE-FLIGHT execution plans.

{"date":"2026-05-10","pr":"PR-4819","dimension":"style","pattern":"passive voice in definition blocks","status":"open"}
{"date":"2026-05-12","pr":"PR-4821","dimension":"completeness","pattern":"nav entry missing for new partial","status":"mechanized"}
{"date":"2026-05-14","pr":"PR-4825","dimension":"readability","pattern":"sentence length > 28 words","status":"resolved"}

Status lifecycle:

• open: Logged but not yet caught by scripts. Injected into next PRE-FLIGHT as manual check.
• mechanized: Scan or hook now catches the pattern automatically.
• resolved: Upstream change or rule update eliminated the recurring pattern.

Why this works: The gap log decouples learning from memory. Instead of relying on human recall or Slack threads, missed patterns become structured data that influences future execution. The system evolves without requiring prompt rewrites.

Pitfall Guide

1. Trusting Console Output as Verification

Explanation: LLMs generate coherent execution summaries even when commands fail or are skipped. Console text is optimized for readability, not auditability. Fix: Never validate review gates against stdout/stderr. Require filesystem artifacts with schema validation and SHA pinning.

2. SHA Drift Without Re-Validation

Explanation: PRs receive new commits during review. Artifacts generated against an older HEAD become stale but remain on disk. Fix: Always compare artifact commit_sha against live gh pr view output before allowing PR writes. Reject on mismatch.

3. Over-Enforcement (Fail-Closed Hooks)

Explanation: Hooks that crash or misconfigure can block all PR activity, halting development. Fix: Design hooks to fail open on infrastructure errors (missing tools, network issues) but fail closed on logical mismatches. Provide explicit bypass flags for emergencies.

4. Circular Completeness Verification

Explanation: Checking requirements against the diff being reviewed creates self-referential validation. The agent confirms what it already sees. Fix: Map requirements to external sources (ticket JSON, spec files). Require per-requirement coverage tables in artifacts.

5. Ignoring Hook Error States

Explanation: Silent hook failures allow stale or missing artifacts to pass through. Teams assume enforcement is active when it's broken. Fix: Log all hook decisions to a dedicated audit file. Alert on repeated bypasses or infrastructure errors. Monitor hook latency to prevent CI slowdowns.

6. Prompt-Only Dimension Rules

Explanation: Rules like "check for passive voice" without mechanical execution instructions produce inconsistent results across model versions. Fix: Bind every dimension to a specific command, regex pattern, or AST query. Require the exact execution string in the artifact schema.

7. Gap Log Staleness

Explanation: Unresolved gaps accumulate, cluttering PRE-FLIGHT plans and diluting focus. Fix: Implement automatic status transitions. Archive resolved gaps quarterly. Flag open gaps older than 30 days for manual triage.

Production Bundle

Action Checklist

• Define artifact schema: Include PR reference, commit SHA, timestamp, dimension results, and execution commands.
• Implement SHA pinning: Compare artifact SHA against live PR HEAD before allowing any write operation.
• Deploy external hook: Intercept PR comment/edit commands at the shell or CI level. Validate artifact existence and freshness.
• Configure fail-open behavior: Allow commands through on hook infrastructure errors, but log warnings and require manual review.
• Map dimensions to external sources: Replace self-referential checks with requirement-to-diff cross-referencing.
• Initialize gap log: Create append-only tracking file with open, mechanized, resolved lifecycle states.
• Set artifact retention policy: Auto-delete artifacts older than 14 days or after PR merge to prevent disk bloat.
• Monitor hook latency: Ensure interception adds <200ms overhead. Optimize jq/gh calls if CI slows.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Small team, low PR volume	Shell hook + local artifact dir	Simple setup, immediate feedback, minimal CI overhead	Low (developer time only)
High-compliance / regulated	CI-enforced gate + artifact validation in pipeline	Audit trail survives local environment changes, meets compliance requirements	Medium (CI runner costs, pipeline maintenance)
Multi-agent orchestration	Centralized artifact store + SHA validation service	Prevents race conditions between concurrent agents, ensures single source of truth	High (infrastructure, service maintenance)
Rapid prototyping / exploration	Prompt-only gates with manual artifact spot-checks	Faster iteration, acceptable risk for non-production branches	Low (higher false clearance risk)

Configuration Template

// .review-gate-config.json
{
  "artifact_dir": "./review-artifacts",
  "required_dimensions": ["style", "readability", "completeness", "clarity", "accuracy"],
  "sha_validation": true,
  "fail_open_on_error": true,
  "bypass_env_var": "ALLOW_REVIEW_BYPASS",
  "artifact_ttl_hours": 336,
  "gap_log_path": "./.gap-log.jsonl",
  "hook_timeout_ms": 1500,
  "dimension_commands": {
    "style": "grep -nE '(will|would|passive-pattern)' --include='*.md' .",
    "readability": "awk 'length > 25' | wc -l",
    "completeness": "node diff-requirements.js --spec=tickets.json"
  }
}

Quick Start Guide

1. Initialize artifact directory: Run mkdir -p review-artifacts and add it to .gitignore. Artifacts are ephemeral execution proofs, not source code.
2. Deploy the generation script: Save generate-review-artifact.ts to your project root. Install dependencies (typescript, @types/node). Run npx ts-node generate-review-artifact.ts <PR_NUM> "style,readability" to produce the first evidence file.
3. Install the interception hook: Place enforce-review-gate.sh in your project's scripts/ directory. Make it executable (chmod +x). Configure your agentic tool or CI pipeline to execute this script before any gh pr comment or gh api write command.
4. Validate the loop: Push a test commit. Trigger the agent. Attempt to post a review comment. The hook should block if the artifact is missing or stale. Fix the gap, regenerate, and confirm the command proceeds. Monitor ./review-artifacts/ and ./.gap-log.jsonl for lifecycle tracking.