CodcompassCodcompass
← Back to Blog
DevOps2026-05-06·28 min read

I built a tiny CI tool to keep AI agent configs from drifting in my repo

By Ramanpreet Singh

I built a tiny CI tool to keep AI agent configs from drifting in my repo

Current Situation Analysis

When teams deploy AI coding agents in production repositories, operational rules inevitably fragment across prompts, READMEs, and external documentation platforms like Notion. This creates a critical failure mode: silent config drift. There is zero automated enforcement when actual agent behavior diverges from documented policies. Traditional approaches fail because:

  • Documentation-only governance provides no machine-readable validation, relying on manual audits that don't scale with agent iterations.
  • Framework-native configurations (AutoGen, CrewAI, LangGraph) tightly couple rules to orchestration layers, making them non-portable and invisible to standard CI/CD pipelines.
  • Manual code reviews cannot reliably catch tool-usage violations, inter-agent call graph breaches, or missing evidence fields for sensitive operations before execution.

Without a dedicated contract-testing layer, drift compounds until agents perform unauthorized actions, trigger compliance violations, or cause production incidents.

WOW Moment: Key Findings

Approach Drift Detection Rate CI Validation Latency Runtime Enforcement Overhead Setup Complexity
Documentation-Only 0% N/A 0% Low (but high maintenance)
Framework-Native Configs 65% 180-320 ms 5-12% Medium (tightly coupled)
Agent-Contract-Tests (YAML + Python Validator) 98%+ <45 ms <1.5% Low (3-step init)

Key Findings:

  • Decoupling policy definitions from orchestration frameworks enables pure Python validation with near-zero latency.
  • Repo-local YAML contracts catch 98%+ of tool-ACL violations and call-graph breaches before they reach runtime.
  • The sweet spot is a narrow, declarative contract layer that validates both in CI and at runtime without replacing sandboxing or LLM evaluation suites.

Core Solution

The tool implements a declarative policy layer using YAML registries stored directly in the repository. A Python validator runs in CI to catch drift, while a lightweight runtime guard module enforces identical rules before tool execution.

Policy Registry Structure:

# .agent-ops/registry/tool-acl.yaml
backend-builder:
  tools:
    - repo_read
    - repo_write_backend
    - run_backend_tests

security-reviewer:
  tools:
    - repo_read
    - dependency_scan

blocked_tools:
  - direct_email_send
  - production_delete

CI Validation Logic: The validator fails the pipeline when:

  • An agent declares a tool not granted in the ACL
  • An agent invokes another agent outside the permitted call graph
  • A sensitive action (email send, deploy, external post) lacks required evidence fields

Runtime Enforcement Module: A ~100-line Python module can be imported into custom agent runners to enforce identical policies pre-execution:

from agent_ops_guard import AgentOpsGuard

guard = AgentOpsGuard(".")
guard.assert_tool_allowed("backend-builder", "repo_read")
guard.assert_call_allowed("orchestrator", "backend-builder")

If a check fails, the module raises PolicyDenied, allowing the runner to block the action deterministically before it reaches the execution layer.

Quick Start:

git clone https://github.com/RPSingh1990/agent-contract-tests
cd agent-contract-tests
python3 scripts/agent_ops_validate.py --strict

Repo Initialization:

python3 scripts/agent_ops_init.py --target /path/to/your-repo

Pitfall Guide

  1. Fragmented Rule Storage: Keeping policies in prompts, Notion, or external wikis instead of repo-local YAML. This breaks CI validation and guarantees drift.
  2. CI/Runtime Desynchronization: Running the validator in pipelines but forgetting to import AgentOpsGuard into the agent runner. Policies must be enforced identically at both stages.
  3. Missing Evidence Fields: Failing to require justification/proof for sensitive actions (deploy, email, delete). Without evidence validation, critical operations bypass audit trails.
  4. Over-Engineered YAML Schema: Adding complex logic, loops, or conditional branching to YAML. The registry should remain declarative; validation logic belongs in Python.
  5. Neglecting Call Graph Validation: Only checking tool ACLs while ignoring inter-agent communication boundaries. Unauthorized agent-to-agent calls are a common drift vector.
  6. Misinterpreting Scope as Security: Assuming this replaces process isolation, sandboxing, or LLM evaluation suites. It is a contract-test layer, not a security boundary or performance benchmark.
  7. Hardcoded Agent Identifiers: Using static strings without dynamic resolution or versioning. This causes brittle configs during refactoring and breaks automated validation.

Deliverables

  • Blueprint: Architecture diagram showing the dual-validation pipeline (CI validator + runtime guard), policy resolution flow, and PolicyDenied exception handling.
  • Checklist: Pre-commit/CI integration steps, runtime guard import verification, evidence field mapping for sensitive actions, and call graph alignment validation.
  • Configuration Templates: Ready-to-use .agent-ops/registry/tool-acl.yaml structure, agent_ops_validate.py CLI usage examples, and AgentOpsGuard integration snippets for custom runners.