A security checklist for AI-generated pull requests

pelines

• Database migrations or schema alterations
• External integrations (webhooks, APIs, message queues)
• Infrastructure or deployment configuration
• AI agent orchestration or tool execution

Engineering Rationale: Blast radius classification prevents review fatigue and ensures deep inspection is reserved for changes that can escalate privileges, leak data, or trigger irreversible side effects.

Phase 2: Trace Untrusted Input Boundaries

AI-generated code frequently assumes input validity. Security review must explicitly map every external data source to its downstream consumers.

External inputs include:

• HTTP request bodies and query parameters
• Custom headers and cookies
• File uploads and binary payloads
• Webhook signatures and event payloads
• User-generated content or comments
• Retrieved documents or external API responses
• LLM completions or agent instructions

Trace the data flow using a strict boundary model:

External Input → Validation → Authorization → Side Effect

If any step is missing, the review must halt until the gap is resolved. AI models often skip validation or assume downstream consumers will sanitize data. This assumption is a primary vector for injection, privilege escalation, and data leakage.

Phase 3: Enforce Object-Level Authorization

Authentication verifies identity. Authorization verifies permission. AI-generated code consistently conflates the two, checking if (user.isLoggedIn) and assuming access is granted.

Object-level authorization requires explicit ownership or tenancy verification before any data mutation or retrieval.

Example: Tenant-Scoped Resource Guard

import { RequestContext } from './auth/context';
import { ResourceRepository } from './data/repository';
import { PolicyViolationError } from './errors/policy';

export class TenantResourceGuard {
  constructor(
    private readonly repo: ResourceRepository,
    private readonly context: RequestContext
  ) {}

  async assertAccess(resourceId: string): Promise<void> {
    const resource = await this.repo.findById(resourceId);
    
    if (!resource) {
      throw new PolicyViolationError('Resource not found');
    }

    const isOwner = resource.tenantId === this.context.tenantId;
    const hasRole = this.context.roles.includes('admin') || 
                    this.context.roles.includes('editor');

    if (!isOwner && !hasRole) {
      throw new PolicyViolationError('Insufficient permissions for resource');
    }
  }
}

Engineering Rationale: Multi-tenant systems require explicit isolation. Role checks alone are insufficient without tenant scoping. The guard pattern centralizes policy enforcement, making it auditable and testable.

Phase 4: Contain Model Output as Untrusted Input

When an LLM influences application behavior, its output must be treated as untrusted external data. Prompt injection is fundamentally a tool authorization problem, not a natural language processing challenge.

Risk pattern:

Untrusted Content → Model Inference → Action Execution

Mitigation requires deterministic controls outside the model:

• Strict tool allowlists with explicit capabilities
• Argument validation before execution
• Credential scoping to minimum required permissions
• Read/write separation for agent tools
• Explicit confirmation gates for destructive operations
• Fail-closed routing when output is ambiguous

Example: Deterministic Tool Router

import { ToolRegistry } from './agent/tools';
import { ExecutionError } from './errors/execution';

export class AgentToolExecutor {
  private readonly allowedTools = new Set(['read_document', 'search_database', 'generate_report']);

  async execute(toolName: string, args: Record<string, unknown>): Promise<unknown> {
    if (!this.allowedTools.has(toolName)) {
      throw new ExecutionError(`Tool '${toolName}' is not permitted`);
    }

    const validatedArgs = this.validateArguments(toolName, args);
    const tool = ToolRegistry.get(toolName);
    
    return await tool.run(validatedArgs);
  }

  private validateArguments(tool: string, args: Record<string, unknown>): Record<string, unknown> {
    // Schema validation, type checking, and boundary enforcement
    // Implementation depends on tool contract
    return args;
  }
}

Engineering Rationale: Prompts are not security boundaries. Deterministic allowlists, argument validation, and capability scoping ensure that model output cannot escalate privileges or trigger unauthorized side effects.

Phase 5: Demand Evidence-Based Validation

"Looks patched" is insufficient for security-sensitive changes. Every PR modifying authorization, data flow, or external integration must include verifiable evidence of invariant enforcement.

Required validation artifacts:

• Regression test covering the exact failure path
• Reproducer script or curl command demonstrating the exploit
• Explicit before/after behavior documentation
• Negative test cases for boundary violations

Example: Negative Test Structure

describe('InvoiceAccessPolicy', () => {
  it('denies cross-tenant invoice retrieval', async () => {
    const attackerContext = createMockContext({ tenantId: 'org_1', roles: ['user'] });
    const targetInvoiceId = 'inv_org_2_999';

    await expect(
      fetchInvoice(attackerContext, targetInvoiceId)
    ).rejects.toThrow(PolicyViolationError);
  });

  it('rejects replayed webhook signatures', async () => {
    const payload = loadFixture('webhook/invoice.created.json');
    const staleSignature = 'sha256=expired_timestamp';

    await expect(
      processWebhook(payload, staleSignature)
    ).rejects.toThrow(SignatureVerificationError);
  });
});

Engineering Rationale: Happy-path tests confirm functionality. Negative tests confirm security. Without explicit boundary validation, AI-generated code will pass CI while leaving privilege escalation vectors open.

Pitfall Guide

1. The Green CI Fallacy

Explanation: Assuming passing tests and linting equate to security compliance. AI-generated tests heavily favor success paths and rarely cover authorization bypasses or data leakage. Fix: Mandate negative test coverage for all auth/data changes. Require explicit boundary assertions in test suites.

2. Authentication/Authorization Conflation

Explanation: Checking user.isAuthenticated and assuming access is granted. This ignores object ownership, tenant isolation, and role scoping. Fix: Implement explicit resource guards that verify tenant ID, ownership, and role permissions before any data access.

3. Prompt-as-Perimeter

Explanation: Relying on system prompts or instructions to restrict model behavior. LLMs can be coerced into ignoring constraints, especially with adversarial input. Fix: Treat model output as untrusted. Enforce deterministic allowlists, argument validation, and capability scoping at the execution layer.

4. Happy-Path Test Reliance

Explanation: Accepting PRs with only success-case tests. This leaves privilege escalation, replay attacks, and boundary violations unguarded. Fix: Require negative tests for every security-sensitive change. Validate denied access, invalid signatures, and cross-tenant isolation.

5. Vague Security Feedback

Explanation: Review comments like "check permissions" or "might be risky" provide no actionable path. They delay resolution and obscure the actual vulnerability. Fix: Structure findings with code path, impact, and remediation direction. Example: "Endpoint validates session but lacks tenant scoping. Cross-tenant data leakage possible. Add tenantId comparison before query execution."

6. Ignoring Indirect Data Flows

Explanation: Focusing only on direct user input while missing data injected via webhooks, AI outputs, or background jobs. These vectors bypass UI-level validation. Fix: Map all data ingress points. Apply identical validation and authorization rules to programmatic, webhook, and model-generated inputs.

7. Over-Reliance on LLM Self-Correction

Explanation: Assuming the model will fix security gaps when prompted. LLMs lack persistent security context and often reintroduce bypasses in subsequent iterations. Fix: Treat AI as a code generator, not a security auditor. Enforce policy checks through static analysis, runtime guards, and mandatory test coverage.

Production Bundle

Action Checklist

• Classify blast radius: Identify high-impact surfaces (auth, billing, data, infra, agents) before deep review.
• Map input boundaries: Trace every external data source through validation, authorization, and side effects.
• Enforce object-level auth: Verify tenant isolation, ownership, and role scoping before data access.
• Contain model output: Treat LLM responses as untrusted. Apply allowlists, validation, and capability scoping.
• Demand negative tests: Require explicit boundary violation tests for all security-sensitive changes.
• Structure review findings: Provide code path, impact, and remediation direction. Eliminate vague warnings.
• Gate CI on policy: Integrate static analysis and invariant checks to catch missing guards before merge.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
UI-only or copy changes	Standard review + linting	Low security surface; functional correctness dominates	Minimal
New API endpoint or data mutation	Policy guard + negative tests + blast radius mapping	High risk of auth bypass and data leakage	Moderate
AI agent or tool integration	Deterministic router + allowlist + credential scoping	Model output requires strict containment	High
Webhook or external integration	Signature verification + replay protection + input validation	External systems introduce untrusted data flows	Moderate
Database migration or schema change	Rollback plan + data isolation check + tenant scoping	Structural changes can bypass existing guards	High

Configuration Template

// policy-gateway.ts
import { Request, Response, NextFunction } from 'express';
import { RequestContext } from './auth/context';
import { TenantResourceGuard } from './guards/tenant-guard';
import { PolicyViolationError } from './errors/policy';

export function requireTenantAccess(resourceParam: string) {
  return async (req: Request, res: Response, next: NextFunction) => {
    try {
      const context = req.context as RequestContext;
      const resourceId = req.params[resourceParam] || req.body[resourceParam];
      
      const guard = new TenantResourceGuard(context);
      await guard.assertAccess(resourceId);
      
      next();
    } catch (err) {
      if (err instanceof PolicyViolationError) {
        return res.status(403).json({ error: 'Access denied' });
      }
      next(err);
    }
  };
}

// usage in route definition
app.get('/invoices/:invoiceId', requireTenantAccess('invoiceId'), async (req, res) => {
  // Safe to proceed; tenant isolation verified
  const invoice = await invoiceRepo.findById(req.params.invoiceId);
  res.json(invoice);
});

Quick Start Guide

1. Audit current PRs: Identify recent AI-generated changes touching auth, data, or external integrations. Map their blast radius and input boundaries.
2. Inject policy guards: Wrap data access endpoints with tenant/object-level authorization checks. Centralize policy logic in reusable guards.
3. Enforce negative testing: Update test suites to include cross-tenant access denial, invalid signature rejection, and tool allowlist enforcement.
4. Gate CI pipeline: Add static analysis rules that flag missing authorization checks, unvalidated external inputs, and absent negative tests. Block merges until invariants are satisfied.
5. Standardize review feedback: Replace vague security comments with structured findings containing code path, impact, and remediation steps. Track resolution velocity to measure protocol effectiveness.