tsc-correctness != runtime-correctness

The Runtime Trap: Why Type-Correct AI Fixes Break Production Systems

Current Situation Analysis

Modern development workflows increasingly rely on AI-driven code repair to resolve TypeScript compiler errors. When tsc emits an error, the standard expectation is that an AI assistant will apply the compiler's suggested fix, resulting in a green build and a functional application. However, a critical failure mode has emerged where AI-generated repairs satisfy the type checker but introduce runtime crashes or security vulnerabilities.

This problem is often misattributed to model incompetence. In reality, it stems from a structural misalignment between static type analysis and dynamic runtime semantics. The failure occurs at the intersection of three factors:

1. Library Breaking Changes: Package maintainers frequently introduce major version updates that alter import semantics without changing type signatures in a way that breaks compilation. For example, vite-plugin-svgr v4 changed the default import of an SVG file from a React component to an asset URL string. The named export ReactComponent was removed.
2. TypeScript Limitations: The TypeScript compiler reasons exclusively about types. It cannot infer runtime behavior introduced by build plugins or bundlers. When vite-plugin-svgr v4 removes a named export, tsc suggests using the default import because it exists in the ambient module declaration. This suggestion resolves the type error but changes the runtime value from a component to a string.
3. LLM Training Distribution: Large language models are trained on historical codebases. If a library's API changed in a recent major version, the model's training data likely reflects the older behavior. When presented with a tsc error and a compiler hint, the model aligns with the hint and its training distribution, both of which may be outdated regarding the current runtime contract.

The result is a "silent break": the code compiles, the CI pipeline passes, but the application crashes upon rendering or exhibits incorrect behavior. This gap is not theoretical; benchmarks on AI repair tools show that without explicit version-aware context, success rates for fixing these specific patterns drop to zero, even when the correct solution is well-documented.

WOW Moment: Key Findings

The most significant insight from analyzing AI repair failures is that context placement determines success more than context volume. Simply providing the model with documentation or migration guides is insufficient if the information is buried in the prompt. The model must be explicitly instructed to prioritize migration logic over compiler hints.

Data from controlled benchmarks demonstrates that injecting migration rules into the system headline (the task description) drastically outperforms standard context injection.

Strategy	Type Compliance	Runtime Integrity	Fix Accuracy
Standard LLM + TS Hint	High	Low	0%
LLM + Mid-Prompt Context	High	Medium	33%
LLM + Headline Injection	High	High	100%

Data derived from benchmarking vite-plugin-svgr v4 migration patterns.

Why this matters: The "Headline Injection" approach works because of how transformer models attend to instructions. The system headline defines the primary task frame. When the headline explicitly states a library migration requirement, the model treats this as a constraint that overrides secondary signals, such as tsc quick-fix suggestions. This reframing allows the model to utilize its latent knowledge of the new API (which exists in its training data but is suppressed by the compiler hint) to generate a runtime-correct fix.

This pattern extends beyond SVG imports. Similar failures occur with:

• Next.js v15: params and searchParams became asynchronous Promises. AI tools often fail to add await, causing runtime undefined errors.
• Drizzle ORM: Major versions shifted to parameterized template literals. AI tools may suggest string concatenation, which passes types but breaks query execution.
• Vercel AI SDK: API rewrites in v3/v6 require structural changes that tsc cannot fully guide.

Core Solution

To bridge the gap between type correctness and runtime safety, you must implement a Version-Aware Context Injection system. This architecture scans project dependencies, identifies known breaking changes, and injects targeted instructions into the AI prompt before the model processes the error.

Architecture Decisions

1. Dependency Scanning: The system must read package.json to determine installed versions. This provides the ground truth for which migration rules apply.
2. Migration Registry: A structured database maps package versions to specific prompt instructions. This registry should be extensible and version-range aware.
3. Prompt Assembly: Instructions must be injected into the taskDescription or system headline. This ensures maximum attention weight. Context placed in the middle of the prompt suffers from attention falloff and is often ignored when conflicting with compiler errors.
4. Deterministic Pre-Filtering: Trivial errors (typos, missing imports) should be resolved deterministically without LLM invocation to reduce cost and latency. The LLM should only handle complex semantic fixes.

Implementation Example

The following TypeScript implementation demonstrates a MigrationRegistry and PromptEngine that injects context based on dependency versions.

// registry.ts
export interface MigrationRule {
  packageName: string;
  minVersion: string;
  maxVersion?: string;
  instruction: string;
  severity: 'runtime' | 'security' | 'api';
}

export const KNOWN_MIGRATIONS: MigrationRule[] = [
  {
    packageName: 'vite-plugin-svgr',
    minVersion: '4.0.0',
    instruction: 
      'vite-plugin-svgr v4+ requires the `?react` query suffix to import SVGs as components. ' +
      'Default imports return asset URLs. Use `import Component from "./file.svg?react"`.',
    severity: 'runtime'
  },
  {
    packageName: 'next',
    minVersion: '15.0.0',
    instruction:
      'Next.js v15 changed `params` and `searchParams` to Promises. ' +
      'Ensure all accesses are awaited. Do not treat them as synchronous values.',
    severity: 'runtime'
  },
  {
    packageName: 'bcrypt',
    minVersion: '5.0.0',
    instruction:
      'Security Critical: Never substitute `bcrypt` with `crypto.subtle` or SHA algorithms. ' +
      'Password hashing requires salted, adaptive-cost functions. Maintain `bcrypt` usage.',
    severity: 'security'
  }
];

// engine.ts
import { readFileSync } from 'fs';
import { satisfies } from 'semver';
import { KNOWN_MIGRATIONS, MigrationRule } from './registry';

export class PromptEngine {
  private rules: MigrationRule[];

  constructor(projectRoot: string) {
    const pkgJson = JSON.parse(readFileSync(`${projectRoot}/package.json`, 'utf-8'));
    this.rules = this.resolveRules(pkgJson.dependencies, pkgJson.devDependencies);
  }

  private resolveRules(deps: Record<string, string>, devDeps: Record<string, string>): MigrationRule[] {
    const allDeps = { ...deps, ...devDeps };
    return KNOWN_MIGRATIONS.filter(rule => {
      const installedVersion = allDeps[rule.packageName];
      if (!installedVersion) return false;
      
      const cleanVersion = installedVersion.replace(/^[^0-9]*/, '');
      return satisfies(cleanVersion, `>=${rule.minVersion}${rule.maxVersion ? ` <=${rule.maxVersion}` : ''}`);
    });
  }

  public buildSystemHeadline(): string {
    if (this.rules.length === 0) return '';

    const instructions = this.rules
      .map(r => `- ${r.packageName}: ${r.instruction}`)
      .join('\n');

    return `
### LIBRARY MIGRATION CONTEXT
The following library versions are installed. You MUST apply these migration rules 
overriding any TypeScript compiler suggestions that conflict with runtime semantics.

${instructions}

### TASK
Fix the provided TypeScript error while ensuring runtime correctness and security.
    `.trim();
  }

  public generatePrompt(errorContext: string): string {
    const headline = this.buildSystemHeadline();
    
    // Structure prompt to prioritize headline
    return `
${headline}

### ERROR CONTEXT
${errorContext}

### SOURCE FILE
[File content would be injected here]
    `.trim();
  }
}

Rationale for Design Choices

• Semver Matching: Using semver allows the registry to handle version ranges accurately. This prevents false positives when a project is on an older version that doesn't require the migration.
• Severity Classification: Tagging rules with security or runtime allows the system to prioritize critical instructions. Security rules can be injected with stronger framing to prevent the model from making dangerous substitutions.
• Headline Injection: The buildSystemHeadline method constructs the instruction block that goes into the system prompt's task description. This placement is non-negotiable for overriding tsc hints.
• Extensibility: The KNOWN_MIGRATIONS array can be populated from an external source or community registry, allowing the system to scale as new library updates are discovered.

Pitfall Guide

When implementing AI-driven code repair, avoid these common mistakes that lead to runtime failures or security regressions.

1. The Type-Safety Illusion

   • Explanation: Assuming that a fix which resolves all tsc errors is safe to deploy. TypeScript cannot detect runtime behavior changes, plugin side effects, or semantic drift.
   • Fix: Always pair type-checking with runtime validation. Use integration tests or sandboxed execution to verify fixes before merging.

2. Context Dilution

   • Explanation: Placing migration instructions in the middle of the prompt or in a separate "reference" section. Models often ignore context that conflicts with immediate signals like compiler errors if it's not framed as a primary task.
   • Fix: Inject critical instructions into the system headline or task description. Use explicit framing like "MUST apply these rules overriding compiler suggestions."

3. Version Blindness

   • Explanation: Sending the same prompt regardless of the project's dependency versions. This causes the model to apply outdated fixes or miss necessary migrations.
   • Fix: Implement dependency scanning and version-aware prompt assembly. The system must know exactly which versions are installed.

4. Security-Type Conflicts

   • Explanation: The model may choose a fix that satisfies types but introduces vulnerabilities. For example, switching from bcrypt to crypto.subtle.digest to resolve a missing import, or using dangerouslySetInnerHTML to bypass type errors with HTML strings.
   • Fix: Include explicit security rules in the migration registry. Tag these rules with high severity and instruct the model to reject fixes that compromise security properties.

5. Semantic Substitution

   • Explanation: The model substitutes semantically different APIs that share similar type signatures. For instance, replacing a parameterized query builder with string concatenation because the types align but the execution model differs.
   • Fix: Define rules that warn against specific substitutions. Emphasize semantic equivalence over type compatibility in the prompt instructions.

6. Plugin Opacity

   • Explanation: Ignoring the impact of build tools and plugins. TypeScript sees ambient declarations but cannot infer how a bundler transforms imports.
   • Fix: Maintain a registry of plugin-specific behaviors. Include rules for common plugins like vite-plugin-svgr, babel-plugin-macros, or custom webpack loaders.

7. Over-Reliance on Quick Fixes

   • Explanation: Trusting tsc Quick Fix suggestions as authoritative. These suggestions are heuristic and may not account for library migrations or runtime constraints.
   • Fix: Treat compiler suggestions as hints, not directives. The AI should evaluate suggestions against the migration registry and runtime requirements before applying them.

Production Bundle

Action Checklist

• Scan Dependencies: Implement a mechanism to read package.json and extract installed versions of all dependencies.
• Build Migration Registry: Create a structured registry of known breaking changes, version ranges, and prompt instructions.
• Inject Headline Context: Ensure migration rules are injected into the system prompt's task description, not buried in context.
• Add Security Rules: Explicitly define rules to prevent security regressions, such as algorithm substitutions or XSS vectors.
• Validate Runtime: Run integration tests or sandboxed execution to verify that AI fixes work at runtime, not just in the type checker.
• Monitor Failures: Track cases where AI fixes fail or introduce regressions to identify new patterns for the registry.
• Update Registry: Regularly update the migration registry as new library versions are released.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Frequent Library Updates	Version-Aware Registry Injection	Provides accurate, low-latency fixes for known breaking changes without retraining.	Low
Custom Internal Libraries	Fine-Tuning or RAG	Registry may not cover internal APIs. Fine-tuning on internal codebase or RAG on internal docs is better.	High
One-Off Fixes	Standard LLM with RAG	No need for registry setup. RAG can retrieve relevant documentation on demand.	Medium
Security-Critical Code	Strict Rules + Human Review	AI should never autonomously fix security-sensitive code. Rules must enforce constraints, and humans must verify.	Medium
Legacy Codebases	Hybrid Approach	Combine registry injection for modern libraries with RAG for legacy patterns.	Medium

Configuration Template

Use this JSON structure to define migration rules for your registry. This template supports version ranges, severity levels, and detailed instructions.

{
  "registryVersion": "1.0.0",
  "rules": [
    {
      "id": "vite-plugin-svgr-v4",
      "packageName": "vite-plugin-svgr",
      "versionRange": ">=4.0.0",
      "severity": "runtime",
      "instruction": "vite-plugin-svgr v4+ requires the `?react` query suffix for component imports. Default imports return URLs. Use `import Component from './file.svg?react'`.",
      "antiPattern": "Do not use default imports for SVG components without the `?react` suffix."
    },
    {
      "id": "next-v15-async-params",
      "packageName": "next",
      "versionRange": ">=15.0.0",
      "severity": "runtime",
      "instruction": "Next.js v15 made `params` and `searchParams` asynchronous. All accesses must be awaited. Do not treat them as synchronous values.",
      "antiPattern": "Avoid accessing `params` or `searchParams` without `await`."
    },
    {
      "id": "bcrypt-security",
      "packageName": "bcrypt",
      "versionRange": ">=5.0.0",
      "severity": "security",
      "instruction": "Security Critical: Never replace `bcrypt` with `crypto` or SHA algorithms. Password hashing requires salted, adaptive-cost functions.",
      "antiPattern": "Do not substitute `bcrypt` with `crypto.subtle` or similar APIs."
    }
  ]
}

Quick Start Guide

1. Initialize Registry: Create a migrations.json file using the configuration template. Add rules for the libraries in your stack.
2. Implement Scanner: Write a script to parse package.json and match installed versions against the registry using semver.
3. Hook into AI Workflow: Integrate the PromptEngine into your CI pipeline or editor extension. Ensure it generates prompts with headline injection before sending to the LLM.
4. Test Fixes: Run the AI repair on a sample error. Verify that the fix respects the migration rules and passes runtime checks.
5. Deploy: Roll out the system to your development environment. Monitor for new failure patterns and update the registry accordingly.