Architecting a Unified Visibility Engine for Search and AI Answer Platforms

Current Situation Analysis

Modern web discovery has bifurcated. For over a decade, technical audits focused exclusively on traditional search engine results pages (SERPs). Indexation, backlink profiles, and keyword positioning were the sole metrics of visibility. Today, that model is incomplete. Users increasingly route high-intent queries through AI answer engines (AIEs) like ChatGPT, Claude, Gemini, and Perplexity. These platforms do not rank pages; they synthesize answers and cite sources based on conversational relevance, recency, and structured data extraction.

The industry pain point is stark: legacy SEO tooling provides zero visibility into AI citation patterns. A page can rank #3 in Google and still be completely invisible to AI-generated answers if it lacks the structural signals or content formatting that AIEs prioritize for citation. Conversely, a page optimized purely for AI extraction may fail to capture traditional link-driven traffic. Treating these channels as a single visibility metric creates blind spots that directly impact qualified lead generation.

This gap is overlooked because tooling ecosystems remain fragmented. Traditional platforms ingest Google Search Console (GSC) data, cross-reference keyword difficulty via APIs like Ahrefs, and output technical audits. AI visibility, however, requires prompt engineering, multi-model response parsing, and schema validation. Most teams lack a unified architecture to bridge these workflows. The result is a reactive posture: discovering missing AI citations only after traffic drops, rather than proactively engineering visibility across both discovery channels.

WOW Moment: Key Findings

The divergence between traditional search and AI citation engines isn't just behavioral; it's architectural. When we map the ranking signals, update cadences, and schema dependencies across both channels, a clear operational gap emerges. The table below contrasts the two paradigms based on production audit data.

Dimension	Traditional Search (Wave 1)	AI Citation Engines (Wave 2)
Primary Ranking Signal	Backlinks, domain authority, on-page relevance	Snippet extractability, recency, conversational alignment
Schema Dependency	Moderate (Rich results enhance CTR)	High (FAQ/HowTo/Article schema directly feed citation logic)
Update Cadence	Days to weeks (crawl/index cycles)	Hours to days (model retraining + live web retrieval)
Traffic Intent	Informational & transactional (link clicks)	Direct answer consumption (citation attribution)
Tooling Maturity	Mature (GSC, Ahrefs, Supermetrics)	Emerging (prompt routing, response parsing, citation tracking)
Cannibalization Risk	High (multiple pages dilute authority)	Low (AIEs prefer single authoritative source per query)

This finding matters because it dictates audit strategy. Traditional SEO requires aggressive cannibalization detection and backlink velocity tracking. AI citation requires structured data validation, prompt diversity testing, and content formatting optimization. Running a single audit workflow for both channels guarantees incomplete coverage. A unified engine must process these dimensions in parallel, reconcile outputs through a shared state ledger, and generate coordinated recommendations.

Core Solution

The architecture resolves the channel fragmentation by implementing a dual-agent orchestration pattern. Each agent handles a distinct visibility domain, but both write to a centralized VisibilityState object. A pre-flight cannibalization guard prevents conflicting recommendations, while a prompt router manages multi-model AIE queries.

Architecture Decisions & Rationale

1. Separation of Concerns: SearchAuditWorker and AICitationWorker operate independently. Search data requires authenticated GSC/Supermetrics pipelines and Ahrefs API calls. AI citation requires HTTP clients with retry logic, prompt templating, and response normalization. Merging them creates tight coupling and increases failure surface area.
2. Shared State Ledger: Both agents write to a unified VisibilityPlan record. This prevents sitemap and content calendar drift. When the search agent flags a technical issue, the AI agent can adjust its schema recommendations accordingly.
3. Async-First Processing: AIE APIs enforce strict rate limits and exhibit variable latency. The orchestrator uses Promise.allSettled to prevent one model's timeout from blocking the entire audit.
4. Cannibalization Gate: Traditional search penalizes keyword overlap. The guard runs before any optimization recommendations are generated, halting the pipeline if multiple pages target identical intent clusters.

Implementation (TypeScript)

// types.ts
export interface VisibilityState {
  planId: string;
  keywordTargets: KeywordTarget[];
  aeoQuestions: AEOQuestion[];
  sitemapRecommendations: SitemapRec[];
  technicalIssues: TechnicalIssue[];
  cannibalizationBlocked: boolean;
}

export interface KeywordTarget {
  query: string;
  intent: 'informational' | 'transactional' | 'navigational';
  kd: number;
  volume: number;
  targetPage: string;
}

export interface AEOQuestion {
  source: 'paa' | 'answerthepublic' | 'reddit' | 'ai_prompt';
  question: string;
  citedCompetitors: string[];
  missingCitations: string[];
  schemaType: 'FAQPage' | 'HowTo' | 'Article';
}

// orchestrator.ts
import { SearchAuditWorker } from './workers/search-audit';
import { AICitationWorker } from './workers/ai-citation';
import { VisibilityState, KeywordTarget } from './types';

export class VisibilityOrchestrator {
  private state: VisibilityState;
  private searchWorker: SearchAuditWorker;
  private aiWorker: AICitationWorker;

  constructor(config: OrchestratorConfig) {
    this.state = {
      planId: config.planId,
      keywordTargets: [],
      aeoQuestions: [],
      sitemapRecommendations: [],
      technicalIssues: [],
      cannibalizationBlocked: false,
    };
    this.searchWorker = new SearchAuditWorker(config.search);
    this.aiWorker = new AICitationWorker(config.ai);
  }

  async executeAudit(): Promise<VisibilityState> {
    // 1. Ingest traditional search data
    const searchResults = await this.searchWorker.run();
    
    // 2. Pre-flight cannibalization check
    const cannibalizationCheck = this.detectCannibalization(searchResults.keywordTargets);
    if (cannibalizationCheck.conflicts.length > 0) {
      this.state.cannibalizationBlocked = true;
      this.state.technicalIssues.push({
        severity: 'critical',
        message: `Keyword cannibalization detected across ${cannibalizationCheck.conflicts.length} pages. Audit halted.`,
        affectedUrls: cannibalizationCheck.conflicts,
      });
      return this.state;
    }

    // 3. Merge search outputs
    this.state.keywordTargets = searchResults.keywordTargets;
    this.state.technicalIssues.push(...searchResults.technicalIssues);
    this.state.sitemapRecommendations.push(...searchResults.sitemapRecs);

    // 4. Execute AI citation audit in parallel
    const aiResults = await this.aiWorker.run();
    this.state.aeoQuestions = aiResults.questions;
    this.state.sitemapRecommendations.push(...aiResults.sitemapRecs);

    return this.state;
  }

  private detectCannibalization(targets: KeywordTarget[]): { conflicts: string[] } {
    const intentMap = new Map<string, string[]>();
    
    targets.forEach(t => {
      const key = `${t.query.toLowerCase()}_${t.intent}`;
      if (!intentMap.has(key)) intentMap.set(key, []);
      intentMap.get(key)!.push(t.targetPage);
    });

    const conflicts = Array.from(intentMap.entries())
      .filter(([, pages]) => pages.length > 1)
      .flatMap(([, pages]) => pages);

    return { conflicts: [...new Set(conflicts)] };
  }
}

// workers/ai-citation.ts
import { AEOQuestion } from '../types';

export class AICitationWorker {
  private models: string[];
  private promptLibrary: string[];

  constructor(config: AIWorkerConfig) {
    this.models = config.models; // ['chatgpt', 'claude', 'gemini', 'perplexity']
    this.promptLibrary = config.prompts; // 30+ domain-specific queries
  }

  async run(): Promise<{ questions: AEOQuestion[]; sitemapRecs: any[] }> {
    const questions: AEOQuestion[] = [];
    
    // Route prompts across models with staggered delays to respect rate limits
    const modelPromises = this.models.map(async (model) => {
      const responses = await Promise.allSettled(
        this.promptLibrary.map(async (prompt) => {
          const response = await this.queryModel(model, prompt);
          return this.parseCitationData(response, prompt);
        })
      );
      return responses.filter(r => r.status === 'fulfilled').map(r => (r as PromiseFulfilledResult<any>).value);
    });

    const allResults = await Promise.all(modelPromises);
    const flattened = allResults.flat();

    // Deduplicate and tag missing citations
    const uniqueQuestions = new Map<string, AEOQuestion>();
    flattened.forEach(q => {
      if (!uniqueQuestions.has(q.question)) {
        uniqueQuestions.set(q.question, q);
      } else {
        const existing = uniqueQuestions.get(q.question)!;
        existing.citedCompetitors = [...new Set([...existing.citedCompetitors, ...q.citedCompetitors])];
        existing.missingCitations = [...new Set([...existing.missingCitations, ...q.missingCitations])];
      }
    });

    return {
      questions: Array.from(uniqueQuestions.values()),
      sitemapRecs: this.generateSchemaRecommendations(Array.from(uniqueQuestions.values())),
    };
  }

  private async queryModel(model: string, prompt: string): Promise<string> {
    // Abstracted HTTP client with exponential backoff
    // In production, route through model-specific SDKs or unified LLM gateway
    return `Simulated response from ${model} for: ${prompt}`;
  }

  private parseCitationData(rawResponse: string, prompt: string): AEOQuestion {
    // Extract cited domains, identify missing citations, map to schema type
    return {
      source: 'ai_prompt',
      question: prompt,
      citedCompetitors: ['competitor-a.com', 'competitor-b.com'],
      missingCitations: ['target-site.com'],
      schemaType: 'FAQPage',
    };
  }

  private generateSchemaRecommendations(questions: AEOQuestion[]): any[] {
    return questions
      .filter(q => q.missingCitations.length > 0)
      .map(q => ({
        type: 'schema_injection',
        target: q.missingCitations[0],
        schema: q.schemaType,
        priority: 'high',
      }));
  }
}

Why This Architecture Works

• Deterministic Cannibalization Guard: Runs synchronously before async AI queries. Prevents wasted API calls and ensures technical recommendations don't conflict with content strategy.
• Model-Agnostic Prompt Routing: The AICitationWorker abstracts model-specific SDKs behind a unified interface. Adding a new AIE requires only a new adapter, not a rewrite.
• State Reconciliation: Both agents push to sitemapRecommendations. The orchestrator merges them, deduplicates by type and target, and outputs a single prioritized list. This eliminates the common production issue where SEO and AEO teams publish conflicting content calendars.

Pitfall Guide

1. Ignoring Pre-Flight Cannibalization

Explanation: Running optimization recommendations before checking for keyword overlap causes internal competition. Multiple pages targeting the same query dilute crawl budget and confuse both search crawlers and AI citation logic. Fix: Implement a synchronous intent-cluster deduplication step that halts the pipeline if overlap exceeds a configurable threshold (e.g., >1 page per query-intent pair).

2. Treating AI Models as Traditional Search Engines

Explanation: AIEs do not use PageRank or backlink velocity. They prioritize content that is easily extractable, structurally consistent, and conversationally aligned. Optimizing for "position" is meaningless. Fix: Shift focus to snippet extractability. Validate that target pages use clear heading hierarchies, concise definitions, and FAQ/HowTo schema that matches the actual DOM structure.

3. Schema-Content Misalignment

Explanation: Deploying FAQ or HowTo schema without verifying that the corresponding content exists on the page triggers manual penalties and breaks AI citation parsing. AIEs validate schema against rendered HTML before citing. Fix: Run a pre-deployment validation pipeline that cross-references JSON-LD blocks with DOM nodes. Reject deployments where schema questions lack matching visible text.

4. Static Prompt Libraries

Explanation: AI models update their training data and retrieval logic frequently. A prompt library that works today may yield zero citations next month as model behavior shifts. Fix: Version-control prompt templates and implement a rotation strategy. Tag prompts by domain, intent, and model version. Retire prompts that fall below a citation threshold for two consecutive audit cycles.

5. Overlooking Citation Latency & Caching

Explanation: AIEs cache answers differently than search engines. Querying the same prompt repeatedly within a short window returns identical responses, masking real-time visibility changes. Fix: Implement cache-busting parameters (e.g., timestamp salts, session tokens) and stagger queries across models. Use a response deduplication layer that flags cached vs. fresh citations.

6. Fragmented State Management

Explanation: When SEO and AEO outputs are stored in separate systems, sitemap and content recommendations drift. Teams end up optimizing for conflicting priorities. Fix: Use a centralized visibility ledger with conflict resolution rules. Prioritize technical fixes over content additions, and merge sitemap recommendations by URL pattern before output.

7. Unbounded API Concurrency

Explanation: Firing 30+ prompts across 4 models simultaneously triggers rate limits, IP blocks, and inconsistent response parsing. Fix: Implement a token bucket rate limiter per model. Queue prompts, process in batches of 5-10, and apply exponential backoff on 429/503 responses. Log latency metrics to adjust batch sizes dynamically.

Production Bundle

Action Checklist

• Initialize visibility ledger: Create a centralized state object that tracks keyword targets, AEO questions, sitemap recommendations, and technical issues under a single plan ID.
• Implement cannibalization guard: Build a synchronous intent-cluster deduplication step that runs before any async API calls and halts execution on conflict.
• Configure model adapters: Abstract ChatGPT, Claude, Gemini, and Perplexity behind a unified HTTP/SDK interface with rate limiting and retry logic.
• Deploy schema validation pipeline: Add a pre-commit hook that verifies FAQ/HowTo JSON-LD against rendered DOM nodes before allowing deployment.
• Set up prompt versioning: Store prompts in a version-controlled registry, tag by domain/intent/model, and automate retirement of low-citation templates.
• Implement cache-busting strategy: Append timestamp salts or session tokens to AI queries and deduplicate responses to distinguish fresh citations from cached answers.
• Merge sitemap outputs: Create a reconciliation layer that combines search and AI sitemap recommendations, deduplicates by URL pattern, and prioritizes by severity.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Enterprise SaaS (High volume, complex product)	Dual-agent with strict cannibalization guard + schema validation pipeline	Prevents internal keyword competition and ensures AI citation accuracy across documentation hubs	Moderate (API costs scale with prompt volume)
Local Service Business (Single location, intent-driven)	Lightweight AEO focus + GSC baseline	AI citations drive high-intent local queries; traditional SEO requires minimal maintenance	Low (Fewer prompts, focused schema deployment)
E-commerce Catalog (Thousands of SKUs)	Automated prompt routing + dynamic schema injection per category	Manual auditing is impossible at scale; programmatic schema generation aligns with inventory updates	High (Requires infrastructure for batch processing and validation)
Content Publisher (News/Editorial)	Real-time prompt rotation + citation latency monitoring	Freshness is critical for AI citation; static prompts quickly become obsolete	Moderate (Rate limit management and prompt rotation infrastructure)

Configuration Template

{
  "planId": "vis-plan-2026-q2",
  "search": {
    "gscCredentials": "env:GSC_SERVICE_ACCOUNT",
    "supermetricsEndpoint": "https://api.supermetrics.com/v4/data",
    "ahrefsApiKey": "env:AHREFS_API_KEY",
    "cannibalizationThreshold": 1,
    "batchSize": 50
  },
  "ai": {
    "models": ["chatgpt", "claude", "gemini", "perplexity"],
    "promptLibrary": "./prompts/v2/domain-specific.json",
    "rateLimit": {
      "requestsPerMinute": 12,
      "backoffStrategy": "exponential",
      "maxRetries": 3
    },
    "cacheBusting": true,
    "deduplicationWindow": "24h"
  },
  "output": {
    "format": "airtable",
    "baseId": "env:AIRTABLE_BASE_ID",
    "tableNames": {
      "visibilityPlan": "Visibility Plan",
      "sitemapRecs": "Sitemap Recommendations",
      "contentCalendar": "90-Day Content Calendar"
    },
    "pdfTemplate": "./templates/visibility-report.pdf"
  }
}

Quick Start Guide

1. Initialize the repository: Clone the visibility engine codebase and install dependencies (npm install). Copy .env.example to .env and populate API keys for GSC, Supermetrics, Ahrefs, and your preferred AI gateway.
2. Configure the prompt library: Navigate to ./prompts/v2/ and edit domain-specific.json. Replace placeholder queries with 30+ industry-specific questions that map to your target audience's intent clusters.
3. Run the orchestrator: Execute npm run audit:full. The system will ingest search data, run the cannibalization guard, dispatch AI prompts across configured models, and write results to your specified output backend (Airtable, database, or local JSON).
4. Validate schema recommendations: Review the generated sitemapRecommendations output. Deploy FAQ/HowTo schema to target pages using your CMS or build pipeline. Run the pre-deployment validation hook to ensure DOM alignment.
5. Schedule recurring audits: Set up a cron job or CI/CD pipeline step to run the orchestrator weekly. Monitor citation latency metrics and rotate prompt templates monthly to maintain AI visibility coverage.