Building an agent-ready website: how to make your site readable for ChatGPT, Perplexity and autonomous agents

endpoints introduces unacceptable security risks. 3. Context Window Optimization: Assets like llms.txt must be concise to fit within agent context windows without consuming excessive tokens. 4. Deterministic Extraction: JSON-LD and OpenAPI specs provide structured data that agents can parse reliably, unlike HTML which requires fragile extraction logic.

Layer 1: The Manifest (llms.txt)

The llms.txt file acts as a canonical introduction for agents. It describes your service, lists key URLs, outlines policies, and defines operational boundaries. This file should be under 10KB to ensure efficient consumption.

Implementation Strategy: Generate llms.txt dynamically from your service catalog and policy configuration. Include a "What We Do Not Do" section to explicitly frame trust boundaries.

Code Example: Manifest Generator

// lib/manifest/generator.ts

export interface ManifestConfig {
  name: string;
  description: string;
  catalogUrls: string[];
  policies: string[];
  exclusions: string[];
  apiSpecUrl: string;
  skillsIndexUrl: string;
  contactEmail: string;
}

export function generateLlmsManifest(config: ManifestConfig): string {
  const lines: string[] = [
    `# ${config.name}`,
    `> ${config.description}`,
    '',
    '## Catalog',
    ...config.catalogUrls.map(url => `- ${url}`),
    '',
    '## Policies',
    ...config.policies.map(p => `- ${p}`),
    '',
    '## Operational Boundaries',
    ...config.exclusions.map(e => `- ${e}`),
    '',
    '## Machine Interfaces',
    `- API Specification: ${config.apiSpecUrl}`,
    `- Agent Skills Index: ${config.skillsIndexUrl}`,
    '',
    `## Contact`,
    config.contactEmail,
  ];

  return lines.join('\n');
}

Rationale:

• Exclusions Block: Explicitly stating limitations (e.g., "No password requests," "No review manipulation") signals integrity. Agents weight these exclusions heavily when evaluating source reliability.
• Dynamic Generation: Ensures the manifest always reflects the current state of the service catalog and policies.

Layer 2: OpenAPI 3.1 Specification

The OpenAPI specification provides a formal contract for agent interactions. By exposing a read-only API spec, you enable agents to safely query your system for data verification, health checks, and public information retrieval.

Implementation Strategy: Define a registry of safe endpoints and generate the OpenAPI 3.1 spec programmatically. Exclude all administrative and mutation endpoints.

Code Example: Safe Endpoint Registry

// lib/api/safe-endpoints.ts

import { OpenAPIV3 } from 'openapi-types';

export type HttpMethod = 'GET' | 'POST' | 'PUT' | 'DELETE' | 'PATCH';

export interface SafeEndpoint {
  path: string;
  method: HttpMethod;
  summary: string;
  readOnly: boolean;
  description?: string;
}

const SAFE_ENDPOINTS: SafeEndpoint[] = [
  {
    path: '/api/health',
    method: 'GET',
    summary: 'System Health Check',
    readOnly: true,
    description: 'Returns current system status.',
  },
  {
    path: '/api/target-lookup',
    method: 'POST',
    summary: 'Verify Public Target',
    readOnly: true,
    description: 'Validates if a public handle exists on supported platforms.',
  },
  {
    path: '/api/orders/{id}',
    method: 'GET',
    summary: 'Public Order Tracking',
    readOnly: true,
    description: 'Retrieves status of a public order by ID.',
  },
];

export function generateOpenApiSpec(endpoints: SafeEndpoint[]): OpenAPIV3.Document {
  const paths: OpenAPIV3.PathsObject = {};

  endpoints.forEach(ep => {
    if (!paths[ep.path]) paths[ep.path] = {};
    paths[ep.path][ep.method.toLowerCase()] = {
      summary: ep.summary,
      description: ep.description,
      responses: {
        '200': { description: 'Successful response' },
      },
    };
  });

  return {
    openapi: '3.1.0',
    info: {
      title: 'Public Agent API',
      version: '1.0.0',
      description: 'Read-only endpoints for autonomous agent interaction.',
    },
    paths,
  };
}

Rationale:

• Read-Only Enforcement: The readOnly flag ensures only safe endpoints are exposed. This prevents agents from accidentally or maliciously triggering state changes.
• Typed Clients: Agents can generate typed clients directly from the spec, reducing integration friction.

Layer 3: Agent Skills via /.well-known

Agent skills are structured declarations of capabilities that agents can discover and utilize. Hosted under /.well-known/agent-skills/, these skills describe specific tasks the site supports, such as reading a catalog or validating input formats.

Implementation Strategy: Create an index file listing available skills, each pointing to a detailed SKILL.md file. Include a SHA-256 digest for integrity verification.

Code Example: Skill Registry

// lib/skills/registry.ts

import crypto from 'crypto';

export interface SkillDefinition {
  slug: string;
  name: string;
  type: 'read' | 'write'; // Enforce read-only in practice
  url: string;
  digest: string;
}

export function computeDigest(content: string): string {
  return `sha256-${crypto.createHash('sha256').update(content).digest('hex')}`;
}

export function generateSkillIndex(skills: SkillDefinition[]): object {
  return {
    skills: skills.map(s => ({
      slug: s.slug,
      name: s.name,
      type: s.type,
      url: s.url,
      digest: s.digest,
    })),
  };
}

// Usage
const skills: SkillDefinition[] = [
  {
    slug: 'read-catalog',
    name: 'Read Service Catalog',
    type: 'read',
    url: '/.well-known/agent-skills/read-catalog/SKILL.md',
    digest: computeDigest('...skill content...'),
  },
  {
    slug: 'validate-target',
    name: 'Validate Public Target',
    type: 'read',
    url: '/.well-known/agent-skills/validate-target/SKILL.md',
    digest: computeDigest('...skill content...'),
  },
];

Rationale:

• Integrity Verification: The SHA-256 digest allows agents to verify that the skill definition hasn't been tampered with.
• Capability Discovery: Skills provide a standardized way for agents to understand what actions are supported, beyond just API endpoints.

Layer 4: Deep JSON-LD Structured Data

Structured data provides semantic context that both search engines and LLMs can consume. While basic schema is common, agent-readiness requires depth, including Product, Offer, FAQPage, and BreadcrumbList types.

Implementation Strategy: Generate JSON-LD blocks from your data models. Ensure nested structures accurately reflect relationships between entities.

Code Example: Schema Composer

// lib/schema/composer.ts

export interface ProductSchema {
  name: string;
  description: string;
  price: number;
  currency: string;
  availability: string;
  faq?: { question: string; answer: string }[];
}

export function composeProductSchema(product: ProductSchema): object {
  const schema: any = {
    '@context': 'https://schema.org',
    '@type': 'Product',
    name: product.name,
    description: product.description,
    offers: {
      '@type': 'Offer',
      price: product.price,
      priceCurrency: product.currency,
      availability: product.availability,
    },
  };

  if (product.faq && product.faq.length > 0) {
    schema.mainEntity = {
      '@type': 'FAQPage',
      mainEntity: product.faq.map(f => ({
        '@type': 'Question',
        name: f.question,
        acceptedAnswer: {
          '@type': 'Answer',
          text: f.answer,
        },
      })),
    };
  }

  return schema;
}

Rationale:

• Deterministic Extraction: JSON-LD allows agents to extract structured facts without parsing HTML, reducing errors.
• Rich Context: Nested schemas like FAQPage provide direct answers to common queries, increasing the likelihood of citation.

Pitfall Guide

Implementing an agent-ready architecture requires discipline. The following pitfalls are common in production environments and can undermine the effectiveness of your machine-readable surfaces.

1. Context Window Overflow

   • Explanation: llms.txt exceeds the optimal size limit, causing agents to truncate content or ignore the file.
   • Fix: Enforce a strict size limit (e.g., <10KB). Summarize long lists and prioritize high-value URLs.

2. Configuration Drift

   • Explanation: Machine-readable assets are manually updated, leading to inconsistencies with the actual service state.
   • Fix: Generate all assets from a single source of truth. Use build-time or runtime generation pipelines.

3. Mutation Exposure

   • Explanation: OpenAPI spec includes endpoints that modify state, posing security risks.
   • Fix: Filter endpoints by readOnly status. Explicitly exclude administrative and payment endpoints from the public spec.

4. Trust Ambiguity

   • Explanation: Missing operational boundaries in llms.txt leaves agents uncertain about safe interactions.
   • Fix: Include a "What We Do Not Do" section with explicit exclusions. This frames trust and reduces hallucination.

5. Shallow Schema

   • Explanation: Only basic schema types like Organization are used, missing opportunities for rich context.
   • Fix: Implement deep schema including Product, Offer, FAQPage, and BreadcrumbList where relevant.

6. Missing Integrity Checks

   • Explanation: Agent skills lack digest verification, making them vulnerable to tampering.
   • Fix: Compute and include SHA-256 digests for all skill definitions in the index file.

7. Non-Standard Paths

   • Explanation: Assets are hosted at non-standard URLs, making them hard for agents to discover.
   • Fix: Adhere to conventions: llms.txt at root, OpenAPI at /openapi.json, skills at /.well-known/agent-skills/.

Production Bundle

Action Checklist

• Audit Data Models: Identify the single source of truth for services, policies, and endpoints.
• Implement Manifest Generator: Create a function to generate llms.txt from your data models, including exclusions.
• Define Safe Endpoints: Register read-only API endpoints and generate an OpenAPI 3.1 spec.
• Register Agent Skills: Create skill definitions with digests and host them at /.well-known/agent-skills/.
• Deepen Structured Data: Add Product, Offer, and FAQPage schema to relevant pages.
• Validate Assets: Test llms.txt size, OpenAPI spec validity, and skill integrity.
• Monitor Citations: Track how often your site is cited by LLMs and agents to measure impact.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Static Content Site	Focus on llms.txt and JSON-LD	Agents need context and facts; no API interaction required.	Low
SaaS with Public API	Add OpenAPI 3.1 and Agent Skills	Enables agents to safely query and verify data.	Medium
E-commerce Platform	Deep JSON-LD + llms.txt exclusions	Builds trust for transactions; agents need clear policies.	Medium
High-Security System	Strict read-only OpenAPI + Skills	Minimizes risk while allowing agent discovery.	High

Configuration Template

// config/agent-ready.config.ts

export const agentConfig = {
  manifest: {
    name: 'YourService',
    description: 'Concise description of your service.',
    exclusions: [
      'No password requests',
      'No review manipulation',
      'No mass messaging',
    ],
  },
  api: {
    readOnly: true,
    endpoints: [
      '/api/health',
      '/api/catalog',
      '/api/verify',
    ],
  },
  skills: [
    { slug: 'read-catalog', type: 'read' },
    { slug: 'validate-input', type: 'read' },
  ],
  schema: {
    types: ['Product', 'Offer', 'FAQPage', 'BreadcrumbList'],
  },
};

Quick Start Guide

1. Define Your SSOT: Create a central configuration file for services, policies, and endpoints.
2. Generate llms.txt: Implement a generator function and expose it at the root path.
3. Publish OpenAPI: Filter endpoints for read-only access and serve the spec at /openapi.json.
4. Register Skills: Create skill definitions with digests and host the index at /.well-known/agent-skills/.
5. Deploy and Validate: Deploy changes and verify assets using LLM testing tools.

By engineering your web assets for machine discovery, you position your site as a trusted, accessible source in the evolving AI ecosystem. This architecture not only improves visibility but also establishes a foundation for safe, structured interactions with autonomous agents.