Web Documentation Delivery Spec

Spec Contents #

• Category 1: Content Discoverability: Checks for llms.txt existence, validity, size, link quality, and directives that point agents to it.
• Category 2: Markdown Availability: Checks for .md URL variants and content negotiation via Accept headers.
• Category 3: Page Size and Truncation Risk: Checks for rendering strategy, markdown/HTML/transfer size, content start position, and single-fetch completeness.
• Category 4: Content Structure: Checks for tabbed content serialization, header quality, code fence validity, link portability, and embedded data.
• Category 5: URL Stability and Redirects: Checks for HTTP status code honesty and redirect behavior.
• Category 6: Observability and Content Health: Checks for llms.txt coverage, markdown/HTML content parity, and cache header hygiene.
• Category 7: Authentication and Access: Checks for auth gating, alternative access paths, and bot-protection interference.
• Interaction Effects: How individual check failures combine into non-linear agent experience degradation.
• Appendices: Known platform truncation limits and notable exclusions.
• Changelog: Version history of the Agent-Friendly Documentation Spec.

Abstract #

Documentation sites are increasingly consumed by coding agents rather than human readers, but most sites are not built for this access pattern. Agents hit truncation limits, get walls of CSS instead of content, can’t follow cross-host redirects, and don’t know about emerging discovery mechanisms like llms.txt. This spec defines 28 checks across 7 categories that evaluate how well a documentation site serves agent consumers. It is grounded in empirical observation of real agent workflows and is intended as a shared standard for documentation teams, tool builders, and platform providers.

Scope #

This spec targets coding agents that fetch documentation during real-time development workflows. These are tools like Claude Code, Cursor, GitHub Copilot, and similar IDE-integrated or CLI-based agents that a developer uses while writing code. The agent fetches a docs page, extracts information, and uses it to complete a task, all in a single session.

This spec does not target:

• Training crawlers (GPTBot, ClaudeBot, etc.) that scrape content for model training. These have different access patterns, different user-agents, and different concerns. See Appendix B.
• Answer engines (Perplexity, Google AI Overviews, ChatGPT search) that retrieve content to generate responses to user queries. These systems have their own retrieval pipelines that may or may not resemble the web fetch pipelines described here.
• RAG pipelines that pre-index documentation into vector stores. These retrieve at query time from an index built ahead of time, so truncation limits and real-time fetch behavior are less relevant. Their ingestion step is a different matter: it fetches from the site like any automated client, and many of this spec’s checks serve it directly. See Serving RAG Ingestion Pipelines.

The findings and checks in this spec are grounded in empirical observation of coding agents. Some recommendations (like providing llms.txt and serving markdown) will benefit other consumers too, but the pass/warn/fail criteria are calibrated for the coding agent use case.

Related Surfaces #

This spec covers the delivery of web-served documentation: whether agents can discover, fetch, and receive content intact. Agent-friendly documentation has at least two other surfaces, deliberately out of scope here and planned as companion specifications as the evidence base for them matures:

• Content composition: what documentation should contain to serve agents well. Factual consistency across pages, structure and density suited to machine consumption, and how representation across a corpus shapes what agents effectively know about a product. These properties require semantic evaluation rather than the mechanical verification this spec’s checks are built on.
• Repository-local documentation: docs serving a coding agent that works inside a repository (README, docs/ directories, agent-facing instruction files). The agent discovers content with local tools (grep, find, directory listings) and consumes it through file reads, so almost none of the web spec’s checks transfer: there is no fetch pipeline, no truncation-by-summarizer, and no llms.txt; greppability and file layout do the work that discovery checks do on the web.

Keeping these separate protects what makes each credible: this spec’s checks are mechanically verifiable against a live site, and that property should not be diluted as adjacent guidance develops.

Background #

Agents don’t use docs like humans. They retrieve URLs from training data rather than navigating table-of-contents structures. They struggle with HTML-heavy pages, lose content to truncation with no indication anything was cut, and don’t know about emerging standards like llms.txt unless explicitly told. These checks codify the patterns that empirically help or hinder agent access to documentation content.

Terminology #

• Agent: An LLM operating in an agentic coding workflow (e.g., Claude Code, Cursor, Copilot) that fetches and consumes documentation as part of a development task. See Scope for what this spec does and does not cover.
• Web fetch pipeline: The chain of processing between “agent requests a URL” and “model sees content.” Typically involves HTTP fetch, HTML-to-markdown conversion, truncation, and sometimes a summarization model.
• Trusted site: A domain hardcoded into an agent platform’s web fetch implementation that receives more favorable processing (e.g., bypassing summarization).
• Truncation: The silent removal of content that exceeds a platform’s size limit. The agent receives partial content with no indication that anything was cut. See Appendix A for known limits by platform.

Conventions #

This spec uses the following language to distinguish between requirements and recommendations:

• Must / Required: The item is an absolute requirement of the spec. Used sparingly; most checks in this spec are recommendations rather than hard requirements, because agent-friendliness is a spectrum.
• Should / Recommended: The item is a strong recommendation. There may be valid reasons to deviate, but the implications should be understood.
• May / Optional: The item is genuinely optional. Implementing it provides additional benefit but omitting it is not a deficiency.

Sections of this spec are either normative (defining checks and their pass/warn/fail criteria) or informational (providing context, evidence, and recommendations). The distinction is noted where it matters:

• Normative sections: Category 1-7 check definitions, Checks Summary table.
• Informational sections: Background, Scope (including Related Surfaces), Start Here, “How Agents Get Content”, “Who Actually Uses llms.txt?”, Progressive Disclosure recommendation, “Making Private Docs Agent-Accessible”, Interaction Effects, “Serving RAG Ingestion Pipelines”, Appendices. Interaction Effects guides how implementations present combined results; it defines no pass/warn/fail criteria of its own.

The progressive disclosure pattern for llms.txt is a recommendation from this spec, not a normative requirement. Sites that keep their llms.txt under 50,000 characters don’t need it.

Start Here: Top Recommendations #

If you’re a documentarian and can only do a few things, start with these. They are ordered by impact based on observed agent behavior:

1. Create an llms.txt that fits in a single agent fetch (under 50K characters). This is the single highest-impact action. Agents that find an llms.txt navigate documentation dramatically better. If your docs set is large, use the nested pattern to keep each file under the limit. Checks: llms-txt-exists, llms-txt-size

2. Serve markdown versions of your pages, and verify what you serve. Either via .md URL variants or content negotiation. Markdown is what agents actually want; HTML conversion is lossy and unpredictable. Then treat the markdown generator as a second rendering pipeline that needs its own QA: it can emit broken links or partial content while the HTML looks fine, and no human reads the markdown to notice. Checks: markdown-url-support, content-negotiation, markdown-link-portability, single-fetch-completeness

3. Keep pages under 50,000 characters of content. If a page has tabbed or dropdown content, or generated data tables, that serialize into a massive blob, break it into separate pages or ensure the markdown version stays under the limit. Checks: page-size-markdown, page-size-html, tabbed-content-serialization, embedded-data-serialization

4. Put a pointer to your llms.txt at the top of every docs page. A simple blockquote directive that tells agents where to find the documentation index. Anthropic does this; it works. Include it in both the HTML DOM and in markdown versions of pages. Checks: llms-txt-directive-html, llms-txt-directive-md

5. Don’t break your URLs. If you must move content, use same-host HTTP redirects. Avoid cross-host redirects, JavaScript redirects, and soft 404s. Checks: http-status-codes, redirect-behavior

6. Make sure bot protection isn’t blocking agents. Bot management tuned for scraper traffic can challenge, throttle, or stall automated clients fetching your public docs, and the failure is invisible from a browser. Exempt documentation routes from behavioral enforcement, or scope enforcement to interactive product surfaces. For affected sites this outranks everything else on this list: nothing above matters if agents can’t fetch at all. Checks: bot-protection-interference

7. Monitor your agent-facing resources. Treat llms.txt and markdown endpoints like any other production surface: check freshness, verify content parity with HTML, and ensure cache headers allow timely updates. Checks: llms-txt-coverage, markdown-content-parity, cache-header-hygiene

Spec Structure #

Each check has:

• ID: A short identifier (e.g., llms-txt-exists).
• Category: The area of agent-friendliness it evaluates.
• What it checks: A description of what the check evaluates.
• Why it matters: The observed agent behavior that motivates the check.
• Result levels: What constitutes a pass, warn, or fail.
• Recommended action: What to do to resolve a warn or failure state.
• Automation: Whether the check can be fully automated, partially automated (heuristic), or is advisory only.

The canonical source of this spec is a single document (SPEC.md in the repository). The website serves it as per-category pages under agentdocsspec.com/spec/web/, following this spec’s own progressive disclosure recommendation: the full document outgrew the 100,000-character truncation threshold its own checks are built around, so it now practices what it prescribes. Each page stays well under the 50,000-character pass threshold.

Check Dependencies #

Some checks depend on the results of others:

• llms-txt-valid, llms-txt-size, llms-txt-links-resolve, and llms-txt-links-markdown only run if llms-txt-exists passes.
• page-size-markdown only runs if markdown-url-support or content-negotiation passes (the site must serve markdown for this check to apply).
• page-size-html and content-start-position results should be flagged as unreliable if rendering-strategy fails (the measurements reflect a shell, not actual content).
• section-header-quality is most relevant when tabbed-content-serialization detects tabbed content.
• markdown-code-fence-validity only runs if markdown-url-support or content-negotiation passes (the site must serve markdown for this check to apply). It also runs against any discovered llms.txt files.
• llms-txt-coverage only runs if llms-txt-exists passes.
• auth-alternative-access only runs if auth-gate-detection returns warn or fail (the site must have auth-gated content for alternative access paths to be relevant).
• bot-protection-interference has no prerequisites, but it inverts the usual dependency direction: when it returns warn or fail, results from every multi-page check should be flagged as computed from a partial sample (see the Bot Protection Degrading Scan Reliability interaction effect). It is also evaluated from evidence gathered across the entire scan rather than run as a discrete Category 7 step.
• markdown-content-parity only runs if markdown-url-support or content-negotiation passes (the site must serve markdown for this check to apply).
• single-fetch-completeness and markdown-link-portability only run if the site serves markdown by some detected path: markdown-url-support or content-negotiation passes, or llms-txt-links-markdown finds markdown links. A site can serve agent-facing markdown through llms.txt alone, without page-level .md variants, and that markdown deserves the same evaluation.

Implementations should run checks in category order (1 through 7) and skip dependent checks when their prerequisites fail.

A Note on Responsible Use #

This spec describes checks that involve making HTTP requests to documentation sites. Implementations should be respectful of the sites being evaluated: introduce delays between requests, cap concurrent connections, honor Retry-After headers, and avoid overwhelming sites with traffic. The goal is to help documentation teams improve agent accessibility, not to load-test their infrastructure.

Checks Summary #

Serving RAG Ingestion Pipelines #

This section offers non-normative guidance for sites whose documentation is consumed by retrieval-augmented generation systems: documentation Q&A bots, docs MCP servers, and internal knowledge bases that chunk and embed content into a vector store. RAG retrieval happens at query time from that index, but ingestion has to get the content from somewhere first, and there are two common paths:

• Web ingestion crawls the published site. It fetches like any automated client, so the properties that make a site agent-friendly at fetch time also make it index-friendly at ingestion time. The check mapping below applies to this path.
• Source ingestion reads the documentation source files (the markdown or MDX in the docs repository) directly, bypassing delivery entirely. This is common for teams indexing their own docs, and it is often a workaround for a site that is hostile to crawling; a site that passes this spec’s checks makes web ingestion a viable alternative. Source ingestion trades delivery problems for build-system problems: the ingester is effectively a second renderer, and build-time constructs (component tags, includes, frontmatter, variable substitution) reach the index unrendered unless the pipeline handles them the way the site generator does. The result can be an index of content nobody publishes. These are source-format processability concerns, out of scope for this spec; note that they are also distinct from the repository-local documentation surface in Related Surfaces, which concerns agents working interactively inside a repository, not batch pipelines reading its files.

This guidance is informed by consumer reports from production RAG builds.

How existing checks serve web ingestion:

• Crawl manifest: llms-txt-exists and llms-txt-coverage give ingesters a complete, curated URL set instead of sitemap heuristics, and the descriptions provide per-page metadata for free.
• Clean source format: markdown-url-support and content-negotiation let pipelines ingest markdown directly and skip HTML extraction, which is the largest source of ingestion noise. Sites failing rendering-strategy or content-start-position poison their own index: boilerplate and empty shells get embedded and then retrieved.
• Chunk boundaries: section-header-quality matters doubly for RAG. Chunkers split on section boundaries, and a chunk’s header may be the only context it carries into retrieval. Self-describing headers produce self-describing chunks; generic headers (“Step 1”) produce chunks that retrieve poorly and confuse whatever reads them. Content that flows across section boundaries without markers chunks badly in practice.
• Incremental re-indexing: cache-header-hygiene is more valuable for RAG than for live fetch. ETag and Last-Modified let pipelines detect what changed and re-embed only that, instead of re-crawling wholesale or serving a stale index.
• Complete, navigable content: single-fetch-completeness and markdown-link-portability failures propagate undetected into an index. A paginated catalog ingests as a partial catalog; broken generated links embed as broken references.

What this spec does not cover: embedding-friendly prose density and guidance on writing self-contained sections are content-composition concerns; see Related Surfaces. Content-categorization metadata (typed fields describing what a page is and what domain it serves) is a different kind of open question, under discussion in this project’s issue tracker: community vocabulary proposals exist and some platforms already serve frontmatter in markdown variants, but whether live agents benefit from receiving that metadata, or merely spend truncation budget on it, is unproven in either direction. This spec takes no position until there is evidence.

Contributing #

This spec is a living document. Feedback, corrections, and contributions are welcome.

• Discussion and feedback: Open an issue on the GitHub repository.
• Proposing changes: Submit a pull request. For significant changes (new checks, changes to pass/warn/fail criteria, new categories), please open an issue first to discuss the proposal.
• Platform truncation data: If you have data about a platform’s web fetch truncation limits (from official documentation, reverse engineering, or empirical testing), please contribute it to the Known Platform Limits table via issue or PR.
• Real-world validation: If you’ve run these checks against your own documentation site and have findings to share, we’d love to hear about it.

References #

• llmstxt.org proposal
• Dachary Carey, “Agent-Friendly Docs”
• Dachary Carey, “Agent Web Fetch Spelunking”
• Giuseppe Gurgone, reverse-engineered Claude Code Web Fetch
• Mikhail Shilkov, “Claude Code Web Tools”
• Liran Yoffe, “Reverse Engineering Claude Code Web Tools”
• Checkly, “State of AI Agent Content Negotiation”
• Kody Jackson, “The speculative and soon to be outdated AI consumability scorecard”
• Longato, “LLMs.txt - Why Almost Every AI Crawler Ignores it”
• OtterlyAI, “llms.txt and AI Visibility: Results from OtterlyAI’s GEO Study”