Category 1: Content Discoverability

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether agents can find and navigate the site’s documentation content. This includes whether the site provides an llms.txt file, whether that file is useful to agents, and whether documentation pages include signals that direct agents to discovery resources.

Location Discovery #

The llmstxt.org proposal specifies that llms.txt should be at the root path (/llms.txt), mirroring robots.txt and sitemap.xml. In practice, the location varies significantly across sites:

The proposal does not address whether sites should serve llms.txt at subpaths, or whether a site with docs at /docs/ should place it at /docs/llms.txt vs /llms.txt. In practice, both patterns exist. Implementations should check multiple candidate locations.

Discovery algorithm: Given a base URL, check for llms.txt at:

1. {base_url}/llms.txt (the exact URL the user provided, plus llms.txt)
2. {origin}/llms.txt (site root, per the proposal)
3. {origin}/docs/llms.txt (common docs subpath)

Where {origin} is the scheme + host of the base URL, and {base_url} is the full URL the user provided (which might be https://example.com/docs or https://example.com or https://docs.example.com). Duplicate URLs are deduplicated before checking.

For each location, record whether llms.txt exists and whether the response involved a redirect (and if so, what kind). All subsequent llms.txt checks run against every discovered llms.txt file.

llms-txt-exists #

• What it checks: Whether llms.txt is discoverable at any of the candidate locations described above.
• Why it matters: llms.txt was the single most effective discovery mechanism observed. When agents found one, it fundamentally changed their ability to navigate a documentation site. Agents don’t know to look for llms.txt by default, but when pointed at one, they treat it as a primary navigation resource.
• Result levels:
  • Pass: llms.txt exists at one or more candidate locations, returning 200 with text content (direct or after same-host redirect).
  • Warn: llms.txt exists but is only reachable via cross-host redirect (agents may not follow it).
  • Fail: llms.txt not found at any candidate location.
• Recommended action:
  • Warn: Serve llms.txt directly from the same host as your documentation, or use a same-host redirect. Cross-host redirects are not followed by some agents.
  • Fail: Create an llms.txt file at your site root containing an H1 title, a blockquote summary, and markdown links to your key documentation pages. This is the single highest-impact improvement for agent access.
• Automation: Full.
• Report details: List all candidate URLs checked and their status (200, 404, redirect chain). When multiple locations return llms.txt, note whether they serve the same or different content.

llms-txt-valid #

• What it checks: Whether the llms.txt follows the structure described in the llmstxt.org proposal. The proposal specifies:
  • An H1 with the project/site name.
  • A blockquote with a short summary.
  • H2-delimited sections containing markdown link lists.
  • Each link entry: [name](url) optionally followed by : description.
  • An optional H2 “Optional” section for secondary content.
  • Optional companion file llms-full.txt with complete content.
• Why it matters: A well-structured llms.txt gives agents a reliable map of the documentation. Inconsistent implementations reduce its value. That said, even a non-standard llms.txt that contains useful links is better than nothing.
• Result levels:
  • Pass: Follows the proposed structure with H1, summary blockquote, and heading-delimited link sections.
  • Warn: Contains parseable markdown links but doesn’t follow the proposed structure (still useful, just non-standard).
  • Fail: Exists but contains no parseable links, or is empty.
• Recommended action:
  • Warn: Add an H1 title as the first line and a blockquote summary (lines starting with >) to improve agent parsing.
  • Fail: Add links in [name](url): description format under heading-delimited sections.
• Automation: Full.
• Checks in detail:
  • H1 present (first line starts with # ).
  • Blockquote summary present (line starting with > ).
  • At least one heading-delimited section with markdown links.
  • Links follow [name](url) format.
  • Optional: check for llms-full.txt companion file.
• Notes on heading levels: The llmstxt.org proposal specifies H2 (##) for section delimiters. In practice, some implementations (notably MongoDB) use H1 (#) for sections instead. Implementations should accept any heading level for section delimiters when evaluating structure. The important thing is that sections exist and contain parseable links, not that they use a specific heading level.

llms-txt-links-resolve #

• What it checks: Whether the URLs listed in llms.txt resolve to the content they promise.
• Why it matters: A stale llms.txt with broken links is worse than no llms.txt at all. It sends agents down dead ends with high confidence.
• Result levels:
  • Pass: All links resolve (200, following same-host redirects) with no soft 404s or representation mismatches.
  • Warn: >90% of links resolve.
  • Fail: <=90% of links resolve.
• Recommended action: Audit and fix or remove broken URLs. A stale llms.txt with broken links is worse than no llms.txt at all because it sends agents down dead ends with high confidence.
• Automation: Full. Verification must go beyond status codes: apply soft-404 heuristics (see http-status-codes) to the response body, and verify the content type for links that promise markdown. A bare status check passes broken links; in one observed production case, a generated catalog’s .md links all soft-404ed as HTML SPA shells while returning 200 (see markdown-link-portability).
• Notes: Requires making HTTP requests to each URL. For large files, implementations may choose to test a random subset rather than every link. Links in llms.txt should be absolute URLs: the file is copied, cached, and aggregated by tools that drop its source URL, so relative links share the portability failure modes described in markdown-link-portability. The progressive disclosure design principles make the same recommendation for links between index levels.

llms-txt-size #

• What it checks: The character count of the llms.txt file, and whether it exceeds the truncation limits of known agent web fetch pipelines.

• Why it matters: An llms.txt that exceeds an agent’s truncation limit defeats its own purpose. The agent sees only a fraction of the index and may miss the section it needs entirely. This is the same truncation problem that affects documentation pages, but arguably worse because llms.txt is supposed to be the solution to discovery.

  Real-world sizes vary enormously:

  Site	Size	Links	Notes
  MongoDB /docs/llms.txt	4.56 MB	21,891	Every version of every product
  Vercel	287 KB	~3,000	Single file
  Stripe	89 KB	~1,000	Single file
  Neon	75 KB	~600	Points to .md URLs
  React	14 KB	~150	Single file
  Claude Code	11 KB	~60	Small, focused
  GitHub Docs	2 KB	~30	Small index
  MongoDB /llms.txt (root)	1.5 KB	6	Top-level index only

  Claude Code’s web fetch pipeline truncates at ~100KB. A 4.56MB file means the agent sees roughly 2% of it. Even Vercel’s 287KB file would be heavily truncated. Only the files under ~100KB are reliably consumable in their entirety by current agent implementations.

• Result levels:

  • Pass: Under 50,000 characters (fits comfortably within all known truncation limits, even accounting for overhead).
  • Warn: Between 50,000 and 100,000 characters (fits within Claude Code’s limit but may not fit others; consider splitting).
  • Fail: Over 100,000 characters (will be truncated by Claude Code and likely all other agent platforms).

• Recommended action:

  • Warn: If the file grows further, split into nested llms.txt files with a root index under 50,000 characters.
  • Fail: Split into a root index linking to section-level llms.txt files, each under 50,000 characters. See Progressive Disclosure for Large Documentation Sets below.

• Automation: Full.

llms-txt-links-markdown #

• What it checks: Whether the URLs in llms.txt point to markdown content (.md extension in the URL, or response with Content-Type: text/markdown).
• Why it matters: Markdown content is dramatically more useful to agents than HTML. An llms.txt that points agents to HTML pages misses an opportunity to deliver content in the most agent-friendly format. The best implementations (like Neon’s) point to .md URLs that serve clean markdown directly.
• Result levels:
  • Pass: All or most links point to markdown content.
  • Warn: Links point to HTML, but markdown versions are available (detected by trying .md variants of the URLs).
  • Fail: Links point to HTML and no markdown alternatives are detected.
• Recommended action: Update llms.txt links to use .md URL variants so agents receive markdown instead of converted HTML.
• Automation: Full.

Progressive Disclosure for Large Documentation Sets #

The llmstxt.org proposal does not address what to do when a documentation site is too large for a single llms.txt file to fit within agent truncation limits. In practice, large documentation sets (like MongoDB’s, with 185 products/versions and 21,891 links) produce llms.txt files that are orders of magnitude beyond what any current agent can consume in a single fetch.

Who Actually Uses llms.txt? #

The original framing of llms.txt drew analogies to robots.txt and sitemap.xml, suggesting it would serve AI crawlers gathering training data. The evidence shows this hasn’t happened:

• An audit of 1,000 domains over 30 days found zero visits to llms.txt from GPTBot, ClaudeBot, or PerplexityBot (Longato, August 2025).
• A 90-day study tracking 62,100+ AI bot visits found only 84 requests (0.1%) to /llms.txt, roughly 3x fewer visits than an average content page (OtterlyAI GEO Study).
• John Mueller from Google stated directly: “no AI system currently uses llms.txt.”

Training crawlers don’t use llms.txt because they have their own discovery mechanisms (sitemaps, link following, pre-built datasets) and probing /llms.txt on every domain would waste crawl budget for an unestablished standard.

The real consumers of llms.txt are agents in real-time workflows: a developer’s coding assistant fetching documentation to verify an API pattern, an agent following a directive on a docs page that points it to llms.txt, or a user explicitly handing their agent an llms.txt URL as a discovery starting point. These are fetch-once, use-now interactions subject to the truncation limits of web fetch pipelines.

This distinction matters for our recommendation. A progressive disclosure pattern that splits llms.txt into nested files has no practical impact on crawler consumption (since crawlers aren’t consuming it). It directly benefits the agent use case, which is where llms.txt actually provides value today.

Recommendation #

We recommend a nested llms.txt pattern for progressive disclosure:

Structure #

A root llms.txt serves as a table of contents, listing the major sections of the documentation with links to section-level llms.txt files. Each section-level file contains the actual page links for that section.

# MongoDB Documentation

> MongoDB is the leading document database. This index covers all MongoDB
> products, drivers, and tools documentation.

## Products

- [Atlas](https://www.mongodb.com/docs/atlas/llms.txt): MongoDB Atlas cloud database
- [Atlas CLI](https://www.mongodb.com/docs/atlas-cli/llms.txt): Command-line interface for Atlas
- [Compass](https://www.mongodb.com/docs/compass/llms.txt): GUI for MongoDB
- [MongoDB Server](https://www.mongodb.com/docs/manual/llms.txt): Server documentation

## Drivers

- [Python Driver](https://www.mongodb.com/docs/drivers/pymongo/llms.txt): PyMongo driver
- [Node.js Driver](https://www.mongodb.com/docs/drivers/node/llms.txt): Node.js driver
- [Java Driver](https://www.mongodb.com/docs/drivers/java/llms.txt): Java sync and reactive drivers

Each linked llms.txt then contains the actual page listings for that product or driver, scoped to the current version (or with a small number of version variants).

Design Principles #

1. The root llms.txt should fit in a single agent fetch. Target under 50,000 characters. This is the entry point that agents will discover first, and it must be fully consumable. It should contain enough descriptive context for an agent to identify which section-level file to fetch next.

2. Section-level files should also fit in a single agent fetch. If a section is still too large (e.g., a product with hundreds of pages across many versions), consider further nesting or limiting the index to the current version only.

3. Version sprawl is the primary size driver. The MongoDB /docs/llms.txt lists every version of every product. Linking to every historical version in the index provides diminishing returns for agents, who almost always want the current version. Historical versions could be listed in a separate llms-versions.txt or under the “Optional” H2 section that the proposal already defines for secondary content.

4. Links between levels should use absolute URLs. An agent following a link from root llms.txt to a section llms.txt needs to resolve it without ambiguity.

5. Each llms.txt should be self-describing. Include the H1 and blockquote summary at every level so an agent landing on a section-level file (via direct URL from training data, for example) has enough context to understand what it’s looking at.

Compatibility Note #

This nested pattern is a recommendation from this spec, not part of the llmstxt.org proposal as of February 2026. It is fully compatible with the existing proposal (which doesn’t prohibit linking to other llms.txt files) but would benefit from formal standardization. The proposal’s existing “Optional” H2 section could be leveraged for secondary/versioned content, but the nesting pattern goes further by distributing content across multiple files.

llms-txt-directive-html #

• What it checks: Whether the HTML version of documentation pages includes a directive, visible to agents but not necessarily to human readers, pointing to llms.txt or another discovery resource.
• Why it matters: Agents that fetch rendered HTML pages have no built-in way to discover that a documentation index exists at /llms.txt or that markdown versions of pages may be available. An in-page directive serves as an agent “You Are Here” marker that points them to the index. The directive can be visually hidden (e.g., using a CSS clip-rect technique or sr-only class) as long as it remains in the DOM and survives HTML-to-markdown conversion. Avoid display: none, which some converters strip. The directive should be present in server-rendered HTML; avoid relying solely on client-side JavaScript injection, since most agents fetch pages without executing JS.
• Detection considerations: Implementations must distinguish intentional agent-facing directives from incidental mentions of llms.txt. Navigation items (e.g., sidebar links to a page about llms.txt), JSON-LD metadata, <script> blocks, and page content that merely discusses llms.txt as a feature do not count as directives. A directive is a standalone element in the page content area whose purpose is to tell agents where to find the documentation index.
• Result levels:
  • Pass: A directive pointing to llms.txt (or equivalent index) is present in the HTML DOM of all (or nearly all) documentation pages, ideally near the top of the content.
  • Warn: A directive exists in the HTML of some pages but is missing from others, or is present but buried deep in the page (past 50% of content, where it may be past truncation).
  • Fail: No agent-facing directive detected in the HTML of any tested page.
• Recommended action:
  • Warn: Ensure the directive appears near the top of every documentation page’s HTML, not just some.
  • Fail: Add a visually-hidden element near the top of each page (e.g., a <div> with CSS clip-rect) containing text like “For AI agents: a documentation index is available at /llms.txt” and, if applicable, a note that markdown versions are available.
• Automation: Heuristic. Search the page HTML for patterns like links to llms.txt, phrases like “documentation index”, or directives near the top of the content area. Check both visible text and visually-hidden elements. Exclude matches in navigation, metadata, script blocks, and content that discusses llms.txt as a topic rather than directing agents to it.

llms-txt-directive-md #

• What it checks: Whether the markdown version of documentation pages includes a directive pointing to llms.txt or another discovery resource.
• Why it matters: Agents that fetch markdown versions of pages (via .md URLs or content negotiation) benefit from a directive that points them to the documentation index. Anthropic’s Claude Code documentation (code.claude.com/docs, hosted on Mintlify) includes a blockquote at the top of every markdown page telling agents to fetch the documentation index at llms.txt. In practice, agents that encounter this directive may follow it to discover the full documentation index. It’s simple, low-effort, and has been observed to work in real agent workflows.
• Result levels:
  • Pass: A directive pointing to llms.txt (or equivalent index) is present in the markdown of all (or nearly all) documentation pages, ideally near the top of the content.
  • Warn: A directive exists in the markdown of some pages but is missing from others, or is present but buried deep in the page (past 50% of content, where it may be past truncation).
  • Fail: No agent-facing directive detected in the markdown of any tested page.
• Recommended action:
  • Warn: Ensure the directive appears near the top of every markdown page, not just some.
  • Fail: Add a blockquote near the top of each markdown page (e.g., “> For the complete documentation index, see llms.txt”). Use an absolute URL; see markdown-link-portability for why relative links are fragile in served markdown.
• Automation: Heuristic. Fetch the markdown version of sampled pages (via .md URL or content negotiation) and search for patterns like links to llms.txt, phrases like “documentation index”, or blockquote directives near the top of the content. This check depends on markdown being available via markdown-url-support or content-negotiation; if neither passes, this check is skipped.