# Category 1: Content Discoverability > For AI agents: the complete documentation index is at [https://agentdocsspec.com/llms.txt](https://agentdocsspec.com/llms.txt). Markdown versions of any page are available by appending `index.md` to the URL path. Part of the [Web Documentation Delivery Spec](https://agentdocsspec.com/spec/web/). The [Checks Summary](https://agentdocsspec.com/spec/web/#checks-summary) lists all checks with links to their definitions. These checks evaluate whether agents can find and navigate the site's documentation content. This includes whether the site provides an `llms.txt` file, whether that file is useful to agents, and whether documentation pages include signals that direct agents to discovery resources. ### Location Discovery The [llmstxt.org proposal](https://llmstxt.org) specifies that `llms.txt` should be at the root path (`/llms.txt`), mirroring `robots.txt` and `sitemap.xml`. In practice, the location varies significantly across sites: | Site | Root `/llms.txt` | `/docs/llms.txt` | Notes | |------|:-:|:-:|-------| | MongoDB | 200 | 200 | Both locations, different content | | Neon | 200 | 200 | Both locations | | Stripe | 200 | 301 -> docs.stripe.com | Root + docs subdomain | | Vercel | 200 | 308 -> root | Root only, /docs redirects | | React | 200 | -- | Root only | | GitHub Docs | 200 | -- | Root only | | Claude Code | 302 -> product page | 200 | /docs only; root is not docs | | Anthropic (old) | 301 -> 404 | -- | Moved domain, redirect breaks | The proposal does not address whether sites should serve `llms.txt` at subpaths, or whether a site with docs at `/docs/` should place it at `/docs/llms.txt` vs `/llms.txt`. In practice, both patterns exist. Implementations should check multiple candidate locations. **Discovery algorithm**: Given a base URL, check for `llms.txt` at: 1. `{base_url}/llms.txt` (the exact URL the user provided, plus llms.txt) 2. `{origin}/llms.txt` (site root, per the proposal) 3. `{origin}/docs/llms.txt` (common docs subpath) Where `{origin}` is the scheme + host of the base URL, and `{base_url}` is the full URL the user provided (which might be `https://example.com/docs` or `https://example.com` or `https://docs.example.com`). Duplicate URLs are deduplicated before checking. For each location, record whether `llms.txt` exists and whether the response involved a redirect (and if so, what kind). All subsequent llms.txt checks run against every discovered `llms.txt` file. ### `llms-txt-exists` - **What it checks**: Whether `llms.txt` is discoverable at any of the candidate locations described above. - **Why it matters**: `llms.txt` was the single most effective discovery mechanism observed. When agents found one, it fundamentally changed their ability to navigate a documentation site. Agents don't know to look for `llms.txt` by default, but when pointed at one, they treat it as a primary navigation resource. - **Result levels**: - **Pass**: `llms.txt` exists at one or more candidate locations, returning 200 with text content (direct or after same-host redirect). - **Warn**: `llms.txt` exists but is only reachable via cross-host redirect (agents may not follow it). - **Fail**: `llms.txt` not found at any candidate location. - **Recommended action**: - **Warn**: Serve `llms.txt` directly from the same host as your documentation, or use a same-host redirect. Cross-host redirects are not followed by some agents. - **Fail**: Create an `llms.txt` file at your site root containing an H1 title, a blockquote summary, and markdown links to your key documentation pages. This is the single highest-impact improvement for agent access. - **Automation**: Full. - **Report details**: List all candidate URLs checked and their status (200, 404, redirect chain). When multiple locations return `llms.txt`, note whether they serve the same or different content. ### `llms-txt-valid` - **What it checks**: Whether the `llms.txt` follows the structure described in the [llmstxt.org proposal](https://llmstxt.org). The proposal specifies: - An H1 with the project/site name. - A blockquote with a short summary. - H2-delimited sections containing markdown link lists. - Each link entry: `[name](url)` optionally followed by `: description`. - An optional H2 "Optional" section for secondary content. - Optional companion file `llms-full.txt` with complete content. - **Why it matters**: A well-structured `llms.txt` gives agents a reliable map of the documentation. Inconsistent implementations reduce its value. That said, even a non-standard `llms.txt` that contains useful links is better than nothing. - **Result levels**: - **Pass**: Follows the proposed structure with H1, summary blockquote, and heading-delimited link sections. - **Warn**: Contains parseable markdown links but doesn't follow the proposed structure (still useful, just non-standard). - **Fail**: Exists but contains no parseable links, or is empty. - **Recommended action**: - **Warn**: Add an H1 title as the first line and a blockquote summary (lines starting with `>`) to improve agent parsing. - **Fail**: Add links in `[name](url): description` format under heading-delimited sections. - **Automation**: Full. - **Checks in detail**: - H1 present (first line starts with `# `). - Blockquote summary present (line starting with `> `). - At least one heading-delimited section with markdown links. - Links follow `[name](url)` format. - Optional: check for `llms-full.txt` companion file. - **Notes on heading levels**: The llmstxt.org proposal specifies H2 (`##`) for section delimiters. In practice, some implementations (notably MongoDB) use H1 (`#`) for sections instead. Implementations should accept any heading level for section delimiters when evaluating structure. The important thing is that sections exist and contain parseable links, not that they use a specific heading level. ### `llms-txt-links-resolve` - **What it checks**: Whether the URLs listed in `llms.txt` resolve to the content they promise. - **Why it matters**: A stale `llms.txt` with broken links is worse than no `llms.txt` at all. It sends agents down dead ends with high confidence. - **Result levels**: - **Pass**: All links resolve (200, following same-host redirects) with no soft 404s or representation mismatches. - **Warn**: >90% of links resolve. - **Fail**: <=90% of links resolve. - **Recommended action**: Audit and fix or remove broken URLs. A stale `llms.txt` with broken links is worse than no `llms.txt` at all because it sends agents down dead ends with high confidence. - **Automation**: Full. Verification must go beyond status codes: apply soft-404 heuristics (see `http-status-codes`) to the response body, and verify the content type for links that promise markdown. A bare status check passes broken links; in one observed production case, a generated catalog's `.md` links all soft-404ed as HTML SPA shells while returning 200 (see `markdown-link-portability`). - **Notes**: Requires making HTTP requests to each URL. For large files, implementations may choose to test a random subset rather than every link. Links in `llms.txt` should be absolute URLs: the file is copied, cached, and aggregated by tools that drop its source URL, so relative links share the portability failure modes described in `markdown-link-portability`. The [progressive disclosure design principles](#progressive-disclosure-for-large-documentation-sets) make the same recommendation for links between index levels. ### `llms-txt-size` - **What it checks**: The character count of the `llms.txt` file, and whether it exceeds the truncation limits of known agent web fetch pipelines. - **Why it matters**: An `llms.txt` that exceeds an agent's truncation limit defeats its own purpose. The agent sees only a fraction of the index and may miss the section it needs entirely. This is the same truncation problem that affects documentation pages, but arguably worse because `llms.txt` is supposed to be the *solution* to discovery. Real-world sizes vary enormously: | Site | Size | Links | Notes | |------|------|-------|-------| | MongoDB `/docs/llms.txt` | 4.56 MB | 21,891 | Every version of every product | | Vercel | 287 KB | ~3,000 | Single file | | Stripe | 89 KB | ~1,000 | Single file | | Neon | 75 KB | ~600 | Points to .md URLs | | React | 14 KB | ~150 | Single file | | Claude Code | 11 KB | ~60 | Small, focused | | GitHub Docs | 2 KB | ~30 | Small index | | MongoDB `/llms.txt` (root) | 1.5 KB | 6 | Top-level index only | Claude Code's web fetch pipeline truncates at ~100KB. A 4.56MB file means the agent sees roughly 2% of it. Even Vercel's 287KB file would be heavily truncated. Only the files under ~100KB are reliably consumable in their entirety by current agent implementations. - **Result levels**: - **Pass**: Under 50,000 characters (fits comfortably within all known truncation limits, even accounting for overhead). - **Warn**: Between 50,000 and 100,000 characters (fits within Claude Code's limit but may not fit others; consider splitting). - **Fail**: Over 100,000 characters (will be truncated by Claude Code and likely all other agent platforms). - **Recommended action**: - **Warn**: If the file grows further, split into nested `llms.txt` files with a root index under 50,000 characters. - **Fail**: Split into a root index linking to section-level `llms.txt` files, each under 50,000 characters. See [Progressive Disclosure for Large Documentation Sets](#progressive-disclosure-for-large-documentation-sets) below. - **Automation**: Full. ### `llms-txt-links-markdown` - **What it checks**: Whether the URLs in `llms.txt` point to markdown content (`.md` extension in the URL, or response with `Content-Type: text/markdown`). - **Why it matters**: Markdown content is dramatically more useful to agents than HTML. An `llms.txt` that points agents to HTML pages misses an opportunity to deliver content in the most agent-friendly format. The best implementations (like Neon's) point to `.md` URLs that serve clean markdown directly. - **Result levels**: - **Pass**: All or most links point to markdown content. - **Warn**: Links point to HTML, but markdown versions are available (detected by trying `.md` variants of the URLs). - **Fail**: Links point to HTML and no markdown alternatives are detected. - **Recommended action**: Update `llms.txt` links to use `.md` URL variants so agents receive markdown instead of converted HTML. - **Automation**: Full. ### Progressive Disclosure for Large Documentation Sets The llmstxt.org proposal does not address what to do when a documentation site is too large for a single `llms.txt` file to fit within agent truncation limits. In practice, large documentation sets (like MongoDB's, with 185 products/versions and 21,891 links) produce `llms.txt` files that are orders of magnitude beyond what any current agent can consume in a single fetch. #### Who Actually Uses llms.txt? The original framing of `llms.txt` drew analogies to `robots.txt` and `sitemap.xml`, suggesting it would serve AI crawlers gathering training data. The evidence shows this hasn't happened: - An audit of 1,000 domains over 30 days found zero visits to `llms.txt` from GPTBot, ClaudeBot, or PerplexityBot ([Longato, August 2025](https://www.longato.ch/llms-recommendation-2025-august/)). - A 90-day study tracking 62,100+ AI bot visits found only 84 requests (0.1%) to `/llms.txt`, roughly 3x fewer visits than an average content page ([OtterlyAI GEO Study](https://otterly.ai/blog/the-llms-txt-experiment/)). - John Mueller from Google stated directly: "no AI system currently uses llms.txt." Training crawlers don't use `llms.txt` because they have their own discovery mechanisms (sitemaps, link following, pre-built datasets) and probing `/llms.txt` on every domain would waste crawl budget for an unestablished standard. The real consumers of `llms.txt` are **agents in real-time workflows**: a developer's coding assistant fetching documentation to verify an API pattern, an agent following a directive on a docs page that points it to `llms.txt`, or a user explicitly handing their agent an `llms.txt` URL as a discovery starting point. These are fetch-once, use-now interactions subject to the truncation limits of web fetch pipelines. This distinction matters for our recommendation. A progressive disclosure pattern that splits `llms.txt` into nested files has no practical impact on crawler consumption (since crawlers aren't consuming it). It directly benefits the agent use case, which is where `llms.txt` actually provides value today. #### Recommendation We recommend a **nested `llms.txt` pattern** for progressive disclosure: #### Structure A **root `llms.txt`** serves as a table of contents, listing the major sections of the documentation with links to **section-level `llms.txt` files**. Each section-level file contains the actual page links for that section. ``` # MongoDB Documentation > MongoDB is the leading document database. This index covers all MongoDB > products, drivers, and tools documentation. ## Products - [Atlas](https://www.mongodb.com/docs/atlas/llms.txt): MongoDB Atlas cloud database - [Atlas CLI](https://www.mongodb.com/docs/atlas-cli/llms.txt): Command-line interface for Atlas - [Compass](https://www.mongodb.com/docs/compass/llms.txt): GUI for MongoDB - [MongoDB Server](https://www.mongodb.com/docs/manual/llms.txt): Server documentation ## Drivers - [Python Driver](https://www.mongodb.com/docs/drivers/pymongo/llms.txt): PyMongo driver - [Node.js Driver](https://www.mongodb.com/docs/drivers/node/llms.txt): Node.js driver - [Java Driver](https://www.mongodb.com/docs/drivers/java/llms.txt): Java sync and reactive drivers ``` Each linked `llms.txt` then contains the actual page listings for that product or driver, scoped to the current version (or with a small number of version variants). #### Design Principles 1. **The root `llms.txt` should fit in a single agent fetch.** Target under 50,000 characters. This is the entry point that agents will discover first, and it must be fully consumable. It should contain enough descriptive context for an agent to identify which section-level file to fetch next. 2. **Section-level files should also fit in a single agent fetch.** If a section is still too large (e.g., a product with hundreds of pages across many versions), consider further nesting or limiting the index to the current version only. 3. **Version sprawl is the primary size driver.** The MongoDB `/docs/llms.txt` lists every version of every product. Linking to every historical version in the index provides diminishing returns for agents, who almost always want the current version. Historical versions could be listed in a separate `llms-versions.txt` or under the "Optional" H2 section that the proposal already defines for secondary content. 4. **Links between levels should use absolute URLs.** An agent following a link from root `llms.txt` to a section `llms.txt` needs to resolve it without ambiguity. 5. **Each `llms.txt` should be self-describing.** Include the H1 and blockquote summary at every level so an agent landing on a section-level file (via direct URL from training data, for example) has enough context to understand what it's looking at. #### Compatibility Note This nested pattern is a recommendation from this spec, not part of the llmstxt.org proposal as of February 2026. It is fully compatible with the existing proposal (which doesn't prohibit linking to other `llms.txt` files) but would benefit from formal standardization. The proposal's existing "Optional" H2 section could be leveraged for secondary/versioned content, but the nesting pattern goes further by distributing content across multiple files. ### `llms-txt-directive-html` - **What it checks**: Whether the HTML version of documentation pages includes a directive, visible to agents but not necessarily to human readers, pointing to `llms.txt` or another discovery resource. - **Why it matters**: Agents that fetch rendered HTML pages have no built-in way to discover that a documentation index exists at `/llms.txt` or that markdown versions of pages may be available. An in-page directive serves as an agent "You Are Here" marker that points them to the index. The directive can be visually hidden (e.g., using a CSS clip-rect technique or `sr-only` class) as long as it remains in the DOM and survives HTML-to-markdown conversion. Avoid `display: none`, which some converters strip. The directive should be present in server-rendered HTML; avoid relying solely on client-side JavaScript injection, since most agents fetch pages without executing JS. - **Detection considerations**: Implementations must distinguish intentional agent-facing directives from incidental mentions of `llms.txt`. Navigation items (e.g., sidebar links to a page *about* `llms.txt`), JSON-LD metadata, `