Category 6: Observability and Content Health

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether the site’s agent-facing resources stay accurate and up to date over time. Categories 1-5 can be evaluated as point-in-time audits; this category addresses the ongoing maintenance dimension. llms.txt files and markdown endpoints are secondary outputs that often aren’t wired into existing monitoring, so they can go stale, break, or drift from primary HTML content without anyone noticing.

llms-txt-coverage #

• What it checks: How much of the site’s documentation is represented in llms.txt.

• Why it matters: llms.txt is an agent’s primary navigational index into a documentation site. Pages missing from the index are effectively invisible to agents that rely on it for discovery. Unlike llms-txt-links-resolve (which catches broken links to pages that are listed), this check catches the opposite problem: pages that exist on the site but aren’t listed at all.

  Not every gap is a problem; many sites intentionally curate their llms.txt to include only a subset of pages. The check’s value is making the coverage level visible so site owners can confirm it reflects their intent.

• Result levels (based on coverage of sitemap doc pages, excluding non-doc pages like blog posts, pricing, and login pages):

  • Pass: llms.txt links cover >=95% of the site’s primary pages.
  • Warn: llms.txt links cover 80-95% of primary pages (some live pages are missing).
  • Fail: llms.txt links cover <80% of primary pages (missing large sections of the documentation). These thresholds are defaults that assume the site intends llms.txt to mirror the sitemap. Sites that intentionally curate their llms.txt should adjust thresholds to match their intent (see Notes below). Implementations should allow thresholds to be configured.

• Recommended action:

  • Warn: Review missing pages. If they should be in llms.txt, add them. If they are intentionally excluded, adjust the coverage threshold or add them to an exclusion list so the check reflects your intent.
  • Fail: If unintentional, regenerate llms.txt from your sitemap or build pipeline. If intentional, lower the threshold or set it to 0 to make the check informational.

• Automation: Heuristic. Compare links in llms.txt against a sitemap or crawled page list; flag pages present in the sitemap but absent from llms.txt. Implementations should support exclusion patterns that remove known-intentional gaps from the sitemap before calculating coverage.

• Notes: Not every sitemap page belongs in llms.txt. Sites intentionally exclude content for good reasons: changelog and release notes archives that would bloat the file, older product versions that aren’t relevant to current development, API reference pages that aren’t useful in markdown form, or directory pages that just link to other pages already listed. This is legitimate curation, not drift.

  The check should accommodate three use cases through configurable thresholds and exclusion patterns:

  • Full parity: The site intends llms.txt to mirror the sitemap. Default thresholds (95/80) apply; no exclusions needed.
  • Curated: The site intentionally includes only a subset of pages. Set thresholds to 0 to make the check informational. It still reports coverage percentage and lists what’s missing, but never warns or fails.
  • Hybrid: The site wants strict coverage but with known exclusions. Exclusion patterns remove intentional gaps from the sitemap before calculating coverage; remaining pages are held to the default thresholds.

  The definition of “primary pages” in the denominator requires judgment. Implementations should document how they construct the URL pool from the sitemap and what filtering they apply.

markdown-content-parity #

• What it checks: Whether markdown versions of pages contain the same substantive content as their HTML counterparts.

• Why it matters: When markdown is generated separately from HTML (rather than being the source that HTML is built from), the two can drift. A site might update an HTML page but forget to regenerate the markdown version, leaving agents with outdated instructions or code examples. This is particularly insidious because agents that receive the markdown version have no signal that a newer HTML version exists.

  However, in some cases, content divergence may be intentional. Some sites intentionally serve different content to different audiences, providing agent-optimized markdown alongside human-optimized HTML. In those cases, the divergence is deliberate. The check’s value is surfacing it so site owners can confirm it reflects their intent.

• Result levels (based on the percentage of content segments in the HTML version that are missing from the markdown version, after normalizing whitespace, case, and formatting):

  • Pass: <5% of content segments missing (or page has fewer than 10 segments, which is too small to produce meaningful parity scores).
  • Warn: 5-20% of content segments missing (minor differences: formatting variations, navigation elements present in one but not the other).
  • Fail: >=20% of content segments missing (substantive content differences: missing sections, outdated code examples, or different instructions between the two versions). These thresholds are defaults that assume the site intends markdown to mirror HTML. Sites that intentionally serve different content per audience should adjust thresholds to match their intent (see Notes below). Implementations should allow thresholds to be configured.

• Recommended action:

  • Warn: Review pages with minor differences. If they are formatting variations that may affect agent comprehension, fix them. If they reflect intentional audience segmentation, adjust thresholds or configure the check to account for it.
  • Fail: If unintentional, agents receiving the markdown version are getting outdated or incomplete content. Regenerate markdown from source or fix the build pipeline. If intentional, lower the threshold or set it to 0 to make the check informational.

• Automation: Heuristic. Fetch both versions, extract text content from HTML (strip tags), and compare key sections (headings, code blocks, paragraph content) for meaningful differences. Minor formatting differences should be ignored. If the HTML contains audience-segmentation tags (see Notes), implementations should strip tagged content before comparing so that intentionally excluded content does not count as missing.

• Notes: Sites where markdown is the source format and HTML is generated from it are less likely to have parity issues, but the check is still valuable as a safety net for build pipeline failures.

  Audience segmentation. Some documentation platforms use HTML tags to control what content appears in each version. For example, a platform might tag certain content as agent-only (included in markdown but not rendered in HTML) or human-only (rendered in HTML but excluded from markdown). Platforms like Fern and Mintlify have implemented this pattern. When the HTML contains recognized audience-segmentation tags, implementations should account for them before comparing: content explicitly tagged for one audience should not count as missing from the other.

  The spec does not define a standard set of segmentation tags or prescribe which vendor conventions to recognize. Implementations should document which tag conventions they support, and vendors or site owners who want their conventions recognized can contribute them to implementations directly.

  As with llms-txt-coverage, the check should accommodate sites at different points on the mirrored-to-curated spectrum:

  • Mirrored (default): Markdown should match HTML. Default thresholds apply.
  • Segmented: The site uses audience-segmentation tags to control per-version content. The check strips tagged content before comparing; remaining shared content is held to the default thresholds.
  • Curated: The site intentionally serves different content with no tag-level signal. Set thresholds to 0 to make the check informational.

  Dynamically generated pages. Pages built from data (catalogs, model listings, compatibility matrices) can diverge between representations without anyone deciding they should, because the HTML and markdown variants are rendered by different pipelines with different defaults. In one observed production case, a catalog’s HTML showed 98 items while its markdown variant listed 102: the HTML applied a default filter the markdown dump didn’t, and the markdown was additionally paginated. For pages with repeated structure, implementations should compare item counts between representations, and should distinguish the likely causes when counts differ: a default filter on the dynamic view (divergent by configuration), pagination on either side (divergent by windowing, see single-fetch-completeness), or staleness (one representation generated from older data). Each has a different owner and fix.

cache-header-hygiene #

• What it checks: Whether llms.txt and markdown endpoints have cache headers that allow timely updates.
• Why it matters: Aggressive caching on agent-facing resources means that even after a site owner updates their llms.txt or markdown content, agents (and intermediary CDNs) may continue serving stale versions for hours or days. Conversely, no cache headers at all leads to ambiguous behavior where different CDN providers apply their own defaults. For resources that are relatively small and infrequently fetched, short cache lifetimes with revalidation are appropriate.
• Result levels:
  • Pass: Cache headers allow timely updates (e.g., max-age under 3600, or uses must-revalidate with ETag/Last-Modified).
  • Warn: Moderate caching (1-24 hours) that could delay updates.
  • Fail: Aggressive caching (over 24 hours) with no revalidation mechanism, or no cache-related headers at all (ambiguous behavior). An exception: responses that lack Cache-Control and Expires but include ETag or Last-Modified should pass, since these validation headers enable conditional revalidation by browsers and CDNs even without explicit cache directives.
• Recommended action:
  • Warn: Updates to llms.txt or markdown content may take hours to propagate. Consider reducing cache lifetimes for these resources.
  • Fail: Set max-age under 3600 or add must-revalidate with ETag/Last-Modified so content updates reach agents promptly.
• Automation: Full. Inspect Cache-Control, Expires, ETag, and Last-Modified response headers.

Ongoing Monitoring Recommendations #

The three checks above can be run as one-time audits, but they’re most valuable when run on a schedule. This section offers non-normative guidance on integrating agent-facing resources into existing monitoring workflows.

Include llms.txt and markdown endpoints in uptime monitoring. These resources should be monitored alongside your primary documentation site. A 200 response from your docs homepage doesn’t guarantee that /llms.txt or .md URL variants are also healthy. Add them to whatever uptime tool you already use (Pingdom, Uptime Robot, Checkly, etc.) as separate check targets.

Set up alerting for response time degradation. If your llms.txt or markdown endpoints start responding slowly, agents may time out before receiving content. This is especially relevant for dynamically generated markdown (as opposed to static files), where a backend issue could cause latency spikes that don’t affect the HTML site.

Run coverage and parity checks on a schedule. Rather than treating llms-txt-coverage and markdown-content-parity as one-time audits, run them weekly or on every deploy. A CI check that compares llms.txt link coverage against the sitemap can catch missing pages before they reach production.

Monitor for silent failures. A 200 response with empty content, a generic error message, or a login page is worse than a clean 404, because agents will try to extract information from the response. Check that llms.txt and markdown responses contain expected content markers (e.g., an H1, a minimum character count) rather than just checking for a 200 status code.