# Changelog

> For AI agents: the complete documentation index is at [https://agentdocsspec.com/llms.txt](https://agentdocsspec.com/llms.txt). Markdown versions of any page are available by appending `index.md` to the URL path.

Part of the [Web Documentation Delivery Spec](https://agentdocsspec.com/spec/web/). The [Checks Summary](https://agentdocsspec.com/spec/web/#checks-summary) lists all checks with links to their definitions.

### v0.6.0 (2026-09-13)

- Added `bot-protection-interference` (Category 7: Authentication and Access).
  Detects bot-protection systems interfering with automated documentation
  fetching: challenge interstitials served as 200, tarpits that return headers
  and then stall the response body indefinitely, and volume-triggered
  throttling or blocking. Grounded in an observed production case where CDN
  bot management responded to a sustained scan by holding response bodies
  open; single-request probes looked healthy throughout, and enforcement
  decayed after a cooldown. Detection is heuristic, observed as a byproduct
  of a normal scan rather than a directed probe.
- Added the [Bot Protection Degrading Scan Reliability](https://agentdocsspec.com/spec/web/interaction-effects/#bot-protection-degrading-scan-reliability)
  interaction effect: when enforcement engages mid-scan, other checks score
  the surviving sample, so implementations should aggregate fetch failures at
  run level and flag results when the failure rate is high.
- Expanded the Category 7 introduction to cover infrastructure-level access
  barriers alongside authentication.
- Added `page-size-transfer` (Category 3: Page Size and Truncation Risk).
  Measures the served byte size of the HTML document response, a failure
  mode `page-size-html`'s post-conversion measurement cannot see. Grounded
  in measurements of production documentation sites where 75-84% of page
  bytes were serialized framework payloads (component trees, resolved
  metadata, embedded duplicate markdown source), with served-bytes-to-content
  ratios from 40:1 to 200:1. Scored on served bytes; the ratio is reported
  as a diagnostic.
- Documented the pipeline distinction in Category 3's "How Agents Get
  Content" section: convert-then-truncate, truncate-then-convert (or raw
  ingestion), and capped fetch, and which size measurement predicts each.
- Added `single-fetch-completeness` (Category 3: Page Size and Truncation
  Risk). Detects pagination in markdown responses and verifies the
  continuation is machine-followable. Grounded in an observed production
  catalog whose markdown variant showed 100 of 102 entries with a trailing
  relative continuation URL that returned an empty body, while the complete
  set would have fit well under the 50,000-character pass threshold.
- Added `markdown-link-portability` (Category 4: Content Structure). Links
  in served markdown should be absolute and should resolve to the
  representation they promise, verified beyond status codes (content type
  and soft-404 heuristics). Grounded in an observed production catalog
  whose generated markdown links all pointed into a wrong path prefix and
  soft-404ed as HTML SPA shells at `.md` URLs while returning 200.
- Added `embedded-data-serialization` (Category 4: Content Structure).
  Attributes page size to machine-generated bulk elements (large uniform
  tables, inline data blobs). Grounded in a measured production reference
  page: 302KB of HTML, 64% table markup including a single 218-row
  generated table, converting to ~83,000 characters while its non-table
  prose totaled ~17,000.
- Extended `markdown-content-parity` notes with guidance for dynamically
  generated pages: compare item counts between representations and
  distinguish default-filter divergence, pagination windowing, and
  staleness as causes.
- Added the [Dynamic Content Rendered Statically](https://agentdocsspec.com/spec/web/interaction-effects/#dynamic-content-rendered-statically)
  interaction effect: the four characteristic ways a dynamic page flattens
  badly into static markdown (too much, too little, inconsistent,
  unnavigable), observed together on a single production catalog page.
- Added the [Related Surfaces](https://agentdocsspec.com/spec/web/#related-surfaces) subsection to Scope,
  naming the planned companion specifications (content composition,
  repository-local documentation) and the boundary that keeps this spec's
  checks mechanically verifiable.
- Added the [Serving RAG Ingestion Pipelines](https://agentdocsspec.com/spec/web/#serving-rag-ingestion-pipelines)
  informational section, mapping existing checks to RAG ingestion needs
  (crawl manifest, clean source, chunk boundaries, incremental
  re-indexing), informed by consumer reports from production RAG builds.
  The Scope section's RAG exclusion now distinguishes query-time retrieval
  (out of scope) from ingestion (served by this spec).
- Restructured the website serving of the spec: the full document exceeded
  the 100,000-character truncation threshold its own checks warn about, so
  it is now served as per-category pages under `/spec/web/`, with `/spec/`
  becoming a landing page for this and future companion specifications.
  The canonical source remains a single SPEC.md in the repository. This
  breaks previously published deep URLs deliberately, in exchange for a
  namespace that accommodates the companion specs.
- Check count: 23 → 28.

### v0.5.1 (2026-05-08)

- Moved per-platform truncation data out of Appendix A into a new
  [Platforms](https://agentdocsspec.com/platforms/) comparison page on the
  site. Appendix A retains the spec's threshold rationale and points readers
  to the platforms page for current per-platform observations. Category 3
  (Page Size and Truncation Risk) now references the platforms page so
  readers can connect threshold choices to empirical pipeline behavior. No
  threshold or check definitions changed. Platforms page authored by
  Rhyannon Rodriguez.

### v0.5.0 (2026-04-25)

- Split `llms-txt-directive` into two independent checks:
  `llms-txt-directive-html` and `llms-txt-directive-md`. The original check
  conflated two distinct signals that serve different audiences. The HTML
  check detects directives in the rendered DOM (for agents fetching HTML
  pages); the markdown check detects directives in markdown source (for
  agents fetching `.md` URLs or using content negotiation). The split also
  adds explicit detection guidance: incidental mentions of `llms.txt` in
  navigation, metadata, or page content discussing the feature do not count
  as directives. `llms-txt-directive-md` depends on `markdown-url-support`
  or `content-negotiation`; it is skipped if neither passes. This is a
  breaking change for implementations that reference the old check ID.
- Check count: 22 → 23.

### v0.4.0 (2026-04-21)

- Renamed `llms-txt-freshness` to `llms-txt-coverage`. The check compares
  `llms.txt` URLs against the sitemap to measure how much of the site is
  represented; that's coverage, not freshness. Whether listed URLs still
  resolve is already handled by `llms-txt-links-resolve`. Rewrote the check
  description to match. This is a breaking change for implementations that
  reference the old check ID.
- Revised `page-size-html` and `content-start-position` to be
  pipeline-agnostic. The previous language prescribed a specific conversion
  approach (Turndown with default configuration) based on one agent's
  behavior. Agent HTML processing pipelines vary and continue to evolve;
  the spec now describes the measurement goal (approximate what agents see)
  and leaves conversion details to implementers. Recommended actions now
  cover all boilerplate sources (navigation, sidebars, serialized tabbed
  content) rather than focusing narrowly on inline CSS/JS.
- Expanded `llms-txt-coverage` to account for intentional curation. Many
  sites deliberately include only a subset of pages in `llms.txt` (excluding
  changelogs, old versions, directory pages, etc.). The check now describes
  three use cases (full parity, curated, hybrid) served by configurable
  thresholds and exclusion patterns, rather than treating all gaps as
  problems.
- Expanded `markdown-content-parity` to distinguish intentional audience
  segmentation from unintentional content drift. Some sites intentionally
  serve different content per audience (agent-optimized markdown vs.
  human-optimized HTML). The check now describes audience-segmentation tags
  as a mechanism implementations can recognize, and supports the same
  mirrored/segmented/curated spectrum as `llms-txt-coverage`. The spec does
  not prescribe specific tag conventions; implementations document which
  they support.

### v0.3.0 (2026-03-31)

- Merged Category 6 (Agent Discoverability Directives) into Category 1,
  renamed to "Content Discoverability." The `llms-txt-directive` check (now
  `llms-txt-directive-html` and `llms-txt-directive-md`) answers
  the same fundamental question as the llms.txt checks: can agents find and
  navigate the content? This reduces categories from 8 to 7.
- Renumbered Category 7 (Observability) to 6, Category 8 (Authentication) to 7.
- Added **Recommended action** field to all 22 check definitions. Provides
  1-2 sentence actionable guidance for each warn and fail state, giving
  documentation teams a clear next step rather than just a diagnosis.
- Added **Interaction Effects** section after Checks Summary. Documents six
  patterns where combinations of check results indicate systemic problems
  worse than individual failures suggest (e.g., undiscoverable markdown,
  no viable content path, oversized pages without markdown escape).
- Category count: 8 → 7. Check count unchanged at 22.

### v0.2.1 (2026-03-15)

Clarifications from implementing the `afdocs` conformance tool against the
spec. No new checks; all changes refine existing check definitions.

- `markdown-code-fence-validity`: Removed warn level. Per CommonMark,
  mismatched delimiters (opening ``` closing ~~~) produce unclosed fences,
  not a distinct "mismatched but balanced" state. The described warn case
  was indistinguishable from a fail.
- `llms-txt-directive` (now `llms-txt-directive-html` and
  `llms-txt-directive-md`): Clarified that pass requires the directive in
  all (or nearly all) pages, not just presence in any single page. Clarified
  warn triggers: missing from some pages, or present but buried past 50%.
- `llms-txt-freshness` (now `llms-txt-coverage`): Added default thresholds (>=95% pass, 80-95% warn,
  <80% fail) for sitemap coverage. The previous language was qualitative;
  implementations need concrete defaults for automation.
- `markdown-content-parity`: Added default thresholds (<5% missing pass,
  5-20% warn, >=20% fail) for content segment comparison. Same rationale.
- `section-header-quality`: Added default thresholds (<=25% generic pass,
  25-50% warn, >50% fail) and clarified that evaluation covers both
  within-group and cross-group header repetition.
- `cache-header-hygiene`: Added exception for responses with `ETag` or
  `Last-Modified` but no `Cache-Control`/`Expires`. These validation
  headers enable conditional revalidation and should not be penalized.
- `rendering-strategy`: Clarified that the note about downstream checks
  (`page-size-html`, `content-start-position`) being unreliable is guidance
  for report consumers, not an implementation dependency requirement.

### v0.2.0 (2026-03-15)

- New check: `rendering-strategy` (Category 3). Detects pages that rely on
  client-side JavaScript to render content, which makes them invisible to
  most coding agents. Covers full SPA shells and the subtler case of
  statically generated pages with client-side content population.
- Check count: 21 → 22.

### v0.1.0 (2026-02-22) - Initial Draft

- Initial spec with 21 checks across 8 categories.
- Progressive disclosure recommendation for large `llms.txt` files.
- Authentication and access category: auth gate detection, alternative access
  paths, and guidance for making private docs agent-accessible.
- Known platform truncation limits (Appendix A).
- Notable exclusions with rationale (Appendix B).