Appendices

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

Appendix A: Known Platform Truncation Limits #

The thresholds used in this spec’s pass/warn/fail levels are derived from observed and documented platform behavior. This appendix tracks known limits so that implementations can calibrate their thresholds appropriately, and so that the spec’s default thresholds can be updated as more data becomes available.

Thresholds Used in This Spec #

The spec uses two threshold tiers across its size-related checks:

• 50,000 characters: The “pass” threshold. Content under this size fits comfortably within all known platform limits.
• 100,000 characters: The “fail” threshold. Content over this size will be truncated by Claude Code and likely by most other platforms.

These are conservative defaults based on the best-documented platform (Claude Code). Implementations should allow these thresholds to be configurable so users can evaluate against specific platform limits or adjust as new data becomes available.

page-size-transfer uses byte thresholds (1MB warn, 10MB fail) rather than character thresholds, because it measures the response before any processing. The 10MB fail line is anchored to Claude Code’s documented fetch buffer; byte-level caps on other platforms are less documented than character-level truncation limits, so these defaults are conservative and should likewise be configurable.

Known Platform Limits #

Compare platform architecture and truncation limits in Platforms.

What This Means for Threshold Selection #

The MCP Fetch reference server’s default of 5,000 characters is worth noting. Many agent setups use MCP-based fetch tools, and if users haven’t changed the default, they’re working with a limit 20x smaller than Claude Code’s. A page that passes at the 50K threshold may still be unusable for MCP Fetch users with default settings.

Implementations may want to support named profiles (e.g., --profile claude-code, --profile mcp-default) that set thresholds to match specific platforms, in addition to allowing custom threshold values.

Appendix B: Notable Exclusions #

This section documents topics that were considered for the spec but intentionally excluded, along with the rationale.

robots.txt and AI User-Agent Blocking #

robots.txt can block known AI training crawlers (ClaudeBot, GPTBot, Google-Extended, etc.) that identify themselves via user-agent strings. However, this is a crawling policy concern rather than an agent-friendliness concern, and the two audiences are distinct.

Training crawlers and coding agents are different request paths with different user-agents. The agents this spec targets (coding assistants fetching docs during real-time workflows) are largely invisible to robots.txt:

Source: Checkly, “State of AI Agent Content Negotiation”

Most coding agents use standard browser user-agent strings and are indistinguishable from human traffic. A site blocking ClaudeBot in robots.txt is blocking Anthropic’s training crawler, not Claude Code fetching a docs page. Since this spec is about making documentation accessible to agents in real-time workflows, robots.txt configuration is out of scope.

Behavioral bot protection is a different matter. Because most coding agents don’t identify themselves, bot-management systems that act on traffic behavior rather than declared identity affect them even when crawling policy does not. That failure mode is in scope; see bot-protection-interference.

GitHub Raw URL Fallback #

GitHub raw URLs (raw.githubusercontent.com/...) were observed to be the single most reliable documentation access pattern in practice. When official docs failed (rate-limited, JavaScript-rendered, or hard to navigate), GitHub was almost always a viable fallback.

However, this is a fallback strategy for agent users rather than a property of the documentation site itself. Whether a project’s docs source happens to be on GitHub, and whether the raw content there is usable as standalone documentation, is outside the control of a docs site evaluation. This spec focuses on what documentation site owners can do to improve agent accessibility of their own sites.