Category 3: Page Size and Truncation Risk

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether page content fits within the transfer and processing limits of agent web fetch pipelines. Truncation is silent: the agent doesn’t know it’s working with partial data.

How Agents Get Content #

Not all agents see the same thing. The format an agent receives depends on the request it makes and the server’s response:

1. Agents that request markdown (Claude Code, Cursor, OpenCode send Accept: text/markdown). If the server honors this and returns markdown, the agent gets clean content. If the server also returns Content-Type: text/markdown and the content is under 100K characters, Claude Code bypasses its summarization model entirely, delivering the content directly to the agent. This is the best-case path.

2. Agents that request HTML (most agents, including Gemini, Copilot, and others, send Accept: text/html or */*). These agents receive the full HTML response. Some pipelines convert HTML to markdown before truncation (Claude Code uses Turndown); others may truncate raw HTML or use their own processing. The HTML path is where boilerplate CSS/JS causes the most damage.

3. Agents that use .md URL variants. If an agent knows to append .md to a URL (because llms.txt told it, or a directive on the page, or persistent context), it gets markdown directly regardless of Accept headers.

Because different agents hit different paths, this spec defines size checks for both the markdown response (if available) and the HTML response. A site that’s only optimized for the markdown path is leaving most agents behind.

Pipelines also differ in when they cut oversized content, and the order determines which measurement predicts the agent’s experience:

1. Convert, then truncate. Scripts and styles are stripped, HTML is converted to markdown, and the size limit applies to the converted output. Post-conversion size (page-size-html) predicts these pipelines.
2. Truncate, then convert (or ingest raw HTML directly). The size limit applies to the bytes as served, so inline scripts and serialized data consume the budget before any content does. Served size (page-size-transfer) predicts these pipelines.
3. Capped fetch. Independent of processing order, some tools cap the response bytes they will read, so a page can fail at the transfer layer before any conversion happens. Served size predicts this too.

A page can score well on one measurement while failing agents on the other, which is why the spec measures both.

For empirical observations of how specific platforms (Claude, Cursor, Copilot, Gemini, Windsurf Cascade, and others) handle retrieval, truncation, and summarization in practice, see Agent platform comparisons.

rendering-strategy #

• What it checks: Whether the HTTP response contains the page’s actual content, or whether content requires JavaScript execution to render (client-side rendering / SPA).

• Why it matters: Most coding agents fetch pages using HTTP libraries that do not execute JavaScript. GitHub Copilot is the only major agent observed to use headless browser rendering. When a site relies on client-side rendering, agents see an empty shell containing framework boilerplate, inline CSS, and navigation chrome, but none of the documentation content.

  This is not a truncation problem. It is a zero-content problem. The page returns HTTP 200, so the agent doesn’t know anything is wrong. It attempts to extract information from whatever text is in the shell (typically nav links and footer text) and produces nonsensical results, or falls back on training data that may be outdated.

  The rendering strategy is a property of the framework configuration, not the framework itself. The same framework can produce either server-rendered or client-rendered output. Sites built with Next.js, Gatsby, and Nuxt appear on both sides: react.dev (Next.js) and docs.github.com (Next.js) are fully agent-accessible, while other sites using the same frameworks deliver empty shells. Text-to-HTML ratio alone is not a reliable signal; GitHub docs and Stripe docs have low ratios due to heavy bundled assets but contain real page content. The distinction is whether page-specific content is present in the response.

  A subtler variant exists where a page is statically generated but a specific component defers content rendering to JavaScript based on user selections (e.g., query parameters choosing a language or deployment type). The static HTML contains the page structure (title, navigation, selector UI) but none of the substantive content. From an agent’s perspective, the effect is the same as a full SPA shell.

• Result levels:

  • Pass: HTTP response contains substantive page content. Detected by the presence of multiple page-specific headings, paragraphs with prose content, or other content elements beyond navigation chrome.
  • Warn: HTTP response contains some content but appears sparse relative to the page’s apparent scope. This covers client-side content population (statically generated pages where a component defers content to JavaScript), partial hydration or lazy loading, and legitimately minimal pages.
  • Fail: HTTP response is an SPA shell. Detected by the combination of known framework markers (e.g., id="___gatsby", id="__next", id="__nuxt", id="root"), minimal visible text content, and absence of page-specific content elements.

• Recommended action:

  • Warn: Verify that key content is present in the server-rendered HTML response. Pages with sparse content may rely on client-side JavaScript to populate.
  • Fail: Enable server-side rendering or pre-rendering for documentation pages. If only specific page templates use client-side content loading, target those templates rather than rebuilding the entire site.

• Automation: Heuristic. Combine framework marker detection with content signal analysis (headings, paragraphs, code blocks after stripping <script>, <style>, and <noscript> elements). Framework markers alone are not conclusive since SSR sites share the same markers.

• Notes: If this check fails, size-related checks (page-size-html, content-start-position) still run but their results should be interpreted with caution, since they are measuring a shell rather than actual content. This is a recommendation for report consumers and implementations presenting results; it does not require downstream checks to declare a dependency on rendering-strategy or alter their own pass/warn/fail logic. If the site passes markdown-url-support or content-negotiation, that provides partial mitigation: agents that request markdown may still get content even when the HTML path is broken.

page-size-markdown #

• What it checks: The character count of the page when served as markdown, via either the .md URL variant or content negotiation with Accept: text/markdown. Only runs if the site serves markdown (as detected by Category 2 checks).
• Why it matters: This is the best-case scenario for agent consumption. Markdown is what agents actually want, and it’s the format where page size most directly corresponds to what the model sees. If the markdown version fits within truncation limits, agents that can request it will get the full content.
• Result levels:
  • Pass: Under 50,000 characters (fits comfortably within all known limits, including Claude Code’s direct-delivery threshold for trusted sites).
  • Warn: Between 50,000 and 100,000 characters (fits within Claude Code’s truncation limit but may exceed others; also exceeds the direct-delivery threshold, meaning a summarization model may process it).
  • Fail: Over 100,000 characters (will be truncated by Claude Code and likely all other platforms).
• Recommended action:
  • Warn: Consider splitting large pages. Pages in this range may be truncated on some platforms or routed through a summarization model.
  • Fail: Break oversized pages into smaller ones, or restructure serialized tabbed content that inflates page size.
• Automation: Full.
• Notes: If the site doesn’t serve markdown at all, this check is skipped and page-size-html becomes the primary size check. The report should note that the site relies entirely on the HTML path.

page-size-html #

• What it checks: The character count of the HTML response, and the character count after converting HTML to markdown (simulating what an agent’s processing pipeline produces). Reports both numbers.
• Why it matters: Most agents receive HTML, not markdown. The raw HTML size determines whether the page even fits in the fetch buffer (Claude Code caps at ~10MB). The post-conversion size is closer to what the agent actually processes, but conversion pipelines vary across agents and are lossy and unpredictable. Navigation boilerplate, serialized tabbed content, and deeply nested page structure can all inflate the converted output well beyond the documentation content itself. Both raw and post-conversion sizes matter.
• Result levels (based on post-conversion size, since that’s what the model receives):
  • Pass: Converted content under 50,000 characters.
  • Warn: Converted content between 50,000 and 100,000 characters.
  • Fail: Converted content over 100,000 characters.
• Recommended action:
  • Warn: Review pages for reducible boilerplate (navigation, serialized tabbed content). Consider providing markdown versions as a smaller alternative path for agents.
  • Fail: Break large pages into smaller units, reduce navigation boilerplate, or provide markdown versions that bypass the HTML conversion overhead.
  • Markdown availability helps agents that request it but does not reduce the HTML page size itself; fixing the HTML remains important.
• Automation: Full. Convert HTML to markdown using a pipeline that approximates what agents see after their own processing. Agent pipelines vary (some strip <script>/<style> elements before conversion, others don’t; some use Turndown, others use different converters or visit pages in-browser). Implementations should document their conversion approach.
• Report details: Show both the raw HTML size and the post-conversion size. A large gap between the two indicates heavy boilerplate. Report the conversion ratio (e.g., “505KB HTML -> 12KB markdown (98% boilerplate)”) as a useful signal for site owners.

page-size-transfer #

• What it checks: The served byte size of the HTML document response: the response body after transfer decoding (decompression), which is what an agent’s HTTP client hands to its processing pipeline. Subresources (linked CSS, JavaScript, images) are not counted, because agents generally don’t fetch them. Inline scripts, styles, and serialized data payloads embedded in the document are counted, because agents can’t avoid receiving them.

• Why it matters: page-size-html measures what survives HTML-to-markdown conversion, which models pipelines that strip scripts and convert before truncating. Served size measures what every agent pays before any processing happens, and it fails differently:

  • Truncate-first and raw-ingestion pipelines consume script payload as content. For them, serialization overhead isn’t invisible; it is the page.
  • Fetch caps apply to response bytes, not converted output. A page can convert to a few kilobytes of clean markdown and still exceed the byte budget of the tool fetching it (Claude Code’s fetch buffer caps at ~10MB).
  • Bandwidth and latency costs apply to every fetch regardless of pipeline, and multi-page reading sessions multiply them.

  Modern server-rendering frameworks can make the gap between served bytes and content arbitrarily large. In measurements of production documentation sites on one hosted platform, pages shipped 75-84% of their bytes as serialized framework payloads inside inline script tags: component trees, resolved metadata, and a complete duplicate of the page’s markdown source. Served-bytes-to-content ratios ran from 40:1 to 200:1. Those pages score well on page-size-html because conversion strips the payload, while every agent fetching them transfers half a megabyte to several megabytes per page.

• Result levels:

  • Pass: Served size under 1MB.
  • Warn: Served size between 1MB and 10MB. No documented cap is exceeded, but truncate-first and raw-ingestion pipelines are consuming mostly non-content bytes, and multi-page sessions pay a real bandwidth and latency cost.
  • Fail: Served size over 10MB. This exceeds the only well-documented transfer cap (Claude Code’s fetch buffer); content beyond the cap is unreachable regardless of how the pipeline processes it.

• Recommended action:

  • Warn: Identify what the non-content bytes are; in practice they are usually inline serialization (framework hydration payloads, embedded duplicate page source, resolved data objects) visible in the page source. Avoid shipping the same content twice in different formats, load large data payloads on demand, and confirm markdown variants are available and discoverable so agents have a cheaper path.
  • Fail: Same actions, urgently. At this size, at least one major platform cuts the page off at the transfer layer.
  • Markdown availability gives agents that discover it an escape hatch but does not reduce what HTML-path agents transfer; fixing the served payload remains important.

• Automation: Full. Fetch with an Accept-Encoding typical of agent HTTP clients, decode the response, and measure the decoded body. Implementations may also report the on-the-wire (compressed) size where available; serialized payloads compress well, so wire size understates the processing burden.

• Report details: Show served bytes alongside the post-conversion content size from page-size-html, and report the ratio between them (e.g., “3.4MB served -> 29KB content (~120:1)”). A high ratio on a large page is an architecture signature (hydration payloads, embedded duplicate content) rather than a content problem; it tells the site owner the fix lives in framework configuration, not in the docs themselves.

• Notes: This check complements rendering-strategy, which catches pages that ship too little server-rendered content. This check catches the opposite failure: pages that render content fine but ship many times its weight in serialization overhead. Byte-level caps are less documented than character-level truncation limits, so the default thresholds here are conservative and should be configurable (see Appendix A).

content-start-position #

• What it checks: How far into the post-conversion content (by character count and as a percentage) the actual documentation content begins.
• Why it matters: After HTML-to-markdown conversion, boilerplate often survives. Navigation menus, breadcrumbs, sidebars, and footer content all convert to text that precedes or surrounds the actual documentation. Depending on the agent’s conversion pipeline, inline CSS and JavaScript may also survive as raw text. If this boilerplate consumes most of the truncation budget, the agent never sees the documentation content. In one observed case, actual content didn’t start until 87% of the way through the output (441K characters of CSS before the first paragraph).
• Result levels:
  • Pass: Content starts within the first 10% of the post-conversion output.
  • Warn: Content starts between 10% and 50%.
  • Fail: Content starts after 50%.
• Recommended action: Reduce navigation, breadcrumb, and sidebar markup that precedes the content area. Agents may never see the documentation content if boilerplate consumes most of the truncation budget.
• Automation: Heuristic. Use the same conversion pipeline as page-size-html, then detect the first meaningful content element (heading, paragraph with prose) past any boilerplate (navigation text, breadcrumbs, sidebar content, inline CSS/JS that survived conversion).
• Notes: This check only applies to the HTML path. Markdown served directly by the site should not have boilerplate preamble; if it does, that’s a separate issue worth flagging but not something this check targets.

single-fetch-completeness #

• What it checks: Whether a markdown response delivers its complete content in one fetch, and when it doesn’t, whether the continuation is machine-followable: declared where agents will see it, linked with an absolute URL, and actually working.

• Why it matters: Pagination is application-level truncation, and it is quieter than the platform truncation this category otherwise measures. A paginated markdown response looks complete: it is well-formed, ends cleanly, and returns 200. The signals that it is partial are easy for agent pipelines to lose:

  • Summarization pipelines process fetched content through a smaller model before the orchestrating agent sees it. A pagination note may or may not survive summarization, and the summarizer cannot perform a follow-up fetch itself; the orchestrator would have to notice the note and choose to fetch again.
  • Trailing pagination notes sit at the end of the content, which is the first region lost to platform truncation. A truncated response loses the only indication that it was also paginated.
  • RAG pipelines chunk fetched content for retrieval. A pagination marker lands in one chunk, unrelated to the content it describes, and effectively disappears.

  This failure was observed in production on a model catalog’s markdown variant: 100 of 102 entries shown, a pagination note at the bottom of the file, a continuation URL that was root-relative rather than absolute, and, when fetched, a continuation response that returned an empty body. An agent fetching that page gets 98% of the catalog and no working way to learn what’s missing.

  Notably, pagination in a markdown variant is often inherited from the HTML UI rather than needed by the markdown itself. The catalog above paginates at 100 entries per page, but the complete set serializes to roughly 32,000 characters, comfortably under this spec’s 50,000-character pass threshold. The markdown variant imported an interaction pattern from a surface that has interaction; markdown doesn’t.

• Result levels:

  • Pass: The response is complete in one fetch (no pagination signals detected), or pagination exists and the continuation is declared at the top of the content with an absolute URL that resolves to the next segment.
  • Warn: Pagination exists and the continuation works, but is fragile: declared only at the bottom of the content, linked with a relative URL, or discoverable only from response headers.
  • Fail: The content is partial and the continuation is missing, relative and unresolvable, or broken (non-success status, empty body, or a soft 404).

• Recommended action:

  • Warn: Move the continuation declaration to the top of the content (before anything truncation could remove) and make continuation links absolute.
  • Fail: First ask whether the markdown variant needs pagination at all: complete content that fits within this spec’s size thresholds should be served in one response, even when the HTML UI paginates. If pagination is genuinely necessary, declare it at the top with absolute links, and verify the continuation URLs actually serve content.

• Automation: Heuristic. Detect pagination signals in markdown responses: “N of M” phrasing, links or instructions containing pagination query parameters (?page=, ?offset=), “next page” link text, and Link: rel="next" response headers. When signals are found, fetch the continuation and verify it returns substantive content of the expected representation. Absence of signals is treated as complete; a page that omits content with no marker at all is not detectable by this check (see markdown-content-parity for the cross-representation comparison that can catch it).

• Notes: Only applies to markdown responses (.md variants, content negotiation, and llms.txt-linked markdown). HTML pagination is an interaction pattern agents share with human readers and is out of scope here.

  Relationship to the spec’s splitting recommendations. This check targets windowing rather than the number of fetches. Other parts of this spec recommend more fetches: progressive disclosure splits an oversized llms.txt into section files, and the page-size checks recommend breaking large pages up. The difference is what each fetch returns. Splitting creates self-contained units, each with its own topic and URL, reached by navigation: an agent fetches an index, chooses the relevant unit, and gets content that is complete as the thing it claims to be. Pagination slices one logical unit into arbitrary windows: no window is complete for any question, every window is required to have the unit at all, and the seams are easy to lose. Multi-fetch by choice is navigation; multi-fetch by obligation is truncation with extra steps. The principle behind both: split by meaning, don’t window by size. A response should either be complete as what it claims to be, or say so plainly where agents will see it.