# Category 3: Page Size and Truncation Risk

> For AI agents: the complete documentation index is at [https://agentdocsspec.com/llms.txt](https://agentdocsspec.com/llms.txt). Markdown versions of any page are available by appending `index.md` to the URL path.

Part of the [Web Documentation Delivery Spec](https://agentdocsspec.com/spec/web/). The [Checks Summary](https://agentdocsspec.com/spec/web/#checks-summary) lists all checks with links to their definitions.

These checks evaluate whether page content fits within the transfer and
processing limits of agent web fetch pipelines. Truncation is silent: the
agent doesn't know it's working with partial data.

### How Agents Get Content

Not all agents see the same thing. The format an agent receives depends on the
request it makes and the server's response:

1. **Agents that request markdown** (Claude Code, Cursor, OpenCode send
   `Accept: text/markdown`). If the server honors this and returns markdown,
   the agent gets clean content. If the server also returns
   `Content-Type: text/markdown` and the content is under 100K characters,
   Claude Code bypasses its summarization model entirely, delivering the
   content directly to the agent. This is the best-case path.

2. **Agents that request HTML** (most agents, including Gemini, Copilot, and
   others, send `Accept: text/html` or `*/*`). These agents receive the full
   HTML response. Some pipelines convert HTML to markdown before truncation
   (Claude Code uses Turndown); others may truncate raw HTML or use their own
   processing. The HTML path is where boilerplate CSS/JS causes the most
   damage.

3. **Agents that use `.md` URL variants.** If an agent knows to append `.md`
   to a URL (because `llms.txt` told it, or a directive on the page, or
   persistent context), it gets markdown directly regardless of Accept headers.

Because different agents hit different paths, this spec defines size checks for
**both** the markdown response (if available) and the HTML response. A site
that's only optimized for the markdown path is leaving most agents behind.

Pipelines also differ in **when** they cut oversized content, and the order
determines which measurement predicts the agent's experience:

1. **Convert, then truncate.** Scripts and styles are stripped, HTML is
   converted to markdown, and the size limit applies to the converted output.
   Post-conversion size (`page-size-html`) predicts these pipelines.
2. **Truncate, then convert** (or ingest raw HTML directly). The size limit
   applies to the bytes as served, so inline scripts and serialized data
   consume the budget before any content does. Served size
   (`page-size-transfer`) predicts these pipelines.
3. **Capped fetch.** Independent of processing order, some tools cap the
   response bytes they will read, so a page can fail at the transfer layer
   before any conversion happens. Served size predicts this too.

A page can score well on one measurement while failing agents on the other,
which is why the spec measures both.

For empirical observations of how specific platforms (Claude, Cursor, Copilot,
Gemini, Windsurf Cascade, and others) handle retrieval, truncation, and
summarization in practice, see [Agent platform comparisons](https://agentdocsspec.com/platforms/).

### `rendering-strategy`

- **What it checks**: Whether the HTTP response contains the page's actual
  content, or whether content requires JavaScript execution to render
  (client-side rendering / SPA).
- **Why it matters**: Most coding agents fetch pages using HTTP libraries that
  do not execute JavaScript. GitHub Copilot is the only major agent observed
  to use headless browser rendering. When a site relies on client-side
  rendering, agents see an empty shell containing framework boilerplate,
  inline CSS, and navigation chrome, but none of the documentation content.

  This is not a truncation problem. It is a zero-content problem. The page
  returns HTTP 200, so the agent doesn't know anything is wrong. It attempts
  to extract information from whatever text is in the shell (typically nav
  links and footer text) and produces nonsensical results, or falls back on
  training data that may be outdated.

  The rendering strategy is a property of the framework configuration, not
  the framework itself. The same framework can produce either server-rendered
  or client-rendered output. Sites built with Next.js, Gatsby, and Nuxt
  appear on both sides: react.dev (Next.js) and docs.github.com (Next.js)
  are fully agent-accessible, while other sites using the same frameworks
  deliver empty shells. Text-to-HTML ratio alone is not a reliable signal;
  GitHub docs and Stripe docs have low ratios due to heavy bundled assets
  but contain real page content. The distinction is whether page-specific
  content is present in the response.

  A subtler variant exists where a page is statically generated but a
  specific component defers content rendering to JavaScript based on user
  selections (e.g., query parameters choosing a language or deployment type).
  The static HTML contains the page structure (title, navigation, selector
  UI) but none of the substantive content. From an agent's perspective, the
  effect is the same as a full SPA shell.

- **Result levels**:
  - **Pass**: HTTP response contains substantive page content. Detected by
    the presence of multiple page-specific headings, paragraphs with prose
    content, or other content elements beyond navigation chrome.
  - **Warn**: HTTP response contains some content but appears sparse relative
    to the page's apparent scope. This covers client-side content population
    (statically generated pages where a component defers content to
    JavaScript), partial hydration or lazy loading, and legitimately minimal
    pages.
  - **Fail**: HTTP response is an SPA shell. Detected by the combination of
    known framework markers (e.g., `id="___gatsby"`, `id="__next"`,
    `id="__nuxt"`, `id="root"`), minimal visible text content, and absence
    of page-specific content elements.
- **Recommended action**:
  - **Warn**: Verify that key content is present in the server-rendered HTML
    response. Pages with sparse content may rely on client-side JavaScript
    to populate.
  - **Fail**: Enable server-side rendering or pre-rendering for documentation
    pages. If only specific page templates use client-side content loading,
    target those templates rather than rebuilding the entire site.
- **Automation**: Heuristic. Combine framework marker detection with content
  signal analysis (headings, paragraphs, code blocks after stripping
  `<script>`, `<style>`, and `<noscript>` elements). Framework markers alone
  are not conclusive since SSR sites share the same markers.
- **Notes**: If this check fails, size-related checks (`page-size-html`,
  `content-start-position`) still run but their results should be
  interpreted with caution, since they are measuring a shell rather than
  actual content. This is a recommendation for report consumers and
  implementations presenting results; it does not require downstream
  checks to declare a dependency on `rendering-strategy` or alter their
  own pass/warn/fail logic. If the site passes `markdown-url-support` or
  `content-negotiation`, that provides partial mitigation: agents that
  request markdown may still get content even when the HTML path is broken.

### `page-size-markdown`

- **What it checks**: The character count of the page when served as markdown,
  via either the `.md` URL variant or content negotiation with
  `Accept: text/markdown`. Only runs if the site serves markdown (as detected
  by Category 2 checks).
- **Why it matters**: This is the best-case scenario for agent consumption.
  Markdown is what agents actually want, and it's the format where page size
  most directly corresponds to what the model sees. If the markdown version
  fits within truncation limits, agents that can request it will get the full
  content.
- **Result levels**:
  - **Pass**: Under 50,000 characters (fits comfortably within all known
    limits, including Claude Code's direct-delivery threshold for trusted
    sites).
  - **Warn**: Between 50,000 and 100,000 characters (fits within Claude Code's
    truncation limit but may exceed others; also exceeds the direct-delivery
    threshold, meaning a summarization model may process it).
  - **Fail**: Over 100,000 characters (will be truncated by Claude Code and
    likely all other platforms).
- **Recommended action**:
  - **Warn**: Consider splitting large pages. Pages in this range may be
    truncated on some platforms or routed through a summarization model.
  - **Fail**: Break oversized pages into smaller ones, or restructure
    serialized tabbed content that inflates page size.
- **Automation**: Full.
- **Notes**: If the site doesn't serve markdown at all, this check is skipped
  and `page-size-html` becomes the primary size check. The report should note
  that the site relies entirely on the HTML path.

### `page-size-html`

- **What it checks**: The character count of the HTML response, and the
  character count after converting HTML to markdown (simulating what an
  agent's processing pipeline produces). Reports both numbers.
- **Why it matters**: Most agents receive HTML, not markdown. The raw HTML size
  determines whether the page even fits in the fetch buffer (Claude Code caps
  at ~10MB). The post-conversion size is closer to what the agent actually
  processes, but conversion pipelines vary across agents and are lossy and
  unpredictable. Navigation boilerplate, serialized tabbed content, and
  deeply nested page structure can all inflate the converted output well
  beyond the documentation content itself. Both raw and post-conversion
  sizes matter.
- **Result levels** (based on post-conversion size, since that's what the
  model receives):
  - **Pass**: Converted content under 50,000 characters.
  - **Warn**: Converted content between 50,000 and 100,000 characters.
  - **Fail**: Converted content over 100,000 characters.
- **Recommended action**:
  - **Warn**: Review pages for reducible boilerplate (navigation, serialized
    tabbed content). Consider providing markdown versions as a smaller
    alternative path for agents.
  - **Fail**: Break large pages into smaller units, reduce navigation
    boilerplate, or provide markdown versions that bypass the HTML conversion
    overhead.
  - Markdown availability helps agents that request it but does not reduce
    the HTML page size itself; fixing the HTML remains important.
- **Automation**: Full. Convert HTML to markdown using a pipeline that
  approximates what agents see after their own processing. Agent pipelines
  vary (some strip `<script>`/`<style>` elements before conversion, others
  don't; some use Turndown, others use different converters or visit pages
  in-browser). Implementations should document their conversion approach.
- **Report details**: Show both the raw HTML size and the post-conversion size.
  A large gap between the two indicates heavy boilerplate. Report the
  conversion ratio (e.g., "505KB HTML -> 12KB markdown (98% boilerplate)")
  as a useful signal for site owners.

### `page-size-transfer`

- **What it checks**: The served byte size of the HTML document response:
  the response body after transfer decoding (decompression), which is what
  an agent's HTTP client hands to its processing pipeline. Subresources
  (linked CSS, JavaScript, images) are not counted, because agents generally
  don't fetch them. Inline scripts, styles, and serialized data payloads
  embedded in the document are counted, because agents can't avoid receiving
  them.
- **Why it matters**: `page-size-html` measures what survives HTML-to-markdown
  conversion, which models pipelines that strip scripts and convert before
  truncating. Served size measures what every agent pays before any
  processing happens, and it fails differently:
  - **Truncate-first and raw-ingestion pipelines** consume script payload as
    content. For them, serialization overhead isn't invisible; it is the
    page.
  - **Fetch caps** apply to response bytes, not converted output. A page can
    convert to a few kilobytes of clean markdown and still exceed the byte
    budget of the tool fetching it (Claude Code's fetch buffer caps at
    ~10MB).
  - **Bandwidth and latency costs** apply to every fetch regardless of
    pipeline, and multi-page reading sessions multiply them.

  Modern server-rendering frameworks can make the gap between served bytes
  and content arbitrarily large. In measurements of production documentation
  sites on one hosted platform, pages shipped 75-84% of their bytes as
  serialized framework payloads inside inline script tags: component trees,
  resolved metadata, and a complete duplicate of the page's markdown source.
  Served-bytes-to-content ratios ran from 40:1 to 200:1. Those pages score
  well on `page-size-html` because conversion strips the payload, while
  every agent fetching them transfers half a megabyte to several megabytes
  per page.
- **Result levels**:
  - **Pass**: Served size under 1MB.
  - **Warn**: Served size between 1MB and 10MB. No documented cap is
    exceeded, but truncate-first and raw-ingestion pipelines are consuming
    mostly non-content bytes, and multi-page sessions pay a real bandwidth
    and latency cost.
  - **Fail**: Served size over 10MB. This exceeds the only well-documented
    transfer cap (Claude Code's fetch buffer); content beyond the cap is
    unreachable regardless of how the pipeline processes it.
- **Recommended action**:
  - **Warn**: Identify what the non-content bytes are; in practice they are
    usually inline serialization (framework hydration payloads, embedded
    duplicate page source, resolved data objects) visible in the page
    source. Avoid shipping the same content twice in different formats,
    load large data payloads on demand, and confirm markdown variants are
    available and discoverable so agents have a cheaper path.
  - **Fail**: Same actions, urgently. At this size, at least one major
    platform cuts the page off at the transfer layer.
  - Markdown availability gives agents that discover it an escape hatch but
    does not reduce what HTML-path agents transfer; fixing the served
    payload remains important.
- **Automation**: Full. Fetch with an `Accept-Encoding` typical of agent
  HTTP clients, decode the response, and measure the decoded body.
  Implementations may also report the on-the-wire (compressed) size where
  available; serialized payloads compress well, so wire size understates
  the processing burden.
- **Report details**: Show served bytes alongside the post-conversion
  content size from `page-size-html`, and report the ratio between them
  (e.g., "3.4MB served -> 29KB content (~120:1)"). A high ratio on a large
  page is an architecture signature (hydration payloads, embedded duplicate
  content) rather than a content problem; it tells the site owner the fix
  lives in framework configuration, not in the docs themselves.
- **Notes**: This check complements `rendering-strategy`, which catches pages
  that ship too little server-rendered content. This check catches the
  opposite failure: pages that render content fine but ship many times its
  weight in serialization overhead. Byte-level caps are less documented than
  character-level truncation limits, so the default thresholds here are
  conservative and should be configurable (see
  [Appendix A](https://agentdocsspec.com/spec/web/appendices/#appendix-a-known-platform-truncation-limits)).

### `content-start-position`

- **What it checks**: How far into the **post-conversion** content (by character
  count and as a percentage) the actual documentation content begins.
- **Why it matters**: After HTML-to-markdown conversion, boilerplate often
  survives. Navigation menus, breadcrumbs, sidebars, and footer content all
  convert to text that precedes or surrounds the actual documentation. Depending
  on the agent's conversion pipeline, inline CSS and JavaScript may also survive
  as raw text. If this boilerplate consumes most of the truncation budget, the
  agent never sees the documentation content. In one observed case, actual
  content didn't start until 87% of the way through the output (441K characters
  of CSS before the first paragraph).
- **Result levels**:
  - **Pass**: Content starts within the first 10% of the post-conversion
    output.
  - **Warn**: Content starts between 10% and 50%.
  - **Fail**: Content starts after 50%.
- **Recommended action**: Reduce navigation, breadcrumb, and sidebar markup
  that precedes the content area. Agents may never see the documentation
  content if boilerplate consumes most of the truncation budget.
- **Automation**: Heuristic. Use the same conversion pipeline as
  `page-size-html`, then detect the first meaningful content element (heading,
  paragraph with prose) past any boilerplate (navigation text, breadcrumbs,
  sidebar content, inline CSS/JS that survived conversion).
- **Notes**: This check only applies to the HTML path. Markdown served directly
  by the site should not have boilerplate preamble; if it does, that's a
  separate issue worth flagging but not something this check targets.

### `single-fetch-completeness`

- **What it checks**: Whether a markdown response delivers its complete
  content in one fetch, and when it doesn't, whether the continuation is
  machine-followable: declared where agents will see it, linked with an
  absolute URL, and actually working.
- **Why it matters**: Pagination is application-level truncation, and it is
  quieter than the platform truncation this category otherwise measures. A
  paginated markdown response looks complete: it is well-formed, ends
  cleanly, and returns 200. The signals that it is partial are easy for
  agent pipelines to lose:
  - **Summarization pipelines** process fetched content through a smaller
    model before the orchestrating agent sees it. A pagination note may or
    may not survive summarization, and the summarizer cannot perform a
    follow-up fetch itself; the orchestrator would have to notice the note
    and choose to fetch again.
  - **Trailing pagination notes** sit at the end of the content, which is
    the first region lost to platform truncation. A truncated response
    loses the only indication that it was also paginated.
  - **RAG pipelines** chunk fetched content for retrieval. A pagination
    marker lands in one chunk, unrelated to the content it describes, and
    effectively disappears.

  This failure was observed in production on a model catalog's markdown
  variant: 100 of 102 entries shown, a pagination note at the bottom of the
  file, a continuation URL that was root-relative rather than absolute, and,
  when fetched, a continuation response that returned an empty body. An
  agent fetching that page gets 98% of the catalog and no working way to
  learn what's missing.

  Notably, pagination in a markdown variant is often inherited from the
  HTML UI rather than needed by the markdown itself. The catalog above
  paginates at 100 entries per page, but the complete set serializes to
  roughly 32,000 characters, comfortably under this spec's 50,000-character
  pass threshold. The markdown variant imported an interaction pattern from
  a surface that has interaction; markdown doesn't.
- **Result levels**:
  - **Pass**: The response is complete in one fetch (no pagination signals
    detected), or pagination exists and the continuation is declared at the
    top of the content with an absolute URL that resolves to the next
    segment.
  - **Warn**: Pagination exists and the continuation works, but is fragile:
    declared only at the bottom of the content, linked with a relative URL,
    or discoverable only from response headers.
  - **Fail**: The content is partial and the continuation is missing,
    relative and unresolvable, or broken (non-success status, empty body,
    or a soft 404).
- **Recommended action**:
  - **Warn**: Move the continuation declaration to the top of the content
    (before anything truncation could remove) and make continuation links
    absolute.
  - **Fail**: First ask whether the markdown variant needs pagination at
    all: complete content that fits within this spec's size thresholds
    should be served in one response, even when the HTML UI paginates. If
    pagination is genuinely necessary, declare it at the top with absolute
    links, and verify the continuation URLs actually serve content.
- **Automation**: Heuristic. Detect pagination signals in markdown
  responses: "N of M" phrasing, links or instructions containing pagination
  query parameters (`?page=`, `?offset=`), "next page" link text, and
  `Link: rel="next"` response headers. When signals are found, fetch the
  continuation and verify it returns substantive content of the expected
  representation. Absence of signals is treated as complete; a page that
  omits content with no marker at all is not detectable by this
  check (see `markdown-content-parity` for the cross-representation
  comparison that can catch it).
- **Notes**: Only applies to markdown responses (`.md` variants, content
  negotiation, and `llms.txt`-linked markdown). HTML pagination is an
  interaction pattern agents share with human readers and is out of scope
  here.

  **Relationship to the spec's splitting recommendations.** This check
  targets windowing rather than the number of fetches. Other parts
  of this spec recommend *more* fetches: progressive disclosure splits an
  oversized `llms.txt` into section files, and the page-size checks
  recommend breaking large pages up. The difference is what each fetch
  returns. Splitting creates self-contained units, each with its own topic
  and URL, reached by navigation: an agent fetches an index, chooses the
  relevant unit, and gets content that is complete as the thing it claims
  to be. Pagination slices one logical unit into arbitrary windows: no
  window is complete for any question, every window is required to have the
  unit at all, and the seams are easy to lose. Multi-fetch by choice is
  navigation; multi-fetch by obligation is truncation with extra steps. The
  principle behind both: split by meaning, don't window by size. A response
  should either be complete as what it claims to be, or say so plainly
  where agents will see it.