# Category 4: Content Structure

> For AI agents: the complete documentation index is at [https://agentdocsspec.com/llms.txt](https://agentdocsspec.com/llms.txt). Markdown versions of any page are available by appending `index.md` to the URL path.

Part of the [Web Documentation Delivery Spec](https://agentdocsspec.com/spec/web/). The [Checks Summary](https://agentdocsspec.com/spec/web/#checks-summary) lists all checks with links to their definitions.

These checks evaluate whether page content is structured in ways that agents can
effectively consume. These are harder to fully automate and rely more on
heuristics.

### `tabbed-content-serialization`

- **What it checks**: Whether pages use tabbed, accordion, or dropdown UI
  patterns that serialize into long sequential content in the source, and if
  so, how large the serialized output is.
- **Why it matters**: Tabbed content is great for humans but can be catastrophic
  for agents. A tutorial with 11 language variants serializes into a single
  massive document where an agent might see only the first 1-3 variants. Source
  order determines what the agent sees; everything past the truncation point is
  invisible. Asking for a specific variant (e.g., Python) does not help if that
  variant is beyond the truncation point.
- **Result levels**:
  - **Pass**: No tabbed content, or tabbed content that serializes to under
    50,000 characters total.
  - **Warn**: Tabbed content serializes to 50,000-100,000 characters.
  - **Fail**: Tabbed content serializes to over 100,000 characters.
- **Recommended action**: Break tab variants into separate pages, or provide
  a mechanism for agents to request specific variants. Agents see only the
  first few variants; content in later tabs is truncated.
- **Automation**: Heuristic. Detect common tab/accordion component patterns
  (e.g., `<Tab>`, `<Tabs>`, role="tabpanel", common CSS class patterns) and
  estimate serialized size.

### `section-header-quality`

- **What it checks**: Whether section headers contain enough context to be
  meaningful without the surrounding UI. Specifically, when tabbed content is
  serialized, do headers distinguish which variant (language, platform,
  deployment type) a section belongs to?
- **Why it matters**: When an agent sees serialized tabbed content, descriptive
  headers are the only way it can tell which section applies to which context.
  Generic headers like "Step 1" repeated across all variants are
  indistinguishable. Headers like "Step 1 (Python/PyMongo)" preserve the
  filtering context that the UI provided to human readers.
- **Result levels** (evaluated both within individual tab groups and across
  tab groups on the same page; the overall result is the worst of both):
  - **Pass**: <=25% of headers within tabbed sections are generic (repeated
    across variants without distinguishing context).
  - **Warn**: 25-50% of headers are generic across variants.
  - **Fail**: >50% of headers are generic, or identical header sets are
    repeated across separate tab groups on the same page with no variant
    context.
  These thresholds are defaults; implementations should allow them to be
  configured.
- **Recommended action**: Add variant context to headers (e.g., "Step 1
  (Python)" instead of "Step 1") so agents can distinguish which section
  belongs to which variant when content is serialized.
- **Automation**: Heuristic. Requires detecting tabbed sections and analyzing
  header patterns within them.

### `markdown-code-fence-validity`

- **What it checks**: Whether markdown content contains unclosed or improperly
  nested code fences (`` ``` `` or `~~~` blocks without a matching closing
  delimiter).
- **Why it matters**: An unclosed code fence causes everything after it to be
  interpreted as code rather than prose. The agent sees documentation text,
  API descriptions, and instructions as if they were inside a code block,
  which fundamentally changes how it processes the content. A model treats
  code blocks as literal content to reproduce or analyze, not as natural
  language instructions to follow. If an unclosed fence appears early in a
  page, the agent effectively loses the rest of the document's meaning. This
  applies to any markdown the site serves directly: pages via `.md` URLs or
  content negotiation, and `llms.txt` files themselves.
- **Result levels**:
  - **Pass**: All code fences in the markdown content are properly opened and
    closed.
  - **Fail**: One or more unclosed code fences detected.
- **Recommended action**: Ensure every opening `` ``` `` or `~~~` has a
  matching closing delimiter. Everything after an unclosed fence is
  interpreted as code, causing agents to misread documentation as literal
  content.
- **Notes on delimiter matching**: Per the CommonMark spec, a backtick fence
  (`` ``` ``) can only be closed by another backtick fence of equal or greater
  length, and likewise for tilde fences (`~~~`). Opening with `` ``` `` and
  attempting to close with `~~~` leaves the backtick fence unclosed. There is
  no intermediate "mismatched but balanced" state; mismatched delimiters
  produce unclosed fences and should be reported as failures.
- **Automation**: Full. Parse the markdown for fence delimiters (`` ``` `` and
  `~~~`, with optional info strings) and verify each opening delimiter has a
  matching close. Run against markdown served via `.md` URLs, content
  negotiation responses, and `llms.txt` files.
- **Notes**: This check applies to markdown the site authors and serves
  directly. Code fences broken by an HTML-to-markdown conversion pipeline are
  outside the site owner's control, though implementations may optionally flag
  them as informational findings.

### `markdown-link-portability`

- **What it checks**: Whether links in served markdown are absolute URLs, and
  whether a sample of them resolves to the representation they promise (a
  `.md` link returns markdown content, not an HTML error page).
- **Why it matters**: Relative URL resolution is well-defined (RFC 3986), but
  it requires knowing the base URL, and agent pipelines routinely lose it. A
  browser always carries the base; markdown fetched by an agent passes
  through summarization models, gets chunked for RAG, or gets pasted into a
  context where the source URL is gone. Once the base is lost, a
  root-relative link is unreconstructable and a path-relative link is
  meaningless. This spec already recommends absolute URLs between `llms.txt`
  levels for the same reason; served markdown deserves the same rule.

  Link verification must go beyond status codes. In one observed production
  case, a catalog's markdown variant emitted over 100 well-formatted links
  that all pointed into a wrong internal path prefix, apparently a build-time
  substitution error. Every link returned 200 with a body. The body was an
  HTML SPA shell whose only acknowledgment of failure was a serialized
  framework error digest inside script payload: a soft 404 served as HTML at
  a `.md` URL. A checker (or agent) that tested status codes alone would
  conclude the links worked; checking the `Content-Type` header alone would
  have caught it.
- **Result levels**:
  - **Pass**: Links are absolute URLs, and sampled links resolve to the
    expected representation.
  - **Warn**: Links are root-relative (resolvable while the base URL is
    known, fragile once it isn't), or sampled links resolve with minor
    mismatches (e.g., a `.md` link that redirects to an HTML page with the
    right content).
  - **Fail**: Links are path-relative, or sampled links are broken: hard
    404s, soft 404s, or a content type that contradicts the link (`.md`
    links returning HTML shells).
- **Recommended action**:
  - **Warn**: Emit absolute URLs when generating markdown variants; the
    site's canonical host is known at build time.
  - **Fail**: Fix the link generation first, then make the links absolute.
    Verify generated links in CI by fetching a sample and checking both
    status and content type; a link set that is generated is a link set
    that can break wholesale.
- **Automation**: Full. Parse links from served markdown, classify as
  absolute, root-relative, or path-relative, resolve a sample against the
  fetch URL, and verify status, `Content-Type`, and soft-404 heuristics
  (reusing the detection from `http-status-codes`).
- **Notes**: Applies to markdown served via `.md` URLs, content
  negotiation, and pages discovered through `llms.txt` links. Links in
  `llms.txt` itself are covered by `llms-txt-links-resolve`, which applies
  the same representation verification. Same-document fragment links
  (`#anchor` with no path) are exempt from the absolute-URL requirement:
  they resolve within the content the agent already holds, and rewriting
  them to absolute URLs adds nothing.

  **Why this check exempts the HTML path.** Relative links in HTML are
  correct web practice (they are what makes staging domains, mirrors, and
  CDN setups work), and the HTML path does not need the site's help: a
  pipeline converting HTML to markdown still holds the fetch URL at
  conversion time, so resolving relative links is the pipeline's job, with
  full information. The base URL is only lost downstream of conversion.
  Served markdown is different on both ends: the generator knows the
  canonical host, so absolute links are free to produce, and on the
  best-case consumption path (direct delivery of `text/markdown` under the
  summarization threshold) the site's bytes reach the model verbatim, with
  no conversion step where anything could be resolved. Tool builders
  implementing fetch pipelines should resolve relative links during
  HTML-to-markdown conversion, and likewise when ingesting raw HTML;
  converted content with relative links has all the same failure modes as
  served markdown once it leaves the converter.

### `embedded-data-serialization`

- **What it checks**: Whether machine-generated bulk data (large uniform
  tables, inline JSON or data blobs, base64 payloads) dominates a page's
  converted content, and attributes the page's size to the specific elements
  responsible.
- **Why it matters**: Dynamic widgets (compatibility matrices, model
  catalogs, spec browsers, pricing tables) flatten into static content when
  a page is rendered for agents. The result can be a page whose size wildly
  exceeds what its author believes they wrote: the author sees a few
  paragraphs and a widget; the built page carries hundreds of serialized
  rows under them. The size checks in Category 3 catch the symptom but
  don't explain it, and without attribution the person who can fix the page
  has no idea what to fix, or that anything is wrong at all.

  In one measured production case, a reference page served 302KB of HTML of
  which 64% was table markup, including a single generated table of 218
  rows. The page converts to roughly 83,000 characters (this spec's warn
  band), while its non-table prose totals about 17,000 characters. The
  page's truncation risk is entirely a property of its generated tables,
  which is invisible in an aggregate size number.
- **Result levels**:
  - **Pass**: No bulk-data elements detected, or bulk elements are present
    but the page passes the Category 3 size checks regardless.
  - **Warn**: Bulk-data elements are the dominant contributor (for example,
    over half of converted content) to a page that lands in the size
    checks' warn band.
  - **Fail**: Bulk-data elements are the dominant contributor to a page
    that exceeds the size checks' fail threshold. Content after the bulk
    element is beyond the truncation point for most platforms.
- **Recommended action**: Bulk data is often legitimate content (a support
  matrix is the point of a support-matrix page), so the goal is structure,
  not removal. Split large generated tables across per-section pages (as
  self-contained units reached from an index, each complete for its scope;
  paginating one table into windows trades this problem for the one
  `single-fetch-completeness` describes),
  provide filtered or queryable views, load embedded data blobs on demand,
  and place prose before bulk elements so truncation removes data rows
  rather than explanation. Report the attribution to content authors:
  a page that an author experiences as two paragraphs should not ship as a
  hundred kilobytes without the author knowing.
- **Automation**: Heuristic. After HTML-to-markdown conversion (same
  pipeline as `page-size-html`), detect bulk elements: tables above a row
  threshold with uniform row structure, fenced or inline JSON blobs above a
  size threshold, and base64 runs. Report each element's share of the
  converted content. Thresholds for "bulk" need calibration against real
  pages and should be configurable. This check sees only bulk that
  survives conversion into content (tables, code fences, prose-embedded
  data); serialized payloads inside `<script>` tags are stripped by
  conversion and are `page-size-transfer`'s domain. The two checks divide
  the data-dump problem by pipeline: script payloads burden every fetch,
  content-embedded bulk burdens what the model reads.
- **Notes**: This is the data-widget sibling of
  `tabbed-content-serialization`, which covers the same flattening failure
  for tab and accordion UI. Together with `content-start-position` (where
  content sits relative to boilerplate), these checks explain *why* a page
  fails the Category 3 size checks, not just that it does.