Category 7: Authentication and Access

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether documentation is accessible to agents at all: without requiring interactive authentication, and without infrastructure-level barriers aimed at automated clients. Docs behind login walls are effectively invisible to coding agents, which has significant implications as agent-assisted development becomes a standard workflow. Bot-protection systems can produce the same invisibility through a different mechanism, and often without the site owner realizing documentation is affected.

Why This Matters #

Enterprises often gate documentation behind authentication to protect intellectual property, enforce licensing terms, or comply with access control policies. These are legitimate business reasons. However, the tradeoff is sharper than most organizations realize: authenticated docs are not just inconvenient for agents, they are completely inaccessible.

When an agent encounters an auth-gated page, it sees one of these:

• A 401 or 403 response, which tells it nothing useful.
• A login page returned as 200, which is a soft 404 from the agent’s perspective. The agent tries to extract documentation from the login form HTML and produces nonsensical results.
• A redirect to an SSO provider, which is a cross-host redirect the agent cannot follow, even if it wanted to.

In all three cases, the agent may take one of two actions:

• Fall back on whatever it absorbed during training, which may be outdated, incomplete, or wrong.
• Leave your official product website and look for secondary sources to learn about your product, including blogs or articles which may inaccurate, outdated, and not reflect your official best practices.

In these scenarios, the developer either gets bad guidance, or has to manually copy-paste docs into the conversation, losing the workflow benefits that agents provide. This may also be completely invisible to the developer, as an agent may “helpfully” turn to blog posts or secondary references without disclosing to the human user that it used secondary sources which should be verified.

The competitive dimension is real. If your product’s documentation requires a login and your competitor’s doesn’t, developers using agents will have a dramatically better experience with the competitor’s product. The agent can read the competitor’s API reference, find code examples, and verify patterns in real time. For your product, the agent is guessing.

Bot protection produces the same invisibility through a different mechanism, and usually without a deliberate decision. Auth gating is at least a policy choice about who may read the docs; bot enforcement is typically configured site-wide for security reasons, with documentation caught as collateral. It is also harder for the site owner to see. A login wall fails every request the same way, but behavioral enforcement can pass a casual spot check and then throttle, challenge, or stall the sustained multi-page sessions that real agent work produces. A site owner who verifies their docs by loading a page in a browser, or fetching a single URL with curl, will conclude everything works.

auth-gate-detection #

• What it checks: Whether documentation pages require authentication to access content.
• Why it matters: A documentation site that returns login pages, 401/403 responses, or SSO redirects for its content pages is completely opaque to agents. This check identifies the problem so site owners can make an informed decision about the tradeoff.
• Result levels:
  • Pass: Documentation pages return content (200 with substantive body) without requiring authentication.
  • Warn: Some pages are accessible but others require authentication (partial gating). This is common for sites that gate advanced content or API references while keeping tutorials public.
  • Fail: All or most documentation pages require authentication.
• Recommended action:
  • Warn: Consider ungating reference documentation and API guides. Agents can access public pages but will fall back on training data for gated content.
  • Fail: Agents cannot access your documentation and will rely on potentially outdated training data or secondary sources. Consider providing alternative access paths (see auth-alternative-access).
• Automation: Full. Fetch a sample of documentation URLs and classify responses: 200 with content (accessible), 401/403 (auth required), 200 with login form heuristics (soft auth gate), or redirect to known SSO providers (auth redirect). Login form detection uses heuristics: look for <input type="password">, common SSO redirect domains (okta.com, auth0.com, login.microsoftonline.com), or page titles containing “sign in” or “log in”.
• Notes: This check is informational for sites that intentionally gate content. It doesn’t prescribe that all docs must be public. It ensures the site owner is aware of the agent accessibility impact and can evaluate whether alternative access paths (see below) are warranted.

auth-alternative-access #

• What it checks: Whether an auth-gated documentation site provides alternative access paths that agents can use.
• Why it matters: Sites that must gate their primary docs can still serve agents through secondary channels. This check looks for evidence that such channels exist, giving the site credit for providing agent access even when the main docs require a login.
• Result levels:
  • Pass: At least one alternative access path is detected (see list below).
  • Warn: The site provides partial alternative access (e.g., an llms.txt exists but only covers a subset of the gated content).
  • Fail: No alternative access paths detected for auth-gated content.
• Recommended action:
  • Warn: Expand alternative access to cover more of the gated documentation.
  • Fail: Consider providing a public llms.txt, ungating reference docs, shipping docs with your SDK, or providing an MCP server for authenticated access. See Making Private Docs Agent-Accessible for options ordered by implementation effort.
• Automation: Partial. Some access paths can be detected automatically; others require manual verification.
• Detectable access paths:
  • Public llms.txt: The site serves an llms.txt file that doesn’t require authentication, even if the underlying docs pages do. This gives agents at least a navigational index.
  • Public markdown or API endpoint: Some pages or a content API respond to unauthenticated requests even when the main docs UI requires login.
  • Bundled documentation: The product ships docs as part of its package or SDK (e.g., a docs/ directory, man pages, or built-in help subcommands). Agents can read local files without authentication.
  • CLI-based doc access: The product provides a CLI command (e.g., yourproduct docs search "topic") that the developer has already authenticated, making content available to agents through tool use.
  • MCP server: The organization provides an MCP server that exposes documentation through tool calls, with authentication handled server-side. This is the most capable option for private docs because it preserves full content access while keeping credentials out of the agent context. (Detection is manual; there’s no standard way to discover whether a company offers an MCP server.)
• Notes: Only applies when auth-gate-detection returns warn or fail. If docs are publicly accessible, this check is skipped.

bot-protection-interference #

• What it checks: Whether bot-protection systems (CDN bot management, WAF rules, behavioral rate enforcement) interfere with automated fetching of documentation content.

• Why it matters: Coding agents are automated clients. Bot management tuned for scraper and attack traffic frequently cannot distinguish an agent fetching docs on a developer’s behalf from abuse, and its enforcement modes are worse for agents than a clean block because the failures are invisible:

  • Challenge interstitials served as 200. A “verifying your browser” page returned with a success status is a soft 404 from the agent’s perspective; the agent extracts challenge boilerplate instead of documentation and may present it as an answer.
  • Tarpits. The server accepts the connection and returns headers, then holds the response body open indefinitely. The agent’s fetch stalls until its own timeout with no error to reason about, and a multi-page reading session dies partway through.
  • Volume-triggered throttling or blocking. Enforcement engages only after several requests, so the first pages of a session succeed and later ones fail. Because enforcement is typically stateful (keyed to IP or client fingerprint) and decays over time, single-page spot checks look healthy while sustained agent sessions fail.

  These modes are not hypothetical. In one observed production case, a CDN’s bot management responded to a sustained documentation scan by holding response bodies open indefinitely. Single-request probes of the same pages looked healthy throughout, and enforcement decayed after a cooldown period.

• Result levels:

  • Pass: A sustained multi-page scan completes with no evidence of interference: no challenge pages, no stalled response bodies, no volume-correlated failures. Because detection is heuristic and enforcement is stateful, pass means no interference was observed during this run, not a guarantee that bot protection will never engage.
  • Warn: Intermittent interference. Some requests during a sustained scan are challenged, stalled, or blocked while others succeed.
  • Fail: Sustained fetching is effectively blocked. Once enforcement triggers, most requests are challenged, stalled, or denied.

• Recommended action:

  • Warn: Identify which bot-management layer is challenging or stalling some requests and exempt public documentation routes from behavioral enforcement. Intermittent interference means enforcement thresholds sit close to normal agent reading cadence, so small configuration changes (or ordinary traffic growth) can tip it into sustained blocking.
  • Fail: Treat public documentation paths as automation-friendly in bot-management configuration. Exempt docs routes from behavioral enforcement, or scope enforcement to interactive product surfaces. Where limits are genuinely needed, prefer an explicit 429 with Retry-After over tarpits or silent blocks: a 429 is an error the agent can see, report, and react to, while a tarpit or challenge page fails invisibly. Never serve challenge interstitials with a 200 status.

• Automation: Heuristic. Interference generally cannot be probed directly without generating the sustained traffic that triggers it, so implementations should detect it as a byproduct of a normal scan: response bodies that stall past the request timeout, challenge-page heuristics in fetched content, or failure rates that climb as the scan progresses. Unlike other checks, this one has no fetch phase of its own; it is evaluated from evidence accumulated across the entire run rather than probed as a discrete step.

  A scan that follows this spec’s guidance in A Note on Responsible Use already resembles a realistic multi-page agent reading session, and that is the correct calibration, because that workload is exactly what this check predicts. If a respectful scan at agent-like cadence triggers enforcement, that is the finding, not a scan artifact. Implementations should not escalate traffic to deliberately provoke enforcement. Because enforcement is stateful and decays, results vary across runs; a clean run does not prove absence.

• Notes: When interference is detected mid-scan, results from other multi-page checks are computed on whatever sample survived. Implementations should surface a run-level warning that scores may not reflect the full site (see the Bot Protection Degrading Scan Reliability interaction effect). This check identifies the condition; it does not prescribe that sites disable bot protection. Like auth gating, this is a tradeoff the site owner should make deliberately, with awareness that coding agents are among the clients being blocked.

  Results are also vantage-point dependent. Enforcement is commonly keyed to client reputation (IP range, ASN, TLS fingerprint), so a scan run from datacenter infrastructure may trigger enforcement that residential traffic would not. This mirrors real agent traffic, which originates from the same mix of vantage points: some harnesses fetch from the developer’s own connection, while others route web fetches through vendor server infrastructure or run in cloud-hosted sessions, both of which present datacenter IPs to the site. A datacenter-origin scan is representative of that second class of agent traffic, not a false positive. Implementations should note the scan’s network context alongside results; even a coarse classification (a developer machine versus CI or cloud infrastructure) lets a reader interpret enforcement findings correctly. Reports should carry the classification rather than the scanner’s raw IP address, since reports are often shared.

  This check is distinct from robots.txt and AI user-agent blocking, which this spec intentionally excludes (see Appendix B). Declared crawling policy is invisible to most coding agents because they don’t identify themselves; behavioral enforcement affects them precisely because their traffic is indistinguishable from the automated traffic it targets.

Making Private Docs Agent-Accessible #

This section offers non-normative guidance for organizations that gate their documentation. The options below are ordered roughly by implementation effort, from lowest to highest.

1. Ungating reference documentation. The simplest option: make API references, SDK docs, and integration guides public while keeping truly sensitive content (internal architecture, security configurations, pricing tiers) behind auth. Many enterprises already do this for developer experience reasons. Agents benefit from the same split.

2. Shipping docs with the product. Include documentation as local files in your SDK, package, or CLI tool. A docs/ directory with markdown files, comprehensive README content, or built-in help text is always available to agents reading the local filesystem. This is particularly effective for API clients and libraries where the docs are version-specific anyway.

3. Providing a public llms.txt. Even if page content is gated, a public llms.txt that describes what documentation exists and how it’s organized gives agents a map. They can tell the developer “the rate limiting docs are at /docs/api/rate-limits, but I can’t access them; could you paste the relevant section?” This is better than the agent having no idea what docs exist at all.

4. Supporting token-based access for agent-facing endpoints. Serve llms.txt and markdown content behind API key or bearer token authentication rather than browser-based SSO. Agents and their tooling can be configured to pass static credentials, similar to how npm or pip authenticate with private registries. This preserves access control while enabling programmatic access.

5. Building an MCP server. An MCP server gives agents structured, authenticated access to documentation through tool calls like search_docs("rate limiting") or get_doc("api/authentication"). Auth credentials are configured on the server; the agent never sees them. This is the richest option because the MCP server can provide search, filtering, and context-aware responses rather than just serving raw files. It also allows fine-grained access control (different API keys could see different content tiers).

6. Providing a CLI with doc access. If your product already has a CLI that developers authenticate with, adding a docs subcommand gives agents access through a channel the developer has already authorized. The agent calls the CLI tool; the CLI handles authentication using the developer’s existing credentials.

Organizations don’t need to implement all of these. A public llms.txt combined with ungated reference docs covers the most common agent use cases with minimal effort. MCP servers are for organizations that want to provide a first-class agent experience with their private documentation.