# Category 7: Authentication and Access

> For AI agents: the complete documentation index is at [https://agentdocsspec.com/llms.txt](https://agentdocsspec.com/llms.txt). Markdown versions of any page are available by appending `index.md` to the URL path.

Part of the [Web Documentation Delivery Spec](https://agentdocsspec.com/spec/web/). The [Checks Summary](https://agentdocsspec.com/spec/web/#checks-summary) lists all checks with links to their definitions.

These checks evaluate whether documentation is accessible to agents at all:
without requiring interactive authentication, and without infrastructure-level
barriers aimed at automated clients. Docs behind login walls are effectively
invisible to coding agents, which has significant implications as agent-assisted
development becomes a standard workflow. Bot-protection systems can produce the
same invisibility through a different mechanism, and often without the site
owner realizing documentation is affected.

### Why This Matters

Enterprises often gate documentation behind authentication to protect
intellectual property, enforce licensing terms, or comply with access control
policies. These are legitimate business reasons. However, the tradeoff is
sharper than most organizations realize: authenticated docs are not just
inconvenient for agents, they are completely inaccessible.

When an agent encounters an auth-gated page, it sees one of these:

- A **401 or 403 response**, which tells it nothing useful.
- A **login page returned as 200**, which is a soft 404 from the agent's
  perspective. The agent tries to extract documentation from the login form
  HTML and produces nonsensical results.
- A **redirect to an SSO provider**, which is a cross-host redirect the agent
  cannot follow, even if it wanted to.

In all three cases, the agent may take one of two actions:

- Fall back on whatever it absorbed during training, which may be outdated,
  incomplete, or wrong.
- Leave your official product website and look for secondary sources to learn
  about your product, including blogs or articles which may inaccurate, outdated,
  and not reflect your official best practices.

In these scenarios, the developer either gets bad guidance, or has to manually
copy-paste docs into the conversation, losing the workflow benefits that agents
provide. This may also be completely invisible to the developer, as an agent
may "helpfully" turn to blog posts or secondary references without disclosing
to the human user that it used secondary sources which should be verified.

The competitive dimension is real. If your product's documentation requires a
login and your competitor's doesn't, developers using agents will have a
dramatically better experience with the competitor's product. The agent can
read the competitor's API reference, find code examples, and verify patterns
in real time. For your product, the agent is guessing.

Bot protection produces the same invisibility through a different mechanism,
and usually without a deliberate decision. Auth gating is at least a policy
choice about who may read the docs; bot enforcement is typically configured
site-wide for security reasons, with documentation caught as collateral. It
is also harder for the site owner to see. A login wall fails every request
the same way, but behavioral enforcement can pass a casual spot check and
then throttle, challenge, or stall the sustained multi-page sessions that
real agent work produces. A site owner who verifies their docs by loading a
page in a browser, or fetching a single URL with curl, will conclude
everything works.

### `auth-gate-detection`

- **What it checks**: Whether documentation pages require authentication to
  access content.
- **Why it matters**: A documentation site that returns login pages, 401/403
  responses, or SSO redirects for its content pages is completely opaque to
  agents. This check identifies the problem so site owners can make an
  informed decision about the tradeoff.
- **Result levels**:
  - **Pass**: Documentation pages return content (200 with substantive body)
    without requiring authentication.
  - **Warn**: Some pages are accessible but others require authentication
    (partial gating). This is common for sites that gate advanced content or
    API references while keeping tutorials public.
  - **Fail**: All or most documentation pages require authentication.
- **Recommended action**:
  - **Warn**: Consider ungating reference documentation and API guides.
    Agents can access public pages but will fall back on training data for
    gated content.
  - **Fail**: Agents cannot access your documentation and will rely on
    potentially outdated training data or secondary sources. Consider
    providing alternative access paths (see `auth-alternative-access`).
- **Automation**: Full. Fetch a sample of documentation URLs and classify
  responses: 200 with content (accessible), 401/403 (auth required), 200
  with login form heuristics (soft auth gate), or redirect to known SSO
  providers (auth redirect). Login form detection uses heuristics: look for
  `<input type="password">`, common SSO redirect domains (okta.com,
  auth0.com, login.microsoftonline.com), or page titles containing "sign in"
  or "log in".
- **Notes**: This check is informational for sites that intentionally gate
  content. It doesn't prescribe that all docs must be public. It ensures the
  site owner is aware of the agent accessibility impact and can evaluate
  whether alternative access paths (see below) are warranted.

### `auth-alternative-access`

- **What it checks**: Whether an auth-gated documentation site provides
  alternative access paths that agents can use.
- **Why it matters**: Sites that must gate their primary docs can still serve
  agents through secondary channels. This check looks for evidence that such
  channels exist, giving the site credit for providing agent access even when
  the main docs require a login.
- **Result levels**:
  - **Pass**: At least one alternative access path is detected (see list
    below).
  - **Warn**: The site provides partial alternative access (e.g., an
    `llms.txt` exists but only covers a subset of the gated content).
  - **Fail**: No alternative access paths detected for auth-gated content.
- **Recommended action**:
  - **Warn**: Expand alternative access to cover more of the gated
    documentation.
  - **Fail**: Consider providing a public `llms.txt`, ungating reference
    docs, shipping docs with your SDK, or providing an MCP server for
    authenticated access. See [Making Private Docs Agent-Accessible](#making-private-docs-agent-accessible)
    for options ordered by implementation effort.
- **Automation**: Partial. Some access paths can be detected automatically;
  others require manual verification.
- **Detectable access paths**:
  - **Public `llms.txt`**: The site serves an `llms.txt` file that doesn't
    require authentication, even if the underlying docs pages do. This gives
    agents at least a navigational index.
  - **Public markdown or API endpoint**: Some pages or a content API respond
    to unauthenticated requests even when the main docs UI requires login.
  - **Bundled documentation**: The product ships docs as part of its package
    or SDK (e.g., a `docs/` directory, man pages, or built-in `help`
    subcommands). Agents can read local files without authentication.
  - **CLI-based doc access**: The product provides a CLI command (e.g.,
    `yourproduct docs search "topic"`) that the developer has already
    authenticated, making content available to agents through tool use.
  - **MCP server**: The organization provides an MCP server that exposes
    documentation through tool calls, with authentication handled
    server-side. This is the most capable option for private docs because it
    preserves full content access while keeping credentials out of the agent
    context. (Detection is manual; there's no standard way to discover
    whether a company offers an MCP server.)
- **Notes**: Only applies when `auth-gate-detection` returns warn or fail.
  If docs are publicly accessible, this check is skipped.

### `bot-protection-interference`

- **What it checks**: Whether bot-protection systems (CDN bot management, WAF
  rules, behavioral rate enforcement) interfere with automated fetching of
  documentation content.
- **Why it matters**: Coding agents are automated clients. Bot management tuned
  for scraper and attack traffic frequently cannot distinguish an agent
  fetching docs on a developer's behalf from abuse, and its enforcement modes
  are worse for agents than a clean block because the failures are invisible:
  - **Challenge interstitials served as 200.** A "verifying your browser" page
    returned with a success status is a soft 404 from the agent's perspective;
    the agent extracts challenge boilerplate instead of documentation and may
    present it as an answer.
  - **Tarpits.** The server accepts the connection and returns headers, then
    holds the response body open indefinitely. The agent's fetch stalls until
    its own timeout with no error to reason about, and a multi-page reading
    session dies partway through.
  - **Volume-triggered throttling or blocking.** Enforcement engages only
    after several requests, so the first pages of a session succeed and later
    ones fail. Because enforcement is typically stateful (keyed to IP or
    client fingerprint) and decays over time, single-page spot checks look
    healthy while sustained agent sessions fail.

  These modes are not hypothetical. In one observed production case, a CDN's
  bot management responded to a sustained documentation scan by holding
  response bodies open indefinitely. Single-request probes of the same pages
  looked healthy throughout, and enforcement decayed after a cooldown period.
- **Result levels**:
  - **Pass**: A sustained multi-page scan completes with no evidence of
    interference: no challenge pages, no stalled response bodies, no
    volume-correlated failures. Because detection is heuristic and
    enforcement is stateful, pass means no interference was observed during
    this run, not a guarantee that bot protection will never engage.
  - **Warn**: Intermittent interference. Some requests during a sustained scan
    are challenged, stalled, or blocked while others succeed.
  - **Fail**: Sustained fetching is effectively blocked. Once enforcement
    triggers, most requests are challenged, stalled, or denied.
- **Recommended action**:
  - **Warn**: Identify which bot-management layer is challenging or stalling
    some requests and exempt public documentation routes from behavioral
    enforcement. Intermittent interference means enforcement thresholds sit
    close to normal agent reading cadence, so small configuration changes
    (or ordinary traffic growth) can tip it into sustained blocking.
  - **Fail**: Treat public documentation paths as automation-friendly in
    bot-management configuration. Exempt docs routes from behavioral
    enforcement, or scope enforcement to interactive product surfaces. Where
    limits are genuinely needed, prefer an explicit `429` with `Retry-After`
    over tarpits or silent blocks: a `429` is an error the agent can see,
    report, and react to, while a tarpit or challenge page fails invisibly.
    Never serve challenge interstitials with a 200 status.
- **Automation**: Heuristic. Interference generally cannot be probed directly
  without generating the sustained traffic that triggers it, so implementations
  should detect it as a byproduct of a normal scan: response bodies that stall
  past the request timeout, challenge-page heuristics in fetched content, or
  failure rates that climb as the scan progresses. Unlike other checks, this
  one has no fetch phase of its own; it is evaluated from evidence accumulated
  across the entire run rather than probed as a discrete step.

  A scan that follows this spec's guidance in
  [A Note on Responsible Use](https://agentdocsspec.com/spec/web/#a-note-on-responsible-use) already resembles a
  realistic multi-page agent reading session, and that is the correct
  calibration, because that workload is exactly what this check predicts. If a
  respectful scan at agent-like cadence triggers enforcement, that is the
  finding, not a scan artifact. Implementations should not escalate traffic to
  deliberately provoke enforcement. Because enforcement is stateful and decays,
  results vary across runs; a clean run does not prove absence.
- **Notes**: When interference is detected mid-scan, results from other
  multi-page checks are computed on whatever sample survived. Implementations
  should surface a run-level warning that scores may not reflect the full site
  (see the [Bot Protection Degrading Scan Reliability](https://agentdocsspec.com/spec/web/interaction-effects/#bot-protection-degrading-scan-reliability)
  interaction effect). This check identifies the condition; it does not
  prescribe that sites disable bot protection. Like auth gating, this is a
  tradeoff the site owner should make deliberately, with awareness that coding
  agents are among the clients being blocked.

  Results are also vantage-point dependent. Enforcement is commonly keyed to
  client reputation (IP range, ASN, TLS fingerprint), so a scan run from
  datacenter infrastructure may trigger enforcement that residential traffic
  would not. This mirrors real agent traffic, which originates from the same
  mix of vantage points: some harnesses fetch from the developer's own
  connection, while others route web fetches through vendor server
  infrastructure or run in cloud-hosted sessions, both of which present
  datacenter IPs to the site. A datacenter-origin scan is representative of
  that second class of agent traffic, not a false positive. Implementations
  should note the scan's network context alongside results; even a coarse
  classification (a developer machine versus CI or cloud infrastructure)
  lets a reader interpret enforcement findings correctly. Reports should
  carry the classification rather than the scanner's raw IP address, since
  reports are often shared.

  This check is distinct from robots.txt and AI user-agent blocking, which
  this spec intentionally excludes (see
  [Appendix B](https://agentdocsspec.com/spec/web/appendices/#robotstxt-and-ai-user-agent-blocking)). Declared crawling
  policy is invisible to most coding agents because they don't identify
  themselves; behavioral enforcement affects them precisely because their
  traffic is indistinguishable from the automated traffic it targets.

### Making Private Docs Agent-Accessible

This section offers non-normative guidance for organizations that gate their
documentation. The options below are ordered roughly by implementation effort,
from lowest to highest.

**1. Ungating reference documentation.** The simplest option: make API
references, SDK docs, and integration guides public while keeping truly
sensitive content (internal architecture, security configurations, pricing
tiers) behind auth. Many enterprises already do this for developer
experience reasons. Agents benefit from the same split.

**2. Shipping docs with the product.** Include documentation as local files
in your SDK, package, or CLI tool. A `docs/` directory with markdown files,
comprehensive README content, or built-in help text is always available to
agents reading the local filesystem. This is particularly effective for
API clients and libraries where the docs are version-specific anyway.

**3. Providing a public `llms.txt`.** Even if page content is gated, a
public `llms.txt` that describes what documentation exists and how it's
organized gives agents a map. They can tell the developer "the rate limiting
docs are at /docs/api/rate-limits, but I can't access them; could you paste
the relevant section?" This is better than the agent having no idea what
docs exist at all.

**4. Supporting token-based access for agent-facing endpoints.** Serve
`llms.txt` and markdown content behind API key or bearer token
authentication rather than browser-based SSO. Agents and their tooling can
be configured to pass static credentials, similar to how `npm` or `pip`
authenticate with private registries. This preserves access control while
enabling programmatic access.

**5. Building an MCP server.** An MCP server gives agents structured,
authenticated access to documentation through tool calls like
`search_docs("rate limiting")` or `get_doc("api/authentication")`. Auth
credentials are configured on the server; the agent never sees them. This
is the richest option because the MCP server can provide search, filtering,
and context-aware responses rather than just serving raw files. It also
allows fine-grained access control (different API keys could see different
content tiers).

**6. Providing a CLI with doc access.** If your product already has a CLI
that developers authenticate with, adding a `docs` subcommand gives agents
access through a channel the developer has already authorized. The agent
calls the CLI tool; the CLI handles authentication using the developer's
existing credentials.

Organizations don't need to implement all of these. A public `llms.txt`
combined with ungated reference docs covers the most common agent use cases
with minimal effort. MCP servers are for organizations that want to provide
a first-class agent experience with their private documentation.