↓Skip to main content

Category 5: URL Stability and Redirects

2 mins
Table of Contents
For AI agents: a documentation index is available at /llms.txt — markdown versions of all pages are available by appending index.md to any URL path.

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether documentation URLs behave in ways that agents can handle, given that agents retrieve URLs from training data and have limited ability to discover moved content.

http-status-codes #

  • What it checks: Whether pages return correct HTTP status codes. In particular, whether “not found” pages return 404 (not 200 with a friendly error page).
  • Why it matters: Soft 404s (200 status with “page not found” content) are worse than real 404s for agents. The agent sees a 200 and tries to extract information from the error page content rather than recognizing the page doesn’t exist. A clean 404 tells the agent to try a different approach.
  • Result levels:
    • Pass: Error pages return appropriate 4xx status codes.
    • Fail: Error pages return 200 (soft 404).
  • Recommended action: Configure your server to return 404 status codes for pages that don’t exist. Agents try to extract information from soft 404 page content instead of recognizing the page is missing.
  • Automation: Full. Test known-bad URLs (e.g., append random strings to real page paths) and check status codes.

redirect-behavior #

  • What it checks: Whether redirects are same-host (transparent to agents) or cross-host (a friction point), and whether redirects use proper HTTP status codes (301/302) vs. JavaScript-based redirects.
  • Why it matters: Same-host redirects work transparently because the HTTP client follows them automatically. Cross-host redirects are a known failure point; Claude Code, for example, doesn’t automatically follow cross-host redirects (security measure against open-redirect attacks). JavaScript redirects don’t work at all because agents don’t execute JavaScript.
  • Result levels:
    • Pass: All redirects are same-host HTTP redirects (301/302).
    • Warn: Cross-host HTTP redirects are present (agents may or may not follow them depending on the platform).
    • Fail: JavaScript-based redirects are detected.
  • Recommended action:
    • Warn: Where possible, use same-host redirects or update URLs to point directly to the final destination.
    • Fail: Replace JavaScript-based redirects with HTTP 301/302 redirects. Agents don’t execute JavaScript and will not follow these redirects.
  • Automation: Partial. HTTP redirects are detectable. JavaScript redirects require fetching the page and scanning for window.location, meta refresh, or similar patterns.