Docs URL Benchmark: Markdown & llms.txt > HTML

Browserbase's CEO reported that Claude Code kept hitting 404s on Mintlify-hosted documentation. When we dug in, the cause was subtle: stripping pages down to markdown for agent consumption also stripped out the navigation. Left without a map, the agents were guessing at URLs that didn't exist.

Adding a single link to llms.txt fixed it. But we wanted to be sure that was actually the optimal move, so we ran a benchmark: 2,400 runs across multiple models. The short version is that a single link to llms.txt does almost all of the work.

The problem

Documentation is built for human readers. When those same pages get converted to stripped markdown for an agent, the navigation disappears entirely, and the agent is left inventing paths, most of which 404.

The fix is to hand the agent a map. Mintlify already publishes llms.txt files: an agent-friendly table of contents listing every page with its title and description. Linking to that file at the bottom of each markdown page made the 404s go away.

How we ran it

We tested four documentation formats, using a local proxy to log every request an agent made: raw HTML, plain markdown, markdown with an llms.txt link, and markdown with the llms.txt content inlined.

The setup: 20 Mintlify docs sites, 5 questions per site, each run 3 times per format, with Claude Code and Codex running Sonnet 5 and GPT-5.5. That is 2,400 total runs.

What we found

Including the llms.txt link produced a big efficiency gain. On raw HTML, agents kept requesting file variants that didn't exist, piling up 404s. Adding the link cut 404 errors by roughly 90%, which meant fewer wasted requests and lower token usage.

Accuracy stayed high, in the mid-to-high 90s, across every format. So the map didn't make the agents more correct. It made them faster and cheaper at getting to the right answer.

It's the map that matters

The result that surprised us: markdown itself wasn't the fix. The map was.

Plain markdown with no guidance actually did worse than HTML, because agents made up their own paths and guessed wrong more often. Handing them the llms.txt map eliminated that.

And linking beat inlining. Inlining the whole file solved the same 404s, but it burned more tokens on every request. The link was the more efficient option.

The pattern held when we added more models across the same 20 sites. 404s per task collapsed from around 2 down to nearly zero for every model we tested.

What this means for your docs

If your docs are being read by agents, give them a navigation map. One llms.txt link buys most of the benefit, across every model we tested. As agents become the primary readers of documentation, structured navigation stops being a nice-to-have.