I saw Bun mention that their docs now return Markdown instead of HTML when an AI like Claude Code fetches them, which cuts token usage by around 10x and even reduces load times.
I thought that was a clever idea, so I made a lightweight version using a Cloudflare Worker. It just checks the request headers for AI user agents and serves Markdown if it finds one. Ended up being only a few lines of code.
Really interesting as we look to the "new internet" and how AI indexes/recommends things vs "the old way" of SEO/Google search. I'd be curious if implementing something like this provides a sense of "priority" for what information a model's crawler would serve, will check your workflow out!
I saw Bun mention that their docs now return Markdown instead of HTML when an AI like Claude Code fetches them, which cuts token usage by around 10x and even reduces load times.
I thought that was a clever idea, so I made a lightweight version using a Cloudflare Worker. It just checks the request headers for AI user agents and serves Markdown if it finds one. Ended up being only a few lines of code.
The repo is here if you want to try it or tweak it: https://github.com/thekevinm/HTML-to-MD-AI
I’m curious if anyone else has played with similar ideas to make docs faster or more efficient for AI crawlers or API-style readers.
Really interesting as we look to the "new internet" and how AI indexes/recommends things vs "the old way" of SEO/Google search. I'd be curious if implementing something like this provides a sense of "priority" for what information a model's crawler would serve, will check your workflow out!
less html parsing == less compute == less spend