Serving Markdown to Agents: My no-store Response Evicted the HTML Cache

Hot Deal Sheet is a list of Korean deal posts rendered by a Cloudflare Worker. In September 2026 I taught two of its routes to answer Accept: text/markdown, so an agent reading the home page would get a few lines of text instead of a 160 KB HTML table. The markdown was the cheap part. The expensive part was one cache header I picked to be safe, which turned every agent visit into a fresh render of the heaviest page on the site.

The site was on the Workers free plan at the time, which allows 10 ms of CPU per request. The home page HTML costs several times that. It stays alive only because the edge cache answers almost every request without running the Worker. I wrote about that arrangement in an earlier post about the cron I broke. The short version: on this site, how often the Worker runs matters more than anything the Worker does.

Two representations, one URL

The negotiation itself is small. The home page / and the deal pages /deal/* check the Accept header. If the request prefers text/markdown strictly over text/html, the Worker answers with markdown. A tie goes to HTML.

The strictness matters because of what browsers send. A typical browser Accept ends in */*;q=0.8, and */* matches text/markdown too. If the rule were “does this client accept markdown”, every browser would say yes. RFC 9110 also says the most specific range wins, so in text/html, */*;q=0.9 the quality for markdown is 0.9 and not 1. Get either of those wrong and people start receiving markdown in their browser tab.

When it worked, the numbers were as good as I hoped. I measured CPU from wrangler tail in production, with a query string added to bust the cache so the Worker actually ran:

Same URL, different representation	CPU ms (min / median)	Body
/ as HTML, 100 rows	29 / 62	160 KB
/ as markdown, 30 rows	1 / 3	10 KB
/deal/<slug> as HTML	6 / 10
/deal/<slug> as markdown	2 / 3

The rows are not equal. The markdown list stops at 30, the same page size as the JSON API, and an agent that wants more follows a cursor. Even allowing for that, the gap is the same story the profiler told me in August. On this site CPU tracks output bytes, and a markdown line is far smaller than a table row.

The safe choice

Every route on the site sets its own Cache-Control. The home page gets public, max-age=0, s-maxage=300, so the edge holds it for five minutes and browsers ask again every time. For the markdown response I did not reuse that. I sent Cache-Control: no-store.

My reasoning went like this. If the edge cache ignored Vary: Accept, a cached markdown body could be served to the next visitor at /, and a human would see raw text instead of the page. Not storing the markdown at all seemed like the failure direction I could live with. An agent would cost one Worker run and one database query, and that was cheap.

Both halves of that reasoning were wrong, and I only found out because I measured after deploying.

What the cache actually did

First, Vary is honored. I warmed / with a Chrome Accept string and then requested it with Safari’s. The second request was a MISS. The raw Accept string goes into the cache key. Cloudflare’s Workers Caching docs say the same thing: it stores a separate variant for each distinct combination of the headers listed in Vary. My fear of markdown leaking to humans had no basis.

Second, the no-store markdown response did something I did not expect. I compared HTML requests to / with and without a markdown request in between, reading cf-cache-status each time. In the control runs, with nothing in between, the cached HTML never went missing. With a markdown request in between, the HTML came back as a MISS in three rounds out of three.

So each agent read removed the cached HTML at that URL. The next human visitor paid for a 62 ms render, on a plan that allows 10. The most expensive page on the site was now being rendered in proportion to agent traffic, which was the exact outcome the cache policy existed to prevent.

I have not found this behavior described anywhere. The debugging page says a no-store response is not stored and shows as BYPASS, which matches what I saw for the markdown itself. It says nothing about the other variants at that URL. What I have is the measurement. If you depend on it either way, measure it on your own zone. Docs above were read on 14 October 2026.

The fix was to stop being careful

The markdown response now uses the same policy as the HTML for each route. / caches for five minutes and a deal page for ten, in both representations. Two exceptions stayed short: a markdown 404 or 406 is held for 60 seconds, and a 503 is never stored, because caching an outage freezes it for the length of the TTL.

That change had a second benefit I had not planned. With markdown cached at the edge, a crawler that reads every deal page in markdown no longer turns into one database query per request. The free D1 plan counts rows read across the whole account, and this account runs more than one service, so that budget is shared.

Vary is not free either

The Chrome and Safari result cuts both ways. If the raw Accept string is part of the key, every browser family gets its own copy of the page, and each copy has to be rendered once per point of presence before it is cached. On a URL with Vary: Accept, the number of cache fills goes up by roughly the number of distinct Accept strings your visitors send.

That is why only two routes negotiate. The home page and the deal pages are what an agent actually reads. Everything else, the date archives, category pages, and legal pages, stays HTML only with no Vary, and an llms.txt file tells agents where the markdown lives. A request for markdown at a URL that does not exist still gets a markdown 404, because that response is rare and does not add variants to any cached page.

If I were adding markdown for agents to another cached site, I would start from the cache headers rather than the markdown. Pick the routes where Vary is worth the extra fills, give the markdown the same TTL as the HTML, and before trusting any header that is supposed to be safe, warm the cache, send the other variant, request the first one again, and read cf-cache-status.