hraness

accept: text/markdown in production

content negotiation for agents without cloaking

Drafted by an AI agent at Ben Guo's direct request from the Hraness source repositories, and checked against those sources before publication.

An agent reading a public page wants the Markdown, not the HTML. The markup, the layout, and the chrome are all noise to a parser that wants the words. The naive way to give it that is a separate URL (page.md) and the slightly-less-naive way is Accept: text/markdown content negotiation. The reason neither is obvious is that both can look like cloaking: serving different content to different clients is exactly what a spam site does. The difference is that honest negotiation serves the same underlying content in a different representation, says so in the response headers, and does it for any client that asks, not only the ones with a crawler user-agent.

On hraness.com every public HTML page has both: a sibling .md URL that is the same representation, and an Accept negotiation that prefers Markdown when the client says so. The page you are reading has a /reference/web-delivery/content-negotiation-markdown.md sibling that contains the same words, and the middleware would have served that same Markdown if this request had sent Accept: text/markdown instead of the browser’s usual header. Both representations are generated from the same registries, so neither can drift from the other.

the produced set is two

The whole negotiation is bounded by a two-element list in accept.ts:

export const HTML_MEDIA_TYPE = "text/html";
export const MARKDOWN_MEDIA_TYPE = "text/markdown";
export const PRODUCED_MEDIA_TYPES = [
  HTML_MEDIA_TYPE,
  MARKDOWN_MEDIA_TYPE,
] as const;

The server produces exactly two representations of a public page. It does not produce JSON, does not produce text/plain, and does not produce a “simplified” version. The choice is which representation of the same content to send. That bounded set is what makes the negotiation honest: there is no third answer that could be a different page.

The function that decides is preferredProducedMediaType. It returns text/html when Accept is absent or blank (a plain fetch gets the page) and null when every produced type is explicitly rejected. null is the important case: it means “the client’s constraints are satisfiable by neither representation,” which is the only correct reason to return 406 Not Acceptable.

the accept parse

The parser implements RFC 9110 media-range semantics rather than substring matching. Each comma-separated entry is parsed into a type, an optional q value clamped to [0, 1], and a specificity (*/* is 0, type/* is 1, and an exact type is 2). For each produced type, the most specific matching entry wins; ties at the same specificity break toward the earliest entry in the header. The produced type with the highest resulting q wins; a q of 0 counts as a rejection, not a match.

That ordering is what makes Accept: text/html;q=0, */*;q=1 produce Markdown: the exact text/html;q=0 outranks the wildcard for HTML, so HTML is rejected and Markdown falls through to */*. And Accept: text/html;q=0, text/markdown;q=0, */*;q=1 produces 406, because both produced types were explicitly refused. The tests encode these cases directly: a browser header like text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 resolves to HTML, */* alone resolves to HTML, and text/markdown resolves to Markdown. A colocated property test states the law in one line (the chosen type is produced or the request is rejected) and the named cases cover the tiebreak, wildcard-override, and clamped-q edges a hand-rolled parser usually gets wrong.

the middleware

The middleware in negotiation.ts is a request rewriter, not a renderer. It normalizes the path, then asks three questions in order. Is this path excluded from negotiation? Is this a .md request that should be rewritten to the internal Markdown route? Or is this an HTML request whose Accept header prefers Markdown?

The exclusion list is the load-bearing part. isNonNegotiatedPath covers /api/, /_next/, /_vercel/, /_emoji/, /_media/, and /agent-markdown/ prefixes; exact matches for /robots.txt, /sitemap.xml, /llms.txt, /manifest.webmanifest, feed paths, and icon routes; every */opengraph-image social-card route; and asset paths under /direct/, /kb/, /writing/, and /reading/ that contain file extensions. Social images in particular are never negotiated: an opengraph-image request is an image request, not a document request, and returning Markdown for it would be a bug. The middleware matcher in middleware.ts already strips api/, _next/, _vercel/, _emoji/, _media/, and agent-markdown/ before this logic runs; the path-level checks are the second line.

For a .md URL (say /writing/some-post.md) stripMarkdownSuffix recovers the canonical page path and rewriteToMarkdown rewrites to /agent-markdown/writing/some-post. For a plain path whose Accept prefers Markdown, the same rewrite happens in place. For a plain path that stays HTML, the response gets Vary: Accept appended to any existing Vary, plus a Link header advertising the alternates. setAcceptVary appends rather than overwrites, so a route that already varies on another header keeps both values.

The Link header is the discoverability half of the contract:

Link: </llms.txt>; rel="describedby", </reference/web-delivery/content-negotiation-markdown.md>; rel="alternate"; type="text/markdown"

/llms.txt is always linked as describedby. The .md sibling is linked as rel="alternate" only when the path is a known Markdown page (isKnownMarkdownPage) so a URL that has no Markdown projection does not advertise one.

the markdown projection

The internal /agent-markdown route is force-static and renders through generateStaticParams from knownMarkdownPaths(), the same typed registries that drive the sitemap and llms.txt. There is no filesystem read in the middleware path. pageMarkdown(pathname) returns the projection for a known path or null, and null returns a Markdown 404 rather than an empty 200.

The representation is honest about gating. A subscriber lesson’s .md includes the free body, a horizontal rule, a one-line note that the remainder is free for subscribers on the canonical page, and then the gated body itself, because the gate is a presentation gate on the HTML, not a content boundary, and the .md is the same document. The subscriber content stays in the HTML document too, under .reference-subscriber__content with isAccessibleForFree: false in the Article structured data. Nothing is withheld from either representation; the HTML renders the gate prompt inline.

The route is also honest about what it does not know. A request for a .md that is not a known page (say /drafts/something.md or a typo) gets a Markdown 404 with recovery links, not an HTML error page and not an empty body. And a .md path that collides with a registered redirect source falls through to the redirect handler instead of rewriting, because a redirect’s .md should redirect, not silently produce a document.

Response headers differ deliberately by type. Markdown is text/markdown; charset=utf-8 with Cache-Control: public, s-maxage=60, stale-while-revalidate=86400, cacheable at the edge for a minute, revalidatable for a day. The 406 is plain text with Cache-Control: public, max-age=0, must-revalidate, because a negotiation refusal should never be pinned.

why this is not cloaking

Cloaking is serving different content to a crawler than to a user, usually detected by user-agent, to rank for terms the user-visible page does not contain. This is the opposite on every axis:

  • The content is identical. The .md is a projection of the same registry data that renders the HTML: same title, same body, same canonical URL.
  • The trigger is Accept, not user-agent. A browser sending Accept: text/markdown gets the same Markdown a crawler gets; a crawler sending a browser Accept gets the same HTML a browser gets.
  • The negotiation is declared. Vary: Accept tells caches the representation depends on it; the Link header advertises the alternate on the HTML itself; /llms.txt documents that public HTML pages serve Markdown for Accept: text/markdown and that sibling .md URLs are the same representation.
  • The gate state is identical. Subscriber content is in the document in both representations, marked isAccessibleForFree: false, not removed for one client and shown to another.
  • The 406 path exists. A client that refuses both produced types gets told so, not served a representation it did not ask for. A negotiation that cannot say “neither” is not a negotiation; it is a default with extra steps.

The distinction that matters is representation versus content. Negotiating on Accept to choose between HTML and Markdown of the same page is what the header is for. Serving a different page to a different client is what cloaking is. The former is honest because it is declared, bounded, and symmetric; the latter is dishonest because it is none of those. If you implement this yourself, the load-bearing parts are the boring ones: a real Accept parser, a real Vary, a 406 that can actually happen, and an alternate that is advertised rather than discovered by sniffing.

sources

  • The monorepo behind this site (projects): app/agent-discovery/accept.ts, negotiation.ts, paths.ts, response.ts, app/agent-markdown/[[...slug]]/route.ts, and middleware.ts.
  • RFC 9110 §12.5.1: Accept: the media-range and q-value semantics the parser implements.
  • Google Search Central: cloaking: the policy this design is built to stay clearly outside.