robots.txt, sitemap.xml, and an Atom feed are usually maintained as three separate files, which means they can tell three different stories. A page can be disallowed in robots but still listed in the sitemap. A post can appear in the feed but be noindex in its metadata. A draft route can be reachable, excluded from the sitemap, and quietly indexable because nobody remembered the meta tag. None of these are syntax errors; each file is individually well-formed. The problem is that they are independently edited descriptions of the same public surface, and independent descriptions drift.
The fix is to treat discovery as a contract, not a checklist: every crawler-facing artifact is generated from one typed registry, so the only way to make a page public is to admit it, and the only way to admit it is in one place. This page’s own sitemap entry exists because its admission record does.
one registry, three artifacts
On hraness.com, the sitemap’s /reference URLs are not a list anyone maintains. They are indexableReferenceRoutes (the projection of the lesson registry filtered by the admission registry) mapped into sitemap entries with lastModified dates that come from the same records. The feed is the same shape: the writing feed derives from indexableWritingPosts, and the reading feed derives from indexableReadingDetailEntries. llms.txt is generated from the same catalogs. There is no second list to forget.
That single-source discipline is what makes the artifacts mutually consistent by construction. A lesson that has not been admitted does not appear in the sitemap, does not appear in llms.txt, and does not appear in the feed. A lesson that is admitted appears in all three with the same canonical URL, because all three compute the URL from the same referenceLessonPath helper rather than concatenating strings.
robots is policy, sitemap is inventory
It helps to be precise about what each artifact claims. robots.txt is fetch policy: which paths a well-behaved crawler may fetch. This site’s app/robots.ts is twelve lines:
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: "*",
allow: "/",
disallow: ["/drafts", "/relay"],
},
sitemap: `${site.canonicalUrl}/sitemap.xml`,
host: site.canonicalUrl,
};
}It allows /, disallows the two private surfaces, publishes the sitemap URL, and declares the canonical host. That is the whole policy. The shared createPublicRobots helper builds the same shape from a validated origin, and createPrivateRobots builds the opposite (a flat disallow: "/") for products that should not be crawled at all.
The sitemap is inventory: the set of URLs worth indexing, with update times. createSitemap validates every entry’s path as an OwnedPath, resolves it against the bare HTTPS origin, and throws if two entries normalize to the same URL. A sitemap that can contain duplicates or a non-owned URL is a sitemap that can lie; the builder makes both impossible to express. And because the entries are computed from the registries at build, a sitemap that disagrees with the site is a build failure, not a stale file.
The asymmetry between the two is the point people miss. robots.txt can only say “don’t fetch”; it cannot un-index a URL that is already known. The sitemap can only say “consider these”; it cannot say “ignore the rest.” Neither one controls whether a fetched page is eligible for the index. That is page-level robots metadata, which is why the two layers have to agree.
The sitemap’s lastModified is part of the same contract. Each entry’s date comes from the content registry that produced the page, converted to an exact UTC date, not the time the sitemap was generated. A sitemap whose dates all say “today” tells a crawler to recheck everything and believe nothing; dates that come from the records themselves describe what actually changed.
the feed is not a sitemap
An Atom feed looks like discovery but has different semantics. It is a subscription document: ordered, dated entries meant for readers that already chose to follow. That means its entries are the same admitted set as the sitemap, but its metadata is richer: each entry carries a published or updated date, a summary, an author, and, for reading notes, a via link to the external source rather than a rewritten authorship.
The serializer is strict about the parts that make a feed trustworthy. createAtomFeed requires absolute HTTP(S) URLs for the feed and every entry, escapes every foreign value before it enters XML, emits type="text" constructs, and refuses to guess an update date for an empty feed:
if (explicitUpdate === null) {
throw new RangeError("An empty Atom feed requires an explicit update date.");
}When entries exist, the feed’s <updated> is the latest entry update unless an explicit later date is provided: the feed timestamp describes the content, not the moment the file happened to be generated. atomResponse serves it as application/atom+xml; charset=utf-8 with Cache-Control: public, max-age=0, must-revalidate, so aggregators revalidate rather than pin a stale copy.
The two feeds are deliberately separate documents: /writing/atom.xml carries first-party posts, /reading/atom.xml carries maintained notes about external work. Combining them would blur the distinction between “things this site published” and “things this site recommends,” which is exactly the kind of drift the single-registry approach exists to prevent. Provenance is also honest at the entry level: when an admitted post is an AI-drafted exception, the writing feed appends the drafting disclosure to that entry’s Atom summary rather than changing the authorship of human-originated entries.
The same projection extends to llms.txt, which is generated from the same catalogs and tells agents the same story the sitemap tells crawlers: the public reference categories and lessons, with the Markdown alternates those pages serve. There is no fourth file to keep in sync.
noindex is page-level, not path-level
The most common discovery bug is treating robots.txt as an indexing control. It is not. A disallowed page can still appear in results (with a bare URL and no snippet) if it is linked from elsewhere, because the crawler that honors Disallow never fetches the page and therefore never sees the noindex it would have carried. The correct order for a private-but-reachable page is: allow the fetch, noindex the page.
That is why the admission model is page-level. The reference index, each category, and each lesson each carry an admission record with a lifecycle (indexable or quarantined) and the route metadata derives its robots from that record. A quarantined page is reachable, has a canonical URL, and serves NOINDEX_ROBOTS; it does not appear in the sitemap, the feeds, or llms.txt. A draft under /drafts goes further: it is both disallowed in robots.txt and absent from every generated artifact, because it should not even be fetched. The two mechanisms answer different questions and the contract keeps them from contradicting each other, and it is why a reader that finds a draft URL still cannot get it indexed, because the page itself carries the noindex the registry assigned.
the admission is the contract
The reason the artifacts can all derive from one place is that “public” is a decision with fields, not a boolean. An admitted reference URL records a reader job, a non-obvious answer, an original contribution, a host fit, an adoption or authorship evidence trail, an accountable owner, a named reviewer identity and type, a review date, a score, and a reassessment date. A route enters the sitemap and feeds only when that record says indexable; everything else is generated but not discovered.
This is what “discovery is a contract” means operationally. The sitemap does not say what to crawl; it says what has been admitted. The feed does not say what is new; it says what has been admitted and when it changed. robots.txt does not say what exists; it says what may be fetched. Each artifact is an honest projection of the same decision, so the only way for any of them to lie is for the registry itself to lie, and that is a much smaller thing to audit than three files.
It is also what keeps the artifacts truthful under edit. A lesson added to the registry appears in the sitemap, the feed, and llms.txt in the same build; a lesson whose admission lapses disappears from all three the same way. There is no window where one file knows about a page and another does not, because there is no second file.
sources
- The monorepo behind this site (projects):
app/robots.ts,app/sitemap.ts,app/feeds/atom.ts,app/feeds/paths.ts, andapp/reference/admissions.ts. - web-discovery:
createPublicRobots,createPrivateRobots,createSitemap, and theOwnedPathvalidation insrc/discovery.ts. - RFC 9309: Robots Exclusion Protocol: what
robots.txtdoes and does not control.