Masthead now generates sitemap.xml, robots.txt and llms.txt

A website should not need an SEO plugin to explain what it contains.

Masthead already knows which pages and posts belong to a site, which ones are published, and what their public URLs are. So it seemed a little silly to make every Masthead site owner configure the machinery that tells search engines and other crawlers about that content.

Masthead now generates three files automatically:

• sitemap.xml
• robots.txt
• llms.txt

There is nothing to configure.

The sitemap

A sitemap is a machine-readable list of the public URLs on a website.

For example, a small Masthead site might expose:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-07-11</lastmod>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-07-12</lastmod>
  </url>
  <url>
    <loc>https://example.com/posts/my-first-post</loc>
    <lastmod>2026-07-12</lastmod>
  </url>
</urlset>

The sitemap is generated from the same underlying data that powers the website. There is no separate list of URLs for the site owner to maintain.

Sitemaps help search engines discover URLs, particularly when those URLs might not otherwise be easy to find through normal crawling. They aren’t a guarantee of indexing or ranking, but they are useful infrastructure for a publishing system.

robots.txt

Masthead also generates robots.txt.

A default Masthead site currently gets something along the lines of:

User-agent: *
Allow: /
Disallow: /search

Sitemap: https://example.com/sitemap.xml

The important part here is that Masthead isn’t trying to decide which pages should rank in search. It is simply describing the site’s crawl boundaries. The sitemap is also declared here, giving crawlers another way to discover it. The /search endpoint is different. A site’s internal search results aren’t really content that needs to be discovered and indexed, so Masthead keeps those out of the crawl surface by default.

What about llms.txt?

The third file is llms.txt. This one is less established than robots.txt and XML sitemaps, and it isn’t something we’re presenting as a magic “AI SEO” switch. The idea is simply interesting: if a site can describe its content to search engine crawlers, why not also provide a concise, machine-readable description of the site and its important content? A Masthead-generated llms.txt can contain a short description of the site followed by links to its important pages and posts.

For example:

# Example Site
A small software company building tools for developers.

## Pages

- [Home](https://example.com/)
- [About](https://example.com/about)
- [Customers](https://example.com/customers)

## Posts

- [Why we built our new tool](https://example.com/posts/why-we-built-our-new-tool)

The important thing is that this is generated from the site’s actual content. The site owner doesn’t have to maintain another document every time they publish something. Whether llms.txt becomes an important convention remains to be seen. Masthead can generate it because the cost of doing so is small and the information is already available.

This is a platform feature, not a theme feature

This is perhaps the most important part.

Masthead themes have complete control over the rendering of a website. A theme decides what the HTML looks like, that’s intentional. But sitemap.xml, robots.txt, and llms.txt aren’t part of the website’s visual rendering. They describe the website itself. So Masthead can provide them independently of the theme.

A theme can look completely different from another Masthead theme while both sites still get the same basic machine-readable infrastructure.

              Masthead
                 │
      ┌──────────┴──────────┐
      │                     │
  Website                 Discovery
      │                     │
   Theme              sitemap.xml
      │               robots.txt
 HTML / CSS /          llms.txt
  JavaScript

The theme is responsible for presenting the content, Masthead is responsible for knowing what the content is.

Custom domains work too

Masthead sites can use a masthead.site subdomain or a custom domain.

That distinction happens at the routing layer, so the generated discovery files don’t need to care how the site is hosted.

Example:

For a site at: example.masthead.site

The sitemap belongs at: https://example.masthead.site/sitemap.xml

---

If the same site uses: example.com

then Masthead generates: https://example.com/sitemap.xml

---

The public URL is derived from the site’s actual public origin rather than being something the theme author needs to configure.

No extra work for the publisher

Publishing software already knows:

• which content exists
• which content is published
• what each piece of content is called
• what its URL is
• when it was last changed
• which domain the site is being served from

So the publisher shouldn’t have to copy that information into three different files.

Masthead can just generate them.

• Publish a post, and it appears in the sitemap.
• Change a post, and its lastmod value can change.
• Unpublish it, and it disappears from the public sitemap.
• Change the site’s domain, and the generated URLs use the new domain.
• There is no plugin to install and no configuration page to maintain.

That’s the kind of thing a publishing system should simply take care of.