Skip to main content
Every domain of a brand can serve three discovery files. They tell search engines and AI crawlers which of your pages exist:
All three are off by default. Until you turn them on, each path returns 404 Not found on every domain of the brand. The same goes for listing a page: turning a file on doesn’t list anything until you tick individual pages.
It’s opt-in because most of a brand’s pages are funnel steps, upsells and checkouts. A sitemap that turned itself on would publish a map of all of them.

​
Setting it up

1

Turn the files on for the brand

Go to Settings → SEO. Under Discovery files, switch on Serve sitemap.xml, Serve robots.txt and, if you want it, Serve llms.txt. Then click Save.Turn on robots.txt whenever you turn on the sitemap. robots.txt is what points crawlers at the sitemap, and without it most of them never find it.
2

Tick the pages you want listed

Open each public content page (home page, blog, articles, product and about pages), go to its SEO tab and tick List this page in sitemap.xml and llms.txt. Then save the page.
3

Check the preview

Back in Settings → SEO, the page list at the bottom shows exactly what the files contain right now. It runs the same rules as the live files. A page you ticked that still can’t be listed shows a Not listed badge with the reason.
4

Verify on the live domain

Open https://your-domain.com/sitemap.xml and /robots.txt. Then submit the sitemap URL in Google Search Console.

​
Setting it up with the ef CLI

The same setup works from the ef CLI (version 0.16.0 or later), run inside a folder linked to the brand.
ef seo status shows which files are on, how many pages are listed, and warns when robots.txt is off or a page you ticked is excluded. ef seo pages prints every page that opted in, the exact URL it publishes, and for each excluded page, the reason.

​
ef seo reference

Booleans take true/false, on/off, yes/no or 1/0. To clear a text key, pass "":
While you’re at it, ef pages settings can also set a page’s SEO title and meta description. llms.txt uses both:
The site address settings (Preferred domain and Redirect http to https) are only in the admin, under Settings → SEO. The CLI doesn’t set them yet.

​
Which pages are listed

A page appears in sitemap.xml and llms.txt only when all of these are true:
  • List this page in sitemap.xml and llms.txt is ticked on the page, or on one of its variants.
  • The page is published.
  • Prevent Search Engine Indexing is off.
  • The page isn’t a checkout or upsell page.
  • It doesn’t require login or a password.
  • It isn’t set to be visible only through a funnel.
  • Its slug has no {placeholder}. A wildcard route stands for many URLs, not one.
  • It belongs to the domain being requested (or its www/non-www twin), or it’s a global page with no domain.
Each page is listed at the same URL as its <link rel="canonical">: / for the index page, /<slug> for everything else. Variants are never listed separately. The parent page stands for them. Each file lists at most 5,000 pages.
Hiding a page: turn on Prevent Search Engine Indexing. Don’t add a Disallow: line for it. robots.txt is public, so a Disallow list is a directory of the pages you wanted hidden. It also stops crawlers from reading the page’s noindex tag, so a page linked from somewhere else can still get indexed.

​
robots.txt

The generated file allows everything, blocks EF’s system paths (checkout processing, previews, redirect shims, /affiliates/, /upsell* and /decline*), and ends with a Sitemap: line when the sitemap is on:
Each system path is anchored ($, ?, /), so a rule like /b never blocks a content page such as /blogs. Additional directives (shown once robots.txt is on) adds your own lines verbatim, one directive per line. Don’t add a Sitemap: line. One is written for you.
If the domain is on Cloudflare with Managed robots.txt turned on, Cloudflare serves its own content-signals file when yours returns 404. Once you turn on Serve robots.txt, check the live file to see what Cloudflare adds to it.

​
llms.txt

Shown once llms.txt is on: Between the summary and the notes, each listed page becomes a link line, using its SEO title and meta description:

​
Site address: www, non-www and https

Search engines treat www.example.com and example.com, and http and https, as four different sites. The Site address section picks one:
  • Preferred domain: Don’t redirect, Without www or With www. The other version gets a permanent 301 redirect to it. It only applies when both versions are added to the brand as domains. A domain with only one version is left alone.
  • Redirect http to https: visitors and crawlers arriving over http get a 301 to https.
Only GET and HEAD requests are redirected. Checkout and tracking POSTs are never redirected, because a browser replays a redirected POST as a GET and loses the body. The path and query string carry over exactly, so affiliate parameters arrive unchanged. Whatever you choose here, sitemap.xml, robots.txt, llms.txt and every page’s canonical link always use https and the preferred domain.

​
Caching

Crawlers get the files with a one-hour Cache-Control. Saving a page or the brand’s settings clears the server-side copy right away, so a newly ticked page shows up on the next fetch. If you still see an old copy, it’s coming from Cloudflare or the browser cache.

​
Troubleshooting