What is llms.txt, and which AI crawlers asked for mine

What is llms.txt, and where does it come from?

A plain-text file at the root of your site, /llms.txt, in Markdown. It lists the pages you consider important, with a short description of each, so a language model can find its way around without crawling everything. The specification is short and worth reading in full.

Think of it as a sitemap written for a reader rather than for a parser: same intention, different audience, no standards body behind it.

What goes in an llms.txt file?

An llms.txt file is plain Markdown with a simple structure. It has a title, a one-line summary and lists of links to the pages that matter, each with a short description. According to the proposal, only the title is required; the rest is optional:

• An H1 with the name of the site. This is the only required part.

• A blockquote with a one-line summary.

• H2 sections with lists of links, each link followed by a colon and a short description.

• Optionally, a Markdown version of each page at the same URL with a .md extension. This is the part that costs real maintenance.

What does Google say about llms.txt?

Google published its guide to optimizing for AI features on May 15, 2026. It says you don't need new machine-readable files, AI text files or Markdown to appear in Google Search, because Google Search doesn't use them. On June 15, 2026 it added that llms.txt files aren't needed and won't help or hurt your visibility or rankings.

The AI features and your website page is equally explicit about the controls that do exist: robots meta tags, nosnippet, max-snippet and data-nosnippet. There's no llms.txt switch: the only llms.txt entry in the documentation change log is that June 15 clarification.

So llms.txt won't rank you on Google. If your page isn't even indexed, start with the seven reasons a website doesn't show up on Google. And if your goal is AI answers, what does work is in my guide on how to rank in AI Overviews.

If you already paid for an llms.txt, there's no need to delete it, because Google says its search engine doesn't use it. What's worth checking is whether it was sold to you instead of what Google does ask for: your page arriving whole in the HTML. That's the first thing I check in the free diagnostic, in writing and without asking for access.

llms.txt or structured data?

Say your goal is that a machine understands who you are and what you sell. Then structured data is the mechanism Google documents for understanding a page. It's declared in the page, it's validated by public tools, and it's read today.

llms.txt is a proposal that some tools honor. Structured data is a standard that the search engines honor. If you only have time for one, it isn't a close call, even though Google says neither is required to appear in its AI features.

Which crawlers asked for it? My own logs, from day one

This site serves a llms.txt generated from the same manifest as the sitemap. From September 15, 2026, the day the domain was registered, to September 30, across 32,811 requests to cardonaseo.com, /llms.txt was requested 54 times. Counted by user agent with the Cloudflare GraphQL Analytics API on October 1, 2026.

Of the 54 requests, 24 were Chrome-Lighthouse (my own PageSpeed runs) and 12 were Googlebot. The other 18 were browsers, curl, two small crawlers and two requests that used the Googlebot name without its full signature. Not one AI crawler asked for it, and the same crawlers asked for robots.txt and the sitemap hundreds of times.

A much larger sample points the same way. Ahrefs looked at 137,210 domains using its web analytics and published its study on June 15, 2026. Among those with a valid llms.txt, 97% got no request for the file at all in May 2026.

The largest group of the requests that did come, 21.7%, were SEO audit tools, Ahrefs' own included. GPTBot accounted for 4.51%.

Ahrefs adds a caveat that applies to my table too: a bot requesting the file isn't proof that it read it or uses it. Requested isn't read. That's why I count requests and don't write that anyone read it.

What about Googlebot's 12 requests? Google writes in its guide to generative AI features that it may discover and crawl many kinds of files besides HTML. It adds that this doesn't mean the file is treated in a special way.

Its June 15, 2026 note says llms.txt won't help or hurt you in Search. So I read those 12 requests as a crawl without use. Googlebot fetched the file, and Google says its search engine doesn't use it.

When is llms.txt still worth it, outside Google?

Documentation sites and API references are the case the proposal itself has in mind. A model that has to answer questions about your product benefits from a curated map. For a site selling a service, it's rarely the place to start.

For everyone else the order is the other way around: first make sure the crawler receives your HTML. Then declare structured data that matches what is visible, and only then worry about a file that nothing asked for. That order is what I work on in AI SEO.

If your platform generates the file for you, check that it says the same as your sitemap. Also check that both lists only carry pages you want found. On the two sites I measured on Base44, the automatic sitemap and llms.txt included pages marked not to be indexed.

Here are two lists from my site. My llms.txt: [paste the text]. My sitemap: [paste the text].
First tell me, in full, the addresses that are in one list but not the other. Then number the addresses in each list and give each total.
Don't summarize or group them: if a list is long, keep numbering. Don't make up addresses: if a list reached you cut off, say so.
If I gave you the files' addresses instead of the text and you can't open them, say so in the first line.

Then open each leftover address yourself. If it answers “not found” or carries a noindex (the noindex check tells you), it shouldn't be in either list.

Careful: a noindex sent in the X-Robots-Tag header doesn't show in View page source. That's how it came on the two leftover pages of a site I measured on October 2, 2026.

Frequently asked questions