What is llms.txt, and which AI crawlers actually asked for mine

What it is and where it comes from

A plain-text file at the root of your site, /llms.txt, in Markdown. It lists the pages you consider important, with a short description of each, so a language model can find its way around without crawling everything. The specification is short and worth reading in full.

Think of it as a sitemap written for a reader rather than for a parser: same intention, different audience, no standards body behind it.

What goes in it

• An H1 with the name of the site. This is the only required part.

• A blockquote with a one-line summary.

• H2 sections with lists of links, each link followed by a colon and a short description.

• Optionally, a Markdown version of each page at the same URL with a .md extension. This is the part that costs real maintenance.

What Google says

Google's guide to optimising for AI features tells you to do the same things that work for Search: unique content, structured data that matches what is visible, and pages a crawler can actually read. It does not mention llms.txt.

The AI features and your website page is equally explicit about the controls that do exist: robots meta tags, nosnippet, max-snippet and data-nosnippet. There is no llms.txt switch, and the documentation change log has never introduced one.

llms.txt against structured data

If your goal is that a machine understands who you are and what you sell, structured data is the mechanism that Google, Bing and the assistants built on their indexes actually consume. It is declared in the page, it is validated by public tools, and it is read today.

llms.txt is a proposal that some tools honour. Structured data is a standard that the search engines honour. If you only have time for one, it is not a close call.

What actually asked for it: 28 days of my own logs

This site serves a llms.txt generated from the same manifest as the sitemap. Between Aug 31 and Sep 27, 2026, across 56,424 requests to the zone, /llms.txt was requested 54 times.

Of the 54 requests, 24 were Chrome-Lighthouse (my own PageSpeed runs), 17 were Googlebot, and the rest were browsers, curl and three data scrapers. Not one AI crawler asked for it, and the same crawlers asked for robots.txt and the sitemap hundreds of times.

When it can still be worth it, outside Google

Documentation sites and API references are the real use case: a model that has to answer questions about your product benefits from a curated map, and some tools do read it.

For everyone else the order is the other way round. Make sure the crawler receives your HTML, declare structured data that matches what is visible, and only then worry about a file that nothing asked for. That order is what I work on in AI SEO.