What Is llms.txt? The Emerging Standard for AI Crawlers

llms.txt is a small markdown file that lives at the root of your website and tells AI systems what your site is about and which pages matter most. It is a proposed standard, published at llmstxt.org, and it exists because language models are bad at reading the average web page: navigation, cookie banners, scripts, and sidebars bury the content an AI actually needs.

Most coverage of llms.txt either sells it as a magic ranking switch or dismisses it as pointless. This post takes the harder middle path — what the file genuinely does, what it cannot do, and where adoption honestly stands, based on what we see scanning real sites. We will cover what llms.txt is, the problem it solves, the exact format, how it differs from llms-full.txt, and whether it deserves your time.

What is llms.txt?

Direct answer: llms.txt is a plain markdown file served at your website's root (yourdomain.com/llms.txt) that gives AI systems a concise, curated overview of your site: what it is, what it offers, and links to your most important pages with short descriptions. It is a proposal, not an enforced web standard.

The proposal came from Jeremy Howard, co-founder of Answer.AI, in September 2024. The idea borrows its shape from files webmasters already know. robots.txt tells crawlers what they may access. sitemap.xml tells search engines what exists. llms.txt does a third job neither of those was designed for: it tells an AI system what your site means.

The choice of markdown is deliberate. Language models read markdown natively — it is close to plain text, it carries structure through headings and lists, and it contains none of the div soup that makes HTML expensive to parse. A model handed a good llms.txt file can understand a site in a few hundred tokens instead of burning thousands on boilerplate.

The other defining trait is curation. A sitemap is exhaustive by design: every indexable URL, no judgment applied. llms.txt is the opposite. You choose a handful of pages, put them in a sensible order, and describe each one in a sentence. That editorial judgment is the entire value of the file. You are answering, in advance, the question every retrieval system asks: of everything on this site, what should I actually read?

Nothing about the file is binding. No crawler is obligated to fetch it, and no standards body has ratified it. It is a convention with growing tooling behind it — closer to where robots.txt was in its early years than to a formal specification.

What problem does llms.txt solve?

Direct answer: AI systems that fetch web pages at answer time work with limited context windows and messy HTML. Navigation, popups, scripts, and boilerplate crowd out the substance. llms.txt solves this by offering a clean, markdown version of your site's structure that a language model can read in one pass, without parsing clutter.

There are two distinct moments when an AI system touches your website. The first is training: large crawls that feed model development. The second is inference — the moment a user asks a question and the system fetches pages to ground its answer. llms.txt is aimed squarely at the second moment.

At inference time, everything is constrained. The system has a token budget, a latency budget, and a user waiting. When it pulls your page, it gets your header, your cookie banner, your newsletter modal, your footer with forty links — and somewhere in the middle, the content. Extraction pipelines strip some of this, but they are imperfect, and every wasted token is context the model cannot spend on your actual message.

llms.txt short-circuits that. It hands the system a pre-cleaned summary: here is who we are, here is what we offer, here are the pages worth reading, each with a one-line description. For a model deciding what to fetch next, that map is exactly the missing piece.

Think about what that means for a service business. When someone asks an assistant to compare providers in an area, the system may fetch several candidate sites in seconds. The site that greets it with a clean map of services, pricing, and proof gets represented accurately. The site that greets it with a script-heavy homepage gets summarized from whatever fragments survive extraction.

An honest caveat belongs here: the large AI search engines run their own indexes and extraction pipelines and do not need your llms.txt to function. The file is a shortcut you offer, not a gate they must pass through. That distinction matters when we get to adoption below.

What does an llms.txt file look like?

Direct answer: An llms.txt file is plain markdown with a fixed structure: an H1 with the site or project name, a blockquote containing a one-paragraph summary, optional free-form notes, and then H2 sections containing bulleted link lists, each link followed by a short description of what that page covers.

Here is a compact example for a fictional plumbing company:

  # Acme Plumbing
  > Acme Plumbing is a family-run plumbing company serving the
  > Portland metro area, offering repair, installation, and
  > emergency service.

  ## Services
  - Drain cleaning: https://acmeplumbing.example/drain-cleaning
  - Water heaters: https://acmeplumbing.example/water-heaters

  ## Company
  - About us: https://acmeplumbing.example/about

  ## Optional
  - Blog: https://acmeplumbing.example/blog

Each element has a job. The H1 is the only required part — it names the site. The blockquote is the elevator pitch, and it is worth writing carefully because it is the single most likely passage to be read and repeated. The link sections group your key pages by theme. One notation detail: in a real file, each bullet is a standard markdown link — the page title in square brackets, the URL in parentheses — followed by a colon and a one-line description of what the page covers; the spec shows the exact syntax. That description tells a model what it will get before it spends a fetch finding out.

The "Optional" section has specific semantics in the spec: it marks links that can be skipped when a system is short on context. Everything above it says read this; everything under it says if you have room.

That is the whole format. No XML, no key-value syntax, nothing to compile. If you can write a readme, you can write an llms.txt file.

How is llms.txt different from llms-full.txt?

Direct answer: llms.txt is an index: a short, curated map that points AI systems to your key pages. llms-full.txt is the expanded version: the full text of your important content compiled into one large markdown file. The first helps a model navigate; the second hands it everything in a single request.

	llms.txt	llms-full.txt
Role	Curated index of key pages	Full content in one file
Typical size	A few kilobytes	Can run very large
How a model uses it	Reads the map, then fetches pages	Reads everything in one pass
Best fit	Most business sites	Documentation-heavy sites
Maintenance	Update when key pages change	Regenerate whenever content changes

The distinction matters because the two files serve different kinds of sites. Documentation sites — developer tools, APIs, software products — get real value from llms-full.txt: their users ask AI assistants detailed questions, and having every doc page compiled into one clean markdown file means an assistant can pull complete, current answers without dozens of fetches. Several documentation platforms now generate both files automatically.

A typical business site rarely needs the full version. Your service pages and guides are not thousands of pages of reference material, and a well-curated llms.txt already points to everything that matters. Generating an llms-full.txt is not harmful, but it is maintenance overhead most sites do not need yet.

If you do publish both, keep them consistent: same root location, same curation logic, and regenerate the full file on every meaningful content change so it never drifts out of date. A stale full-text file is worse than none, because it hands AI systems an outdated copy of your site with your name on it.

Who actually reads llms.txt today?

Direct answer: Honestly: adoption is early and uneven. The major AI providers document their crawlers' behavior toward robots.txt, but none has published equivalent, unambiguous support for llms.txt. Some AI-adjacent tools and frameworks read it today. Treat publishing one as cheap insurance for where AI search is heading, not as a switch that turns on citations.

This is the section most write-ups fudge, so let us be precise about what is and is not known. OpenAI, Anthropic, Perplexity, and Google all publish the user agents of their crawlers and document how those crawlers treat robots.txt. There is no equivalent official documentation committing any of them to fetching llms.txt on a schedule and feeding it into answers. Meanwhile, a growing set of developer tools, AI coding assistants, and documentation platforms do read and generate the files, and the convention keeps spreading from that direction.

Building GroundScore, the pattern I keep seeing when we scan sites is that almost nobody has an llms.txt file yet — and among the sites that do, many were clearly generated once and never touched again, with links pointing at pages that have since moved. An accurate, current file is rare enough to be a small differentiator on its own.

So why bother? Because the cost-benefit is lopsided. The file takes under an hour to create, carries no technical risk, and sits exactly where AI systems will look if support broadens — the same adoption path robots.txt itself walked decades ago. If you would rather not write it by hand, our free llms.txt generator builds one from your site in a few minutes.

Cheap insurance is still insurance. Just do not let anyone sell it to you as a citation machine.

Frequently asked questions

Where does llms.txt go on my website?

At the root of your domain, so it loads at yourdomain.com/llms.txt — the same level as robots.txt. It should serve as plain text or markdown, not as a rendered HTML page. If your business spans multiple subdomains, each subdomain that matters should serve its own file.

Does llms.txt improve Google rankings?

No. llms.txt is not a ranking signal for traditional search, and Google has not adopted the standard. It is aimed at AI systems that read content when composing answers. Keep your SEO fundamentals — crawlable pages, sitemaps, structured data — and treat llms.txt as a separate, AI-facing layer.

Do I still need robots.txt and sitemap.xml?

Yes, both. robots.txt controls which crawlers may access your site, and sitemap.xml helps search engines discover your URLs. llms.txt does a third job — helping AI systems understand your content — and replaces neither. The three files coexist at your site root and cover different stages of the pipeline.

Can publishing llms.txt hurt my site?

There is no meaningful technical risk. Systems that do not support the standard simply ignore the file. The realistic failure modes are self-inflicted: letting links go stale, describing pages inaccurately, or listing weak content you would not want an AI to treat as representative. Curate it and keep it current.

How long should an llms.txt file be?

Short. The file is an index, not an archive — for a typical business site, a one-line summary plus ten to thirty well-described links is plenty. If you find yourself listing hundreds of URLs, you have recreated your sitemap and lost the curation that makes the file useful.

How often should I update llms.txt?

Whenever a key page launches, moves, or retires — and on a quarterly review as a backstop. An llms.txt file that points at dead URLs or describes an old version of your business misleads the exact systems you published it to inform. Automate regeneration if your platform supports it.

The bottom line

llms.txt is a modest, sensible idea: a curated markdown map of your site, published where AI systems can find it. It will not transform your AI visibility overnight, and anyone claiming otherwise is selling something. But it costs almost nothing, it cannot hurt, and it is exactly the kind of groundwork that compounds if adoption follows the path robots.txt once took.

Publishing the file is one small piece of AI search readiness — crawler access, structured data, and content that answers real questions still carry more weight today. See where you stand on all of it with a free AI visibility check; it takes about a minute and does not require an account.