What is `llms.txt`, Why It Matters, and How to Use It
by Swasthik K,
Your docs have a new audience, and most of them aren't human. According to recent research, nearly 35% of web traffic is now bots. Search crawlers have been around forever. What's changed is AI agents like coding assistants, chatbots, and answer engines, fetching your pages because someone asked them a question.
People have started calling the response AEO (Agentic Engine Optimization). Same idea as SEO: help automated readers find the right content. The timing works differently though. Google crawls your site ahead of time and builds an index. An agent usually grabs a handful of pages the moment it needs an answer. Your sitemap helps the first case. It doesn't do much for the second.
That's where llms.txt comes in: a Markdown file at your domain root that tells agents what your project is and which pages are worth reading.
What is it?
Jeremy Howard from Answer.AI proposed this in September 2024. You put a file at yourdomain.com/llms.txt (or /docs/llms.txt if your docs live under a subpath). It's a curated index for models, not a page meant to render in a browser. No hero section, no sidebar. Just links with enough context that an agent knows which one to fetch. Original proposal.
markdown
# Acme Docs
> Acme is an API for sending transactional email.
## Docs
- [Quickstart](/docs/quickstart.md): Send your first email in 5 minutes
- [Authentication](/docs/auth.md): API keys and OAuth setup
- [Webhooks](/docs/webhooks.md): Delivery and bounce event payloads
## Optional
- [Changelog](/changelog.md): Release history, safe to skip for most tasks
That's basically the whole spec:
- H1 with the project name
- A blockquote summary
- H2 sections grouping links, each with a short description
- An "Optional" section for pages an agent can skip when context runs tight
What about llms-full.txt?
Docs platforms like GitBook and Mintlify often generate llms.txt as the index and llms-full.txt with your entire documentation concatenated into one file. They do this so agents can choose to follow links from the index, or grab everything in a single fetch.
llms.txt is what you need. Reach for llms-full.txt when an agent needs broad context from your docs without making a dozen HTTP requests. Once the file outgrows a model's context window, it stops being useful and agents are better off with the index anyway.
How is this different from robots.txt and sitemap.xml?
Fair question. Those files already exist.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Tells crawlers what they're allowed to access | Lists every indexable page that exists | Curates the pages that matter, with context |
| Audience | Search engine crawlers | Search engines | LLMs and AI agents, usually on-demand |
| When it's used | Crawl time | Crawl/index time | When someone (or an agent) is actively asking a question |
| Format | Plain text, permission rules | XML, exhaustive | Markdown, curated |
| Includes external links? | No | No | Yes, if relevant |
robots.txt is about permission. sitemap.xml is about completeness. llms.txt is about curation. Three different jobs.
How to add one
If your site is static, put the file in public/:
/public/llms.txt
markdown
# Acme
> We build a self-serve API for sending transactional email.
## Docs
- [Quickstart](/docs/quickstart.md): Setup and first API call
- [API reference](/docs/api.md): Full API reference
## Policies
- [Privacy policy](/privacy.md): Privacy policy
- [Terms of service](/terms.md): Terms of service
Done. It's now live at yourdomain.com/llms.txt.
If your content changes often, generate it from a route handler:
/app/llms.txt/route.ts
TypeScript
export async function GET() {
const posts = await db.post.findMany({
take: 5,
orderBy: { createdAt: "desc" },
});
const content = `# My Blog
> Latest posts and resources.
## Recent Posts
${posts.map((p) => `- [${p.title}](/blog/${p.slug})`).join("\n")}
`;
return new Response(content, {
headers: { "Content-Type": "text/markdown; charset=utf-8" },
});
}
Same URL. Regenerates every time someone hits it.
Should you bother?
I don't want to oversell this. Adding llms.txt won't suddenly get you cited by every AI answer engine on the internet. Adoption is still patchy. Some agents check for it, many just crawl whatever HTML they find. Chrome Lighthouse now checks for llms.txt in its audits, which could push more sites to add one.
It's also about ten minutes of work. An extra file in public/ that either helps an agent find your quickstart or sits there doing nothing. At least you're not leaving agents to figure out your API from a marketing page.