What is llms.txt? The New Standard AI Systems Use to Understand Your Website

Your website has a robots.txt that tells search engine crawlers where they can go. Maybe a sitemap.xml that lists your pages. But when ChatGPT, Claude, or Perplexity visits your site, those files tell the AI almost nothing about what your site actually does.

That's the gap llms.txt fills. It's a plain text file that sits at your site root and gives AI systems a structured summary of your website — what it is, what pages matter, and how to describe it. No JSON, no XML. Just a text file with some Markdown-ish formatting.

The standard was proposed by Jeremy Howard (the fast.ai founder) and documented at llmstxt.org. It's been picked up by Stripe, Vercel, Cloudflare, Anthropic, and a growing list of developer-focused companies. If you haven't heard of it yet, you will.

Where llms.txt Came From

The problem is straightforward. When an LLM browses the web to answer a question about your product, it lands on your homepage and starts reading HTML. It has to figure out what you do by parsing marketing copy, navigation menus, and footer links. That works okay for big, well-known brands. For everyone else, the AI's understanding of your site is incomplete at best, wrong at worst.

Jeremy Howard proposed llms.txt in late 2024 as a solution. The idea: give AI systems a single file they can read to understand the essentials about your site, without crawling dozens of pages and piecing things together.

It's the same logic behind robots.txt (which solved the "where can I go?" question for crawlers) and sitemap.xml (which solved "what pages exist?"). llms.txt answers a different question: "What is this website, and what should I know about it?"

The proposal gained traction quickly. By mid-2025, several major developer platforms had published their own llms.txt files. By early 2026, it's becoming a default part of website setup for any company that cares about AI visibility.

The Format: What Goes in an llms.txt File

An llms.txt file uses a simple Markdown-style structure. Here's a minimal example:

# Acme Dev Tools

> A CLI toolkit for deploying containerized apps to any cloud provider.
> Supports AWS, GCP, and Azure. Open source, written in Go.

## Docs

- [Getting Started](https://acme.dev/docs/quickstart): Install the CLI and deploy your first app
- [Configuration Reference](https://acme.dev/docs/config): All config options explained
- [Plugin System](https://acme.dev/docs/plugins): How to write and publish plugins

## Blog

- [Why We Built Acme](https://acme.dev/blog/why): The problem with existing deployment tools
- [v3.0 Release Notes](https://acme.dev/blog/v3): Multi-cloud support, new plugin API

## Optional

- [Pricing](https://acme.dev/pricing): Free for open source, $49/mo for teams
- [Status Page](https://status.acme.dev): Uptime and incident history

That's the whole thing. Let's break down the sections.

The H1: your site name

The top-level heading is your site or product name. One line. Nothing else.

The blockquote: your elevator pitch

Right after the H1, a blockquote gives a 1-3 sentence description of what your site does. This is the single most important part of the file. When an AI needs to describe your product in one sentence, it'll use this. Make it specific. "A project management tool" is vague. "A project management tool for engineering teams with Kanban boards, sprint planning, and GitHub integration" is useful.

The sections: your key pages

Below the blockquote, you list your important pages grouped by H2 headings. Each entry is a Markdown link followed by a colon and a short description. The spec uses sections like ## Docs, ## Blog, and ## Optional, but you can use whatever headings make sense for your site.

The ## Optional section is specifically for pages that provide useful context but aren't essential. Pricing, status pages, legal docs — things an AI might reference in certain contexts but doesn't need for a basic understanding of your site.

Why llms.txt Matters for AI Discovery

Here's the practical problem llms.txt solves. Without it, this is what happens when someone asks ChatGPT about your product:

1. The AI visits your homepage
2. It reads through your marketing copy (which is written for humans, not machines)
3. It maybe follows a few links
4. It synthesizes an answer from whatever fragments it found

The result is usually vague, sometimes wrong, and never as good as it could be. The AI might describe your product using language from a competitor's comparison page. It might miss your key differentiators entirely. It might cite an outdated blog post instead of your current docs.

With llms.txt, you control the narrative. You tell the AI: "Here's what we are. Here's what matters. Here are the authoritative sources." The AI gets a structured overview before it starts crawling, so its understanding starts from a much better baseline.

This isn't theoretical. Sites with llms.txt get more accurate citations in AI-generated answers. When Perplexity or ChatGPT references your product, it's more likely to use your own description rather than guessing.

AI discovery vs. traditional SEO

Traditional SEO is about ranking in a list of 10 blue links. AI discovery is about being the source an AI pulls from when it generates an answer. Different game, different rules.

Google cares about backlinks, keyword density, page speed, and a hundred other signals. AI systems care about one thing: can I understand this site well enough to give an accurate answer about it? llms.txt is the most direct way to help with that.

This doesn't replace SEO. You still want Google traffic. But AI-generated answers are eating into that traffic, and you need to be present in both channels.

How to Create Your Own llms.txt

Here's the step-by-step. The whole process takes about 20 minutes.

Step 1: Write your description

Start with a clear, specific one-liner about what your site does. Avoid marketing speak. Write it like you'd describe your product to another developer at a conference.

# Your Site Name

> One to three sentences. What does this site/product/service do?
> Be specific. Mention the tech stack, target audience, or key capability.

Bad: "We provide innovative solutions for modern businesses."

Good: "An open-source error tracking platform for Python and JavaScript apps. Self-hosted or cloud. Integrates with Slack, PagerDuty, and GitHub."

Step 2: List your key pages

Think about the 5-15 pages that are most important for understanding your site. Group them:

## Docs

- [API Reference](https://yoursite.com/docs/api): REST API with auth, rate limits, webhooks
- [SDKs](https://yoursite.com/docs/sdks): Python, JavaScript, Go, Ruby client libraries
- [Self-Hosting](https://yoursite.com/docs/deploy): Docker Compose and Kubernetes guides

## Product

- [Features](https://yoursite.com/features): Error grouping, stack traces, source maps
- [Pricing](https://yoursite.com/pricing): Free tier (5k events/mo), Pro $29/mo, Enterprise

## Resources

- [Blog](https://yoursite.com/blog): Engineering posts, release notes, case studies
- [Changelog](https://yoursite.com/changelog): Version history and migration guides

Step 3: Save and deploy

Save the file as llms.txt at your site root, so it's accessible at https://yoursite.com/llms.txt. No special MIME type needed — plain text is fine. Most web servers serve .txt files correctly by default.

Verify it works:

curl https://yoursite.com/llms.txt

If you see your content, you're done. If you get a 404, check your web server config. For static sites, just drop the file in your public directory. For frameworks like Next.js, put it in the public/ folder. For WordPress, you might need a plugin or a rewrite rule.

Step 4: Reference it in robots.txt (optional but recommended)

Add a reference to your llms.txt in your robots.txt, similar to how you reference your sitemap:

# robots.txt
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

# AI Discovery
# llms.txt: https://yoursite.com/llms.txt

This isn't part of the official spec, but it helps AI crawlers find your llms.txt even if they don't check for it by default.

llms.txt vs. llms-full.txt

The spec defines two files, and they serve different purposes.

llms.txt is the index. It lists your key pages with short descriptions and links. It's compact — usually under 100 lines. Think table of contents.

llms-full.txt is the full content. It includes the actual text of your key pages in a single file, so an AI can read everything without following links. It can be thousands of lines long. Think complete reference manual.

When you need llms-full.txt

For most websites, llms.txt alone is enough. The AI can follow the links to read individual pages when it needs more detail.

llms-full.txt becomes valuable when:

• You have extensive documentation — developer tools, APIs, SDKs. Giving the AI everything in one file means it can answer questions without making multiple requests.
• Your pages require JavaScript to render — if your content is behind a React or Next.js app that renders client-side, the AI might not be able to read the page content by visiting the URL. llms-full.txt gives it the raw text.
• You want maximum accuracy — the more context an AI has in one place, the less likely it is to misunderstand or misrepresent your content.

Stripe, for example, publishes both. Their llms.txt lists API endpoints and guides. Their llms-full.txt contains the actual documentation text, so an AI can answer "how do I create a Stripe checkout session?" without crawling docs.stripe.com.

Who Already Has llms.txt

Adoption is strongest among developer-facing companies, but it's spreading. Here are some notable examples as of early 2026:

• Stripe (docs.stripe.com/llms.txt) — Full API documentation with both llms.txt and llms-full.txt
• Vercel (vercel.com/llms.txt) — Platform docs, SDK references, deployment guides
• Cloudflare (developers.cloudflare.com/llms.txt) — Developer documentation and API references
• Anthropic (docs.anthropic.com/llms.txt) — Claude API documentation
• Supabase — Database, auth, and edge function docs
• Mintlify — Their docs platform generates llms.txt automatically for all hosted docs

The pattern is clear: companies whose products get discussed in AI conversations are adopting this first. If developers ask ChatGPT "how do I do X with Stripe?" and Stripe has an llms.txt, the answer is more likely to be accurate and link to the right docs page.

But it's not just developer tools. Any business that gets mentioned in AI-generated answers benefits from controlling how AI understands their site. E-commerce, SaaS, media, education — if people ask AI about you, llms.txt helps you show up correctly.

Common Mistakes When Creating llms.txt

After looking at dozens of llms.txt files across different sites, these mistakes show up repeatedly:

Being too vague in the description. "We build tools for developers" tells an AI nothing. What kind of tools? For what language? What problem do they solve? Be specific enough that the AI could accurately recommend your product to someone who needs it.

Listing every page. llms.txt is not a sitemap. You don't need to list every blog post or every help article. List the 10-20 pages that matter most. The AI can discover the rest through normal crawling.

Forgetting to update it. If you launch a new product, add a major feature, or deprecate an old one, update your llms.txt. An outdated file is worse than no file, because the AI will confidently cite wrong information.

Making it too long. The base llms.txt should be concise. If you need more than ~100 lines, the extra content probably belongs in llms-full.txt instead.

Using it as a marketing document. Don't stuff your llms.txt with superlatives and sales pitches. AI systems will parrot whatever you write. If you say "the world's fastest deployment platform" and you're not, the AI will repeat that claim and it'll look ridiculous. Stick to facts.

How llms.txt Fits Into the Bigger AI Readiness Picture

llms.txt is one piece of a larger puzzle. To fully prepare your website for AI systems, you need to think about five areas:

1. robots.txt — Controls which AI crawlers can access your site. Without this, nothing else matters.
2. llms.txt — Tells AI what your site is about and where to find the important stuff.
3. Structured data — Schema.org markup (JSON-LD) that helps AI understand the type of content on each page.
4. Content citability — Whether your content is written in clear, quotable statements that AI can extract and cite.
5. AI meta directives — Page-level controls like noai and bot-specific meta tags.

Getting all five right means AI systems can find your site, understand it, quote it accurately, and respect your preferences about how your content gets used. Miss any one of them and there's a gap in the chain.

We wrote a complete guide to AI readiness that covers all five areas in detail.

Should You Add llms.txt to Your Site?

Short answer: yes, if any of these apply to you:

• You have a product or service that people might ask AI about
• You publish documentation, tutorials, or reference content
• You run an API or developer platform
• You care about how your brand appears in AI-generated answers
• You want to be an early adopter before your competitors catch on

The time investment is minimal. Twenty minutes to create the file, another ten to verify it's working. The upside is that every AI system that visits your site gets a better understanding of what you do, which translates to more accurate mentions, better citations, and a stronger presence in the AI-driven discovery layer that's replacing traditional search for a growing number of queries.

The sites doing this now have a head start. Most websites still don't have an llms.txt file. That's exactly why adding one today gives you an outsized advantage — you're competing against a field where almost nobody has shown up yet.