To create an llms.txt file, write a Markdown file with your site's name as the only H1, a one- or two-sentence summary in a blockquote, and H2 sections that list your key pages as - [title](url): description lines, then serve it at https://yoursite.com/llms.txt. FindUtils llms.txt Generator builds that file from a form, resolves your links against your domain, and warns about duplicates, off-site links and placeholder titles. It also gives you a matching robots.txt snippet. The generator fetches nothing and runs in your browser.
This guide covers what the format is, what it can and cannot do for you, how to fill in the generator, and where to put the result.
What Is llms.txt?
llms.txt is a proposal published by Jeremy Howard at llmstxt.org in September 2024. It describes a Markdown file at the root path /llms.txt of a website (or at a subpath such as /docs/llms.txt) that gives language models a short, curated index of the site. The format, in order:
- An H1 with the name of the site or project. The proposal calls this the only required section.
- A blockquote with a short summary.
- Optional prose: paragraphs or lists, but no headings.
- H2 sections of "file lists": Markdown links, each optionally followed by
:and a note. - An
## Optionalsection, by convention holding secondary links that a tool can skip when its context is short.
The proposal also suggests offering clean Markdown versions of pages, for example at the page URL with .md appended.
What llms.txt Does and Does Not Do
llms.txt is a proposal, not a standard from a standards body, and no search engine or AI provider guarantees that its systems read the file or cite sites that publish one. What it offers is a clean, human-written index for any agent or tool that does look for it.
| File | Tells machines | Enforced? |
|---|---|---|
robots.txt | Which paths a crawler may fetch | Advisory; crawlers choose to obey |
sitemap.xml | Every URL you want discovered, with dates | A hint for search engines |
llms.txt | What the site is and which pages matter, in prose | Nothing reads it by obligation |
That makes llms.txt cheap to add and easy to overrate. Publish it because a clear index of your best pages is useful, not because it promises citations.
How to Build the File
Step 1: Name the site and the origin
Open the llms.txt Generator. Enter the site name, which becomes the H1, and the origin, such as https://docs.example.com. Links you write as /guides/start are resolved against that origin.
Step 2: Write the summary
Write one or two sentences for the blockquote. Say what the site is and, if the name is ambiguous, what it is not. The generator warns above 320 characters.
Step 3: Add sections and links
Add H2 sections such as Guides, API or Pricing, and give each link a title, a URL and one sentence saying why an agent should open it. Put secondary links, such as a changelog, in the Optional section; it is always written last.
Step 4: Fix the warnings and copy the outputs
Read the warnings, then copy or download llms.txt, and copy the robots.txt snippet and the header snippet if you want them.
A Worked Example
Input: site name Example Docs, origin https://docs.example.com, summary Example Docs is the public documentation for Example., a Guides section with Start at /start ("Begin here."), an optional link Changelog at /changelog ("Release history."), and the sitemap URL https://docs.example.com/sitemap.xml.
The generated llms.txt:
# Example Docs > Example Docs is the public documentation for Example. ## Guides - [Start](https://docs.example.com/start): Begin here. ## Optional - [Changelog](https://docs.example.com/changelog): Release history.
The robots.txt snippet:
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / Sitemap: https://docs.example.com/sitemap.xml
The header snippet is Link: </llms.txt>; rel="describedby"; type="text/plain", and the HTML version is <link rel="describedby" href="/llms.txt" type="text/plain">. Both are pointers for tools that look for one; nothing requires an agent to follow them.
What the Generator Checks
| Problem | Result |
|---|---|
| Origin missing, not absolute, or not http(s) | Error |
A link URL that is relative without a leading / (page.html) | Error |
A URL with an angle-bracket placeholder (https://x.com/<page>) | Error |
| A link with no title | Error |
| A link on another host than the origin | Warning |
| The same URL listed twice | Warning |
A title copied from a template, such as Example Corp or Page Title | Warning |
| A link with no description | Warning |
| A summary over 320 characters, or a file over about 100 KB | Warning |
| Headings inside the prose | Escaped to plain text, with a warning, so they cannot start a new section |
| An empty section | Left out, with a warning |
[, ] and \ in titles are escaped, and parentheses in URLs are percent-encoded, so a link cannot break the Markdown. The generator checks the shape of each URL only: it does not fetch the pages, so it cannot tell you a link returns 404.
Decide Which AI Crawlers to Allow
The robots.txt snippet allows five AI user-agent tokens by default. They do different jobs, according to their operators:
- GPTBot (OpenAI) crawls content that may be used in training OpenAI's foundation models. ChatGPT search uses a separate token, OAI-SearchBot.
- ClaudeBot (Anthropic) collects content that could contribute to model training.
- PerplexityBot surfaces and links websites in Perplexity's search results.
- Google-Extended controls whether content may be used for Gemini training and grounding, and Google states it does not affect Google Search.
- Applebot-Extended does not crawl pages; Apple uses it to let publishers opt out of training Apple's foundation models on content Applebot already collected.
Allowing a training crawler is a choice about your content, not a technical requirement for llms.txt. Delete any group you do not want before pasting the snippet into your robots.txt, and check the result with robots.txt Tester.
The optional Content-Signal line (for example search=yes, ai-input=yes, ai-train=no) states your preferences in robots.txt. It is a newer proposal, and nothing obliges a crawler to honour it.
Where to Put the File
Serve the file at the root of the origin, https://yoursite.com/llms.txt, as plain text or Markdown, with a 200 status and no login. In most frameworks that means placing it in the public or static folder. Keep it short: the useful file lists the pages you would want quoted, not every URL on the site. That is what a sitemap, built with the XML Sitemap Generator, is for.
Common Mistakes
Listing everything. An llms.txt with hundreds of links is a sitemap with extra steps. Pick the pages that explain the site.
Leaving template text. Example Corp, Page Title and <page> placeholders ship more often than you would think. The generator flags them.
Links without descriptions. The one sentence after the link is what tells a reader why to open it.
Expecting traffic. Publishing llms.txt does not make any AI system cite you. Measure what you can, such as whether AI crawlers can reach your pages, with the GEO Analyzer.
Use It From Code
The generator is the llms_txt_generate tool on the FindUtils REST API and MCP server.
curl -X POST https://api.findutils.com/api/tools/llms-txt-generate/execute \
-H "Content-Type: application/json" \
-d '{"site_name":"Example Docs","origin":"https://docs.example.com","summary":"Example Docs is the public documentation for Example.","sections":[{"heading":"Guides","links":[{"title":"Start","url":"/start","description":"Begin here."}]}]}'Other arguments: details, optional_links, content_signal and sitemap_url. The response holds llms_txt, warnings, robots_snippet, link_header and html_link. See the API reference and the MCP reference.
Tools Used in This Guide
| Tool | Use |
|---|---|
| llms.txt Generator | Build a valid llms.txt and matching snippets |
| GEO Analyzer | Check a page's readiness for AI search |
| Robots.txt Generator | Build the robots.txt the snippet goes into |
| robots.txt Tester | Check which crawlers your robots.txt allows |
| Bot Crawl Checker | Check whether a live URL is reachable for well-known search, AI and social bots |
FAQ
Is llms.txt required for AI search?
No. It is an optional proposal. No search engine or AI provider guarantees it reads the file, and pages are found and cited without one.
Where does llms.txt go?
At the root of your site, https://yoursite.com/llms.txt. The proposal also allows a subpath, such as /docs/llms.txt, for a section of a site.
What is the difference between llms.txt and robots.txt?
robots.txt tells crawlers which paths they may fetch. llms.txt tells a reader what the site is and which pages matter. One is about access, the other about orientation.
Does the generator check that my links work?
No. It checks that each URL is well formed and on your origin, but fetches nothing, so a broken link is not detected.
Should I allow GPTBot and ClaudeBot?
That is your decision. Both operators describe these crawlers as collecting content that may be used for model training. Remove their groups from the snippet if you do not want that.
Next Steps
Build your file with the llms.txt Generator, then read How to Make Your Website AI-Agent Ready for the other files that sit beside it.
