robots.txt Tester

Tests the robots.txt you paste; it does not fetch your live site. robots.txt is a request to crawlers, not access control.

Runs in your browser · nothing uploaded

robots.txt

Start with your live file

Open yoursite.com/robots.txt, copy everything and paste it here. Or load the sample to see how groups, Allow/Disallow precedence and wildcards are resolved.

User agent

Pick one crawler

Paste a robots.txt to see which group Googlebot uses

Paths to test0 paths

Longest matching rule wins. On a tie, Allow wins (RFC 9309).

No results yet

Each tested path gets a verdict with the rule, line number and group that decided it, plus a summary of groups, sitemaps and crawl-delay.

Why Use a robots.txt Tester?

A robots.txt is short, but the way crawlers read it is not obvious. A crawler follows only the group that names it and ignores the * group entirely, so a Disallow you wrote for everyone does nothing for Googlebot once Googlebot has a group of its own. Inside a group the longest matching rule decides, not the first one, and a single missing colon or misspelled directive makes a crawler skip the line without telling you. This tester applies those rules to your file and shows, for every path, the verdict, the rule and line that decided it, and the group the crawler used. It is most useful before you publish a change: paste the new file, pick Googlebot, GPTBot or any other token, and check the URLs you care about.

robots.txt Tester checks paths and URLs against a robots.txt file for the crawler you choose. It follows RFC 9309, the Robots Exclusion Protocol, the way Google applies it. The file is split into groups, each starting with one or more User-agent lines. The crawler uses the group whose User-agent matches its product token, combining every group that names it, and falls back to the * group only when no group names it. Inside that group, every Allow and Disallow pattern is compared with the path. The longest matching pattern decides the verdict, and when an Allow and a Disallow of the same length both match, Allow wins. A path that no rule matches is allowed.

Patterns support two wildcards. * matches any sequence of characters, including none, and $ at the end of a pattern means the path must end there. Paths are compared after percent-encoding is normalised, so /caf%C3%A9 and /café are treated alike. The file /robots.txt itself is always allowed. A full URL is reduced to its path and query string before testing, and a #fragment is ignored.

Every verdict shows the rule that decided it, the line it is on and the group the crawler followed, and the table notes when a longer rule overrode a shorter one. Lines that crawlers skip, such as unknown directives, lines without a colon, Noindex lines and Sitemap lines with a relative URL, are listed with their line numbers and highlighted in the file. The summary counts the groups, the sitemaps and the Crawl-delay that applies to the chosen crawler.

The tester reads only what you paste. It does not fetch your live robots.txt and it does not crawl your site. To write a new file, use the robots.txt Generator. For AI assistants you can also publish an llms.txt with the llms.txt Generator, and you can check the sitemap your robots.txt points to with the Sitemap Validator.

How it compares

Google Search Console has a robots.txt report. It shows the robots.txt files Google found for your verified site, when Google last fetched them and any problems it had parsing them. That report is the authority on what Google actually fetched from your live site, and it only covers Google. This tester works on any text you paste, before it is published, and tests it as any crawler token, including Bingbot, GPTBot and ClaudeBot. It shows the rule, line and group behind each verdict, so you can see why a URL is blocked. Use this tester to check a change before you deploy it, and use Search Console to confirm what Google fetched afterwards.

Tips for Testing robots.txt

Test the agent you care about by name. A crawler with its own group ignores the * group, so testing only as * can hide a problem.
Put all rules for one crawler in one group. Several groups naming the same crawler are combined, which is easy to miss when you read the file top to bottom.
Use $ to anchor the end of a path: Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?download=1.
Remember that Disallow: with an empty value allows everything, and Disallow: / blocks everything.
To keep a page out of search results, allow crawling and use a noindex meta tag or X-Robots-Tag header. A crawler that cannot fetch the page never sees the noindex.

Frequently Asked Questions

How does robots.txt decide whether a URL is blocked?

The crawler first picks the group that names it. Inside that group every Allow and Disallow pattern that matches the path is a candidate, and the longest pattern wins. If an Allow and a Disallow of the same length both match, Allow wins. If nothing matches, the path is allowed. This is the rule in RFC 9309 and the one Google uses. The order of the lines does not matter.

Which group does a crawler follow?

The group whose User-agent line matches the crawler's product token, compared without letter case. If several groups name it, their rules are combined. The * group is used only when no group names the crawler, so a crawler with its own group ignores every rule in the * group.

How do the * and $ wildcards work?

* matches any sequence of characters, including none, so Disallow: /*?session= blocks any path that contains ?session=. $ at the end of a pattern anchors it to the end of the path, so Disallow: /*.pdf$ blocks /files/report.pdf but not /files/report.pdf?v=2. Without $, a pattern matches any path that starts with it.

What do tokens such as GPTBot, ClaudeBot and Google-Extended control?

GPTBot and ClaudeBot are crawlers run by OpenAI and Anthropic, and they read the group that names them. Google-Extended and Applebot-Extended are different: no separate crawler fetches pages under those names. Google and Apple read them from your robots.txt to decide how content crawled by their other crawlers may be used for their AI models. Vendors also run separate agents for search features and for fetches a user asks for, each with its own token, so check each vendor's crawler documentation for what a token covers.

Does Google support Crawl-delay?

No. Google ignores the Crawl-delay line. Some other crawlers, such as Bingbot, read it. The tester shows the Crawl-delay set in the group the chosen crawler follows, and whether a crawler honours it depends on that crawler.

Can I use Noindex in robots.txt?

No. Google stopped supporting Noindex in robots.txt in 2019, and the tester flags such a line. To keep a page out of search results, allow crawling and add a noindex meta tag or an X-Robots-Tag HTTP header.

Can a page blocked by robots.txt still appear in search results?

Yes. robots.txt controls crawling, not indexing. If other pages link to a blocked URL, a search engine can still index the URL without its content. Because the crawler cannot fetch the page, it also cannot see a noindex tag on it.

Does the tester fetch my live robots.txt or crawl my site?

No. It tests only the text you paste against the paths you enter. Copy your live file from yoursite.com/robots.txt to test it. robots.txt is also a request to crawlers, not access control: it does not stop anyone from opening a URL.

Is anything I paste uploaded?

Runs in your browser; nothing you paste is sent to FindUtils. Analytics and ads may load on the page.

Can I run this test from code or an AI assistant?

Yes. Send a POST request to https://api.findutils.com/api/tools/robots-txt-tester/execute with a JSON body containing robots (the file text), paths (an array of up to 500 paths or URLs) and an optional user_agent. The same test is available to AI assistants through the FindUtils MCP server as the robots_txt_tester tool.

Run it from code

$ curl -X POST \
  api.findutils.com/api/tools/robots-txt-tester/execute \
  -d '{"robots":"User-agent: *\nDisallow: /admin","paths":["/admin","/blog/"]}'

$ claude mcp add findutils \
  --transport http https://mcp.findutils.com/
# then call findutils:robots_txt_tester

API reference Add to Claude