To test a robots.txt file, pick the crawler, find the group of rules written for it, and check each path against that group: the longest matching rule decides, and Allow wins a tie. FindUtils robots.txt Tester does this for any user agent, including Googlebot, Bingbot, GPTBot and ClaudeBot. For every path it says allowed or blocked and names the deciding rule and its line number, following RFC 9309. It tests the text you paste and fetches nothing.
This guide covers how a crawler picks its group, how the winning rule is chosen, wildcards, and what a robots.txt test can and cannot tell you.
Why Test Before You Publish?
A robots.txt mistake is silent. Nothing breaks on the site; pages just stop being crawled, or keep being crawled when you meant to stop it. Typical causes:
- Prefix matching.
Disallow: /shopalso blocks/shop-saleand/shopping, because a rule matches the start of the path. - Group override. A crawler with its own group ignores the
User-agent: *group entirely, so rules you wrote for everyone no longer apply to it. - Conflicting rules. An
Allowand aDisallowboth match, and it is not obvious which wins. - Unsupported lines.
Noindex:orCrawl-delay:in the file does not do what some people expect in Google.
How to Test a robots.txt File
Step 1: Paste the robots.txt
Open robots.txt Tester and paste the whole file, whether it is your live file copied from https://yoursite.com/robots.txt or a draft you have not published yet.
Step 2: Choose the crawler
Enter or pick a user agent, such as Googlebot, Bingbot, GPTBot or ClaudeBot. Leave it as * to test the catch-all group.
Step 3: Add the paths
Add the paths or full URLs to check. For a full URL only the path and query string are tested; a #fragment is ignored.
Step 4: Read the verdicts
Each path shows allowed or blocked, with the rule that decided it and its line. The report also lists the matched group, the Sitemap lines, the crawl delay for that group, and warnings with line numbers.
A Worked Example
User-agent: * Disallow: /shop Allow: /shop/public User-agent: GPTBot Disallow: /
| Crawler | Path | Result | Deciding rule |
|---|---|---|---|
* | /shop/public/a | Allowed | Allow: /shop/public (line 3) |
* | /shop/cart | Blocked | Disallow: /shop (line 2) |
* | /blog | Allowed | No rule matches |
GPTBot | /blog | Blocked | Disallow: / (line 6) |
ClaudeBot | /shop/cart | Blocked | Disallow: /shop (line 2): no ClaudeBot group, so * applies |
GPTBot never sees the * rules, because it has a group of its own.
How Does a Crawler Pick Its Group?
RFC 9309 says crawlers match their product token case-insensitively and obey that group; if more than one group names them, the groups are combined, and only when no group matches do they fall back to User-agent: *. The tester follows the same order:
- Exact token.
googlebotandGooglebotgroups both apply to Googlebot, merged into one. - A shorter token the crawler extends with a hyphen. With no
Googlebot-Imagegroup, Googlebot-Image follows theGooglebotgroup. *, only when nothing names the crawler.
The product token is the leading letters, _ and - of the user agent, so Googlebot/2.1 is tested as Googlebot.
Which Rule Wins?
RFC 9309 states that "the most specific match found MUST be used", meaning the rule with the most octets, and that when an allow and a disallow rule are equivalent, "the 'allow' rule SHOULD be used".
| Rules in the group | Path | Result | Why |
|---|---|---|---|
Disallow: /shop, Allow: /shop/public | /shop/public/a | Allowed | The Allow pattern is longer |
Allow: /page, Disallow: /page | /page | Allowed | Same length: Allow wins the tie |
Disallow: (empty) | /anything | Allowed | An empty Disallow matches nothing |
Disallow: / | /robots.txt | Allowed | The RFC says /robots.txt is implicitly allowed |
Order inside the file does not matter; only length and type do.
Wildcards, the $ Anchor and Encoding
* matches any run of characters and $ anchors the end of the path:
| Rule | Path | Matches? |
|---|---|---|
Disallow: /*.pdf$ | /docs/a.pdf | Yes: blocked |
Disallow: /*.pdf$ | /docs/a.pdf?x=1 | No: the path does not end in .pdf |
Disallow: /*?sort= | /list?sort=asc | Yes: blocked |
Paths and rules are compared in one canonical form: escapes of unreserved characters are decoded (%7e becomes ~), other escapes get uppercase hex, and non-ASCII characters are percent-encoded as UTF-8. So Disallow: /caf%c3%a9 blocks https://example.com/café.
Warnings the Tester Reports
- A rule before any User-agent line, which crawlers ignore.
- Unknown directives, such as a typo like
Dissalow. - A path without a leading
/, which is tested as if it had one. - A Sitemap that is not an absolute URL, and a Crawl-delay that is not a number.
- Noindex. Google states that a
noindexrule in robots.txt is not supported; use a meta robots tag or anX-Robots-Tagheader instead. - A file over 500 KiB. Google ignores content after 500 KiB, and the tester stops at the same point.
The crawl delay is shown for the matched group, but Google lists crawl-delay among the fields it does not support. Other crawlers may honour it.
What a robots.txt Test Cannot Tell You
- It does not fetch your live file. Paste the current version; a test of an old copy answers an old question.
- robots.txt is advisory. RFC 9309 says the rules "are not a form of access authorization". Well-behaved crawlers follow them; nothing forces the others to.
- Blocked is not the same as not indexed. Google states that a page disallowed in robots.txt can still be indexed if linked to from other sites, and a blocked page's
noindextag is never seen. - It does not check the live server. A 403 from the server, a login wall or a redirect loop is invisible here. For a reachability check of a live URL, use Bot Crawl Checker.
Common Mistakes
Writing rules for everyone after naming a crawler. Once GPTBot has a group, rules in * do not reach it. Repeat the rules in its group.
Forgetting the trailing slash. Disallow: /admin also blocks /administrator. Use Disallow: /admin/ to block only the folder.
Testing only one crawler. Test at least *, Googlebot and each AI crawler you name in the file.
Use It From Code
The tester is the robots_txt_tester tool on the FindUtils REST API and MCP server.
curl -X POST https://api.findutils.com/api/tools/robots-txt-tester/execute \
-H "Content-Type: application/json" \
-d '{"robots":"User-agent: *\nDisallow: /shop\nAllow: /shop/public\n\nUser-agent: GPTBot\nDisallow: /","paths":["/shop/cart","/shop/public/a","https://example.com/blog"],"user_agent":"GPTBot"}'Up to 500 paths per call. The response lists results with allowed, rule and line for each path, plus matched_group, sitemaps, crawl_delay and warnings. See the API reference and the MCP reference.
Tools Used in This Guide
| Tool | Use |
|---|---|
| robots.txt Tester | Test paths against robots.txt for any crawler |
| Robots.txt Generator | Build a robots.txt from a form |
| XML Sitemap Validator | Check the sitemap your Sitemap line points to |
| Bot Crawl Checker | Check whether a live URL is reachable for well-known bots |
| llms.txt Generator | Write an llms.txt and an AI crawler snippet for robots.txt |
FAQ
How do I check if a URL is blocked by robots.txt?
Paste the robots.txt into robots.txt Tester, choose the crawler, and add the URL. The result says allowed or blocked and names the rule and line that decided it.
Does Allow or Disallow win?
The longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins.
Do the rules for User-agent: * apply to Googlebot?
Only when the file has no group for Googlebot. A crawler with its own group follows that group and ignores *.
Can robots.txt remove a page from Google?
No. It controls crawling, not indexing. To keep a page out of results, let it be crawled and add a noindex meta tag or header.
Does the tester fetch my robots.txt?
No. It tests the text you paste, in your browser, and fetches nothing.
Next Steps
Test your file with robots.txt Tester. To write a new one, start with the Robots.txt Generator and its guide.
