--- title: "robots.txt Tester" description: "Test paths and URLs against a robots.txt for Googlebot, GPTBot or any crawler. See the rule, line and group behind each verdict. Runs in your browser." url: https://findutils.com/seo/robots-txt-tester/ category: seo --- # robots.txt Tester Test paths and URLs against a robots.txt for Googlebot, GPTBot or any crawler. See the rule, line and group behind each verdict. Runs in your browser. **Use this tool:** [robots.txt Tester](https://findutils.com/seo/robots-txt-tester/) ## Programmatic access - REST id `robots-txt-tester`: POST https://api.findutils.com/api/tools/robots-txt-tester/execute (reference: https://findutils.com/api/robots-txt-tester/) - MCP tool `robots_txt_tester` on https://mcp.findutils.com (reference: https://findutils.com/mcp/robots-txt-tester/) ## Why Use a robots.txt Tester? A robots.txt is short, but the way crawlers read it is not obvious. A crawler follows only the group that names it and ignores the * group entirely, so a Disallow you wrote for everyone does nothing for Googlebot once Googlebot has a group of its own. Inside a group the longest matching rule decides, not the first one, and a single missing colon or misspelled directive makes a crawler skip the line without telling you. This tester applies those rules to your file and shows, for every path, the verdict, the rule and line that decided it, and the group the crawler used. It is most useful before you publish a change: paste the new file, pick Googlebot, GPTBot or any other token, and check the URLs you care about. ## Frequently Asked Questions ### How does robots.txt decide whether a URL is blocked? The crawler first picks the group that names it. Inside that group every Allow and Disallow pattern that matches the path is a candidate, and the longest pattern wins. If an Allow and a Disallow of the same length both match, Allow wins. If nothing matches, the path is allowed. This is the rule in RFC 9309 and the one Google uses. The order of the lines does not matter. ### Which group does a crawler follow? The group whose User-agent line matches the crawler's product token, compared without letter case. If several groups name it, their rules are combined. The * group is used only when no group names the crawler, so a crawler with its own group ignores every rule in the * group. ### How do the * and $ wildcards work? * matches any sequence of characters, including none, so Disallow: /*?session= blocks any path that contains ?session=. $ at the end of a pattern anchors it to the end of the path, so Disallow: /*.pdf$ blocks /files/report.pdf but not /files/report.pdf?v=2. Without $, a pattern matches any path that starts with it. ### What do tokens such as GPTBot, ClaudeBot and Google-Extended control? GPTBot and ClaudeBot are crawlers run by OpenAI and Anthropic, and they read the group that names them. Google-Extended and Applebot-Extended are different: no separate crawler fetches pages under those names. Google and Apple read them from your robots.txt to decide how content crawled by their other crawlers may be used for their AI models. Vendors also run separate agents for search features and for fetches a user asks for, each with its own token, so check each vendor's crawler documentation for what a token covers. ### Does Google support Crawl-delay? No. Google ignores the Crawl-delay line. Some other crawlers, such as Bingbot, read it. The tester shows the Crawl-delay set in the group the chosen crawler follows, and whether a crawler honours it depends on that crawler. ### Can I use Noindex in robots.txt? No. Google stopped supporting Noindex in robots.txt in 2019, and the tester flags such a line. To keep a page out of search results, allow crawling and add a noindex meta tag or an X-Robots-Tag HTTP header. ### Can a page blocked by robots.txt still appear in search results? Yes. robots.txt controls crawling, not indexing. If other pages link to a blocked URL, a search engine can still index the URL without its content. Because the crawler cannot fetch the page, it also cannot see a noindex tag on it. ### Does the tester fetch my live robots.txt or crawl my site? No. It tests only the text you paste against the paths you enter. Copy your live file from yoursite.com/robots.txt to test it. robots.txt is also a request to crawlers, not access control: it does not stop anyone from opening a URL. ### Is anything I paste uploaded? Runs in your browser; nothing you paste is sent to FindUtils. Analytics and ads may load on the page. ### Can I run this test from code or an AI assistant? Yes. Send a POST request to https://api.findutils.com/api/tools/robots-txt-tester/execute with a JSON body containing robots (the file text), paths (an array of up to 500 paths or URLs) and an optional user_agent. The same test is available to AI assistants through the FindUtils MCP server as the robots_txt_tester tool. ## Related Tools - [Robots.txt Generator](https://findutils.com/seo/robots-txt-generator/) - [llms.txt Generator](https://findutils.com/seo/llms-txt-generator/) - [XML Sitemap Validator](https://findutils.com/seo/sitemap-validator/) - [XML Sitemap Generator](https://findutils.com/seo/xml-sitemap-generator/) - [GEO Analyzer](https://findutils.com/seo/geo-analyzer/) - [Bot Crawl Checker](https://findutils.com/network/bot-crawl-checker/)