---
url: https://findutils.com/guides/robots-txt-tester
title: "Test robots.txt Rules: Is a URL Blocked for Googlebot or GPTBot?"
description: "Check whether a path is allowed or blocked by your robots.txt for any crawler, with the exact rule and line that decides it. RFC 9309 matching, in your browser."
category: seo
content_type: guide
guide_type: subtopic
cluster: developer
locale: en
read_time: 7
status: published
author: "olgunozoktas"
published_at: 2026-09-26T12:00:00Z
excerpt: "A robots.txt edit that blocks /shop also blocks /shopping, and a GPTBot group written above the catch-all group quietly overrides it. Testing paths against the file, crawler by crawler, shows which rule wins before search engines and AI crawlers find out."
tag_ids: ["robots-txt", "seo", "crawlers", "ai-crawlers"]
tags: ["robots.txt", "SEO", "Crawlers", "AI Crawlers"]
primary_keyword: "robots.txt tester"
secondary_keywords: ["test robots.txt", "is url blocked by robots.txt", "check robots.txt for googlebot", "robots.txt gptbot test", "robots.txt allow disallow precedence"]
tool_tag: "robots-txt-tester"
related_tool: "robots-txt-tester"
related_tools: ["robots-txt-tester", "robots-txt-generator", "sitemap-validator", "bot-crawl-checker", "llms-txt-generator"]
og_image: "/images/content/guides/robots-txt-tester-cover-20260926.webp"
image_alt: "A hedge maze seen from above with small rover robots at its entrances; some paths glow green and others are closed by gates with a red glow."
updated_at: "2026-09-26T12:00:00Z"
---

To test a robots.txt file, pick the crawler, find the group of rules written for it, and check each path against that group: the longest matching rule decides, and Allow wins a tie. FindUtils [robots.txt Tester](/seo/robots-txt-tester/) does this for any user agent, including Googlebot, Bingbot, GPTBot and ClaudeBot. For every path it says allowed or blocked and names the deciding rule and its line number, following [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309). It tests the text you paste and fetches nothing.

This guide covers how a crawler picks its group, how the winning rule is chosen, wildcards, and what a robots.txt test can and cannot tell you.

## Why Test Before You Publish?

A robots.txt mistake is silent. Nothing breaks on the site; pages just stop being crawled, or keep being crawled when you meant to stop it. Typical causes:

- **Prefix matching.** `Disallow: /shop` also blocks `/shop-sale` and `/shopping`, because a rule matches the start of the path.
- **Group override.** A crawler with its own group ignores the `User-agent: *` group entirely, so rules you wrote for everyone no longer apply to it.
- **Conflicting rules.** An `Allow` and a `Disallow` both match, and it is not obvious which wins.
- **Unsupported lines.** `Noindex:` or `Crawl-delay:` in the file does not do what some people expect in Google.

## How to Test a robots.txt File

### Step 1: Paste the robots.txt

Open [robots.txt Tester](/seo/robots-txt-tester/) and paste the whole file, whether it is your live file copied from `https://yoursite.com/robots.txt` or a draft you have not published yet.

### Step 2: Choose the crawler

Enter or pick a user agent, such as `Googlebot`, `Bingbot`, `GPTBot` or `ClaudeBot`. Leave it as `*` to test the catch-all group.

### Step 3: Add the paths

Add the paths or full URLs to check. For a full URL only the path and query string are tested; a `#fragment` is ignored.

### Step 4: Read the verdicts

Each path shows allowed or blocked, with the rule that decided it and its line. The report also lists the matched group, the Sitemap lines, the crawl delay for that group, and warnings with line numbers.

## A Worked Example

```text
User-agent: *
Disallow: /shop
Allow: /shop/public

User-agent: GPTBot
Disallow: /
```

| Crawler | Path | Result | Deciding rule |
|---|---|---|---|
| `*` | `/shop/public/a` | Allowed | `Allow: /shop/public` (line 3) |
| `*` | `/shop/cart` | Blocked | `Disallow: /shop` (line 2) |
| `*` | `/blog` | Allowed | No rule matches |
| `GPTBot` | `/blog` | Blocked | `Disallow: /` (line 6) |
| `ClaudeBot` | `/shop/cart` | Blocked | `Disallow: /shop` (line 2): no ClaudeBot group, so `*` applies |

GPTBot never sees the `*` rules, because it has a group of its own.

## How Does a Crawler Pick Its Group?

RFC 9309 says crawlers match their product token case-insensitively and obey that group; if more than one group names them, the groups are combined, and only when no group matches do they fall back to `User-agent: *`. The tester follows the same order:

1. **Exact token.** `googlebot` and `Googlebot` groups both apply to Googlebot, merged into one.
2. **A shorter token the crawler extends with a hyphen.** With no `Googlebot-Image` group, Googlebot-Image follows the `Googlebot` group.
3. **`*`**, only when nothing names the crawler.

The product token is the leading letters, `_` and `-` of the user agent, so `Googlebot/2.1` is tested as `Googlebot`.

## Which Rule Wins?

RFC 9309 states that "the most specific match found MUST be used", meaning the rule with the most octets, and that when an allow and a disallow rule are equivalent, "the 'allow' rule SHOULD be used".

| Rules in the group | Path | Result | Why |
|---|---|---|---|
| `Disallow: /shop`, `Allow: /shop/public` | `/shop/public/a` | Allowed | The Allow pattern is longer |
| `Allow: /page`, `Disallow: /page` | `/page` | Allowed | Same length: Allow wins the tie |
| `Disallow:` (empty) | `/anything` | Allowed | An empty Disallow matches nothing |
| `Disallow: /` | `/robots.txt` | Allowed | The RFC says /robots.txt is implicitly allowed |

Order inside the file does not matter; only length and type do.

## Wildcards, the $ Anchor and Encoding

`*` matches any run of characters and `$` anchors the end of the path:

| Rule | Path | Matches? |
|---|---|---|
| `Disallow: /*.pdf$` | `/docs/a.pdf` | Yes: blocked |
| `Disallow: /*.pdf$` | `/docs/a.pdf?x=1` | No: the path does not end in `.pdf` |
| `Disallow: /*?sort=` | `/list?sort=asc` | Yes: blocked |

Paths and rules are compared in one canonical form: escapes of unreserved characters are decoded (`%7e` becomes `~`), other escapes get uppercase hex, and non-ASCII characters are percent-encoded as UTF-8. So `Disallow: /caf%c3%a9` blocks `https://example.com/café`.

## Warnings the Tester Reports

- **A rule before any User-agent line**, which crawlers ignore.
- **Unknown directives**, such as a typo like `Dissalow`.
- **A path without a leading `/`**, which is tested as if it had one.
- **A Sitemap that is not an absolute URL**, and a Crawl-delay that is not a number.
- **Noindex.** Google states that a `noindex` rule [in robots.txt is not supported](https://developers.google.com/search/docs/crawling-indexing/block-indexing); use a meta robots tag or an `X-Robots-Tag` header instead.
- **A file over 500 KiB.** Google [ignores content after 500 KiB](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), and the tester stops at the same point.

The crawl delay is shown for the matched group, but Google lists `crawl-delay` among the fields it does not support. Other crawlers may honour it.

## What a robots.txt Test Cannot Tell You

- **It does not fetch your live file.** Paste the current version; a test of an old copy answers an old question.
- **robots.txt is advisory.** RFC 9309 says the rules "are not a form of access authorization". Well-behaved crawlers follow them; nothing forces the others to.
- **Blocked is not the same as not indexed.** Google states that a page disallowed in robots.txt [can still be indexed if linked to from other sites](https://developers.google.com/search/docs/crawling-indexing/robots/intro), and a blocked page's `noindex` tag is never seen.
- **It does not check the live server.** A 403 from the server, a login wall or a redirect loop is invisible here. For a reachability check of a live URL, use [Bot Crawl Checker](/network/bot-crawl-checker/).

## Common Mistakes

**Writing rules for everyone after naming a crawler.** Once `GPTBot` has a group, rules in `*` do not reach it. Repeat the rules in its group.

**Forgetting the trailing slash.** `Disallow: /admin` also blocks `/administrator`. Use `Disallow: /admin/` to block only the folder.

**Testing only one crawler.** Test at least `*`, `Googlebot` and each AI crawler you name in the file.

## Use It From Code

The tester is the `robots_txt_tester` tool on the FindUtils REST API and MCP server.

```bash
curl -X POST https://api.findutils.com/api/tools/robots-txt-tester/execute \
  -H "Content-Type: application/json" \
  -d '{"robots":"User-agent: *\nDisallow: /shop\nAllow: /shop/public\n\nUser-agent: GPTBot\nDisallow: /","paths":["/shop/cart","/shop/public/a","https://example.com/blog"],"user_agent":"GPTBot"}'
```

Up to 500 paths per call. The response lists `results` with `allowed`, `rule` and `line` for each path, plus `matched_group`, `sitemaps`, `crawl_delay` and `warnings`. See the [API reference](/api/robots-txt-tester/) and the [MCP reference](/mcp/robots-txt-tester/).

## Tools Used in This Guide

| Tool | Use |
|---|---|
| [robots.txt Tester](/seo/robots-txt-tester/) | Test paths against robots.txt for any crawler |
| [Robots.txt Generator](/seo/robots-txt-generator/) | Build a robots.txt from a form |
| [XML Sitemap Validator](/seo/sitemap-validator/) | Check the sitemap your Sitemap line points to |
| [Bot Crawl Checker](/network/bot-crawl-checker/) | Check whether a live URL is reachable for well-known bots |
| [llms.txt Generator](/seo/llms-txt-generator/) | Write an llms.txt and an AI crawler snippet for robots.txt |

## FAQ

### How do I check if a URL is blocked by robots.txt?

Paste the robots.txt into robots.txt Tester, choose the crawler, and add the URL. The result says allowed or blocked and names the rule and line that decided it.

### Does Allow or Disallow win?

The longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins.

### Do the rules for User-agent: * apply to Googlebot?

Only when the file has no group for Googlebot. A crawler with its own group follows that group and ignores `*`.

### Can robots.txt remove a page from Google?

No. It controls crawling, not indexing. To keep a page out of results, let it be crawled and add a `noindex` meta tag or header.

### Does the tester fetch my robots.txt?

No. It tests the text you paste, in your browser, and fetches nothing.

## Next Steps

Test your file with [robots.txt Tester](/seo/robots-txt-tester/). To write a new one, start with the [Robots.txt Generator](/seo/robots-txt-generator/) and its [guide](/guides/robots-txt-generator-guide/).