# Robots Txt Tester — MCP tool `findutils:robots_txt_tester`

Test URLs or paths against robots.txt text for a given crawler, following RFC 9309 as Google implements it: the group naming the crawler's product token (all such groups merged, "*" only as a fallback), * and $ wildcards, the longest matching rule wins and Allow wins a tie. Returns allowed/blocked per path with the deciding rule and its line, plus sitemaps, crawl-delay and line-numbered warnings. Checks the pasted text only and fetches nothing.

- Category: seo
- MCP server: https://mcp.findutils.com/ (Streamable HTTP, no API keys, 120 req/min per IP)
- REST endpoint: POST https://api.findutils.com/api/tools/robots-txt-tester/execute (no API keys, 60 req/min per IP)
- Reference page: https://findutils.com/mcp/robots-txt-tester/
- Same tool on the other surface: https://findutils.com/api/robots-txt-tester/

## Connect

```bash
claude mcp add findutils --transport http https://mcp.findutils.com/
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "findutils": {
      "url": "https://mcp.findutils.com/"
    }
  }
}
```

## Call the tool (verified example)

```bash
curl -X POST https://mcp.findutils.com/ \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "robots_txt_tester",
      "arguments": {
        "robots": "User-agent: *\nDisallow: /admin",
        "paths": [
          "/admin",
          "/blog/"
        ]
      }
    }
  }'
```

## Input schema

| Argument | Type | Required | Description |
|---|---|---|---|
| `robots` | string | yes | The full robots.txt text. |
| `paths` | array | yes | Paths ("/admin?x=1") or full URLs to test, at most 500. For a URL only the path and query are tested. |
| `user_agent` | string | no | Crawler product token, e.g. "Googlebot", "GPTBot", "Bingbot". Default "*". Default: `"*"`. |

Example arguments (verified):

```json
{
  "robots": "User-agent: *\nDisallow: /admin",
  "paths": [
    "/admin",
    "/blog/"
  ]
}
```

## Also a REST endpoint

```bash
curl -X POST https://api.findutils.com/api/tools/robots-txt-tester/execute \
  -H "Content-Type: application/json" \
  -d '{
    "robots": "User-agent: *\nDisallow: /admin",
    "paths": [
      "/admin",
      "/blog/"
    ]
  }'

# Parameter schema
curl https://api.findutils.com/api/tools/robots-txt-tester
```

Full REST reference: https://findutils.com/api/robots-txt-tester/ · OpenAPI 3.1 spec: https://findutils.com/api/openapi.json

---
Full catalog: POST https://mcp.findutils.com/ with method `tools/list` · https://findutils.com/mcp/ · https://findutils.com/llms.txt