# Bot Crawl Checker — MCP tool `findutils:bot_crawl_checker`

Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or blocked for a URL by the site's robots.txt, with the matching rule and user-agent group per bot.

- Category: seo
- MCP server: https://mcp.findutils.com/ (Streamable HTTP, no API keys, 120 req/min per IP)
- REST endpoint: POST https://api.findutils.com/api/tools/bot-crawl-checker/execute (no API keys, 60 req/min per IP)
- Network tool: fetches a fixed, hard-coded public upstream (never a private host).
- Reference page: https://findutils.com/mcp/bot-crawl-checker/
- Same tool on the other surface: https://findutils.com/api/bot-crawl-checker/

## Connect

```bash
claude mcp add findutils --transport http https://mcp.findutils.com/
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "findutils": {
      "url": "https://mcp.findutils.com/"
    }
  }
}
```

## Call the tool (verified example)

```bash
curl -X POST https://mcp.findutils.com/ \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "bot_crawl_checker",
      "arguments": {
        "url": "https://findutils.com/"
      }
    }
  }'
```

## Input schema

| Argument | Type | Required | Description |
|---|---|---|---|
| `url` | string | yes | Public http(s) URL to check, e.g. "https://example.com/blog/post". |

Example arguments (verified):

```json
{
  "url": "https://findutils.com/"
}
```

## Also a REST endpoint

```bash
curl -X POST https://api.findutils.com/api/tools/bot-crawl-checker/execute \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://findutils.com/"
  }'

# Parameter schema
curl https://api.findutils.com/api/tools/bot-crawl-checker
```

Full REST reference: https://findutils.com/api/bot-crawl-checker/ · OpenAPI 3.1 spec: https://findutils.com/api/openapi.json

---
Full catalog: POST https://mcp.findutils.com/ with method `tools/list` · https://findutils.com/mcp/ · https://findutils.com/llms.txt