Bot Crawl Checker MCP tool

Network tool

MCP findutils:bot_crawl_checker

Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or blocked for a URL by the site's robots.txt, with the matching rule and user-agent group per bot.

Needs an API key. Running a tool needs an API key. Keys are not for sale yet: ask for early access. About API keys

Open the tool Try it

Arguments

application/json

url

string required

Public http(s) URL to check, e.g. "https://example.com/blog/post".

Example arguments

Verified

{
  "url": "https://findutils.com/"
}

claude mcp add findutils --transport http https://mcp.findutils.com/ --header "Authorization: Bearer YOUR_API_KEY"

{
  "mcpServers": {
    "findutils": {
      "url": "https://mcp.findutils.com/",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

curl -X POST https://mcp.findutils.com/ \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "bot_crawl_checker",
      "arguments": {
        "url": "https://findutils.com/"
      }
    }
  }'

Also a REST endpoint

The same tool answers plain HTTP at POST /api/tools/bot-crawl-checker/execute.

REST reference

Rate limit: 120 a minute, per IP
Auth: API key
REST id: bot-crawl-checker
MCP name: bot_crawl_checker
Category: Network
Machine-readable: openapi.json · markdown.md

This tool fetches a fixed, hard-coded public upstream, never a private host. Results depend on that upstream.

Arguments

Example arguments

More Network tools

More to explore