Bot Crawl Checker API

Network tool

POST https://api.findutils.com/api/tools/bot-crawl-checker/execute

Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or blocked for a URL by the site's robots.txt, with the matching rule and user-agent group per bot.

Needs an API key. Running a tool needs an API key. Keys are not for sale yet: ask for early access. About API keys

Open the tool Try it

Request body

application/json

url

string required

Public http(s) URL to check, e.g. "https://example.com/blog/post".

Example arguments

Verified

{
  "url": "https://findutils.com/"
}

curl -X POST https://api.findutils.com/api/tools/bot-crawl-checker/execute \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://findutils.com/"
  }'

const response = await fetch('https://api.findutils.com/api/tools/bot-crawl-checker/execute', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', Authorization: 'Bearer YOUR_API_KEY' },
  body: JSON.stringify({"url":"https://findutils.com/"}),
});
const data = await response.json();

import requests

response = requests.post(
    "https://api.findutils.com/api/tools/bot-crawl-checker/execute",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url":"https://findutils.com/"},
)
data = response.json()

$ch = curl_init('https://api.findutils.com/api/tools/bot-crawl-checker/execute');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'Authorization: Bearer YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => '{"url":"https://findutils.com/"}',
    CURLOPT_RETURNTRANSFER => true,
]);
$data = json_decode(curl_exec($ch), true);

Also an MCP tool

Call bot_crawl_checker from Claude, Cursor or any MCP client. Same arguments, same output.

https://mcp.findutils.com

MCP reference

Rate limit: 60 a minute, per IP
Auth: API key
REST id: bot-crawl-checker
MCP name: bot_crawl_checker
Category: Network
Machine-readable: openapi.json · markdown.md

This tool fetches a fixed, hard-coded public upstream, never a private host. Results depend on that upstream.

Request body

Example arguments

More Network tools

More to explore