Bot Crawl Checker

Seo REST API MCP network tool

Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or blocked for a URL by the site's robots.txt, with the matching rule and user-agent group per bot.

This page documents the MCP tool findutils:bot_crawl_checker. See the REST reference →

This tool fetches a fixed, hard-coded public upstream — never a private host. Results depend on that upstream.

Use in the browser View as Markdown

Call it over MCP

Tool name findutils:bot_crawl_checker · no API keys · 120 requests/min per IP

Claude Code

claude mcp add findutils --transport http https://mcp.findutils.com/

Claude Desktop — claude_desktop_config.json

{
  "mcpServers": {
    "findutils": {
      "url": "https://mcp.findutils.com/"
    }
  }
}

Raw JSON-RPC (any MCP client) — verified example

curl -X POST https://mcp.findutils.com/ \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "bot_crawl_checker",
      "arguments": {
        "url": "https://findutils.com/"
      }
    }
  }'

Input schema

Argument	Type	Required	Description
url	string	yes	Public http(s) URL to check, e.g. "https://example.com/blog/post".

Example arguments (verified)

{
  "url": "https://findutils.com/"
}

Also a REST endpoint

The same tool answers plain HTTP at POST /api/tools/bot-crawl-checker/execute — no key, 60 requests/min.

Open the REST reference for Bot Crawl Checker →

More Seo tools

Domain Rating Checker

Return the Ahrefs Domain Rating (0-100) for up to 25 domains in one call, with a per-domain status.

Faq Schema Generator

Generate FAQPage JSON-LD structured data from question-answer pairs, and return the JSON-LD string, a ready <script> tag, the schema object,

Form To Json Schema

Returns a form schema object ({ formId, method, action, fields, submitText }) extracted from HTML form markup.

Geo Analyzer

Return a Generative Engine Optimization (GEO) report for a public web page: an overall 0-100 score, per-category scores (technical, content,

Graphql Schema Validator

Validate a GraphQL SDL schema and return errors, naming-convention warnings, and counts of types, queries, mutations, and subscriptions.

Heading Structure Analyzer

Analyze the H1-H6 heading hierarchy of an HTML document and return every heading, SEO warnings (missing or multiple H1, skipped levels, empt

Hreflang Tag Generator

Generate hreflang annotations for multilingual pages and return them as HTML <link> tags, an HTTP Link header, and an XML sitemap block, plu

Json Ld Generator

Generate JSON-LD structured data for 12 schema.org types (Article, BlogPosting, Product, Organization, LocalBusiness, Person, Event, FAQPage