Bot Crawl Checker

Seo REST API MCP network tool

Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or blocked for a URL by the site's robots.txt, with the matching rule and user-agent group per bot.

This page documents the HTTP endpoint POST /api/tools/bot-crawl-checker/execute. See the MCP reference →

This tool fetches a fixed, hard-coded public upstream — never a private host. Results depend on that upstream.

Use in the browser View as Markdown

Call it over REST

POST https://api.findutils.com/api/tools/bot-crawl-checker/execute · no API keys · 60 requests/min per IP

Execute — verified arguments

curl -X POST https://api.findutils.com/api/tools/bot-crawl-checker/execute \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://findutils.com/"
  }'

Parameter schema for this endpoint

curl https://api.findutils.com/api/tools/bot-crawl-checker

Interactive docs · OpenAPI 3.1 spec · All REST tools

Input schema

Argument	Type	Required	Description
url	string	yes	Public http(s) URL to check, e.g. "https://example.com/blog/post".

Example arguments (verified)

{
  "url": "https://findutils.com/"
}

Also an MCP tool

Claude, Cursor, and any MCP client can call this tool as findutils:bot_crawl_checker after one connect command.

Connect once

claude mcp add findutils --transport http https://mcp.findutils.com/

Open the MCP reference for Bot Crawl Checker →

More Seo tools

Domain Rating Checker

Return the Ahrefs Domain Rating (0-100) for up to 25 domains in one call, with a per-domain status.

Faq Schema Generator

Generate FAQPage JSON-LD structured data from question-answer pairs, and return the JSON-LD string, a ready <script> tag, the schema object,

Form To Json Schema

Returns a form schema object ({ formId, method, action, fields, submitText }) extracted from HTML form markup.

Geo Analyzer

Return a Generative Engine Optimization (GEO) report for a public web page: an overall 0-100 score, per-category scores (technical, content,

Graphql Schema Validator

Validate a GraphQL SDL schema and return errors, naming-convention warnings, and counts of types, queries, mutations, and subscriptions.

Heading Structure Analyzer

Analyze the H1-H6 heading hierarchy of an HTML document and return every heading, SEO warnings (missing or multiple H1, skipped levels, empt

Hreflang Tag Generator

Generate hreflang annotations for multilingual pages and return them as HTML <link> tags, an HTTP Link header, and an XML sitemap block, plu

Json Ld Generator

Generate JSON-LD structured data for 12 schema.org types (Article, BlogPosting, Product, Organization, LocalBusiness, Person, Event, FAQPage