Return a robots.txt file built from user-agent rules (allow, disallow, crawl-delay), sitemap URLs and an optional Host directive, or from a preset: allow-all, block-all, wordpress, ecommerce.
This page documents the HTTP endpoint POST /api/tools/robots-txt-generator/execute. See the MCP reference →
Call it over REST
POST https://api.findutils.com/api/tools/robots-txt-generator/execute · no API keys · 60 requests/min per IP
Execute — verified arguments
curl -X POST https://api.findutils.com/api/tools/robots-txt-generator/execute \
-H "Content-Type: application/json" \
-d '{
"preset": "allow-all",
"sitemaps": [
"https://example.com/sitemap.xml"
],
"include_comments": false
}'Parameter schema for this endpoint
curl https://api.findutils.com/api/tools/robots-txt-generatorInput schema
| Argument | Type | Required | Description |
|---|---|---|---|
| rules | array | no | Rule groups: objects with user_agent (e.g. "*", "Googlebot", "GPTBot"), allow (array of paths), disallow (array of paths), crawl_delay (seconds). Overrides preset. |
| preset | string (allow-all | block-all | wordpress | ecommerce) | no | Starting rule set when rules is omitted. Default: "allow-all". |
| sitemaps | array | no | Absolute sitemap URLs. |
| host | string | no | Optional Host directive value (Yandex), e.g. "https://example.com". |
| include_comments | boolean | no | Prepend the generator comment header with the date. Default: true. |
Example arguments (verified)
{
"preset": "allow-all",
"sitemaps": [
"https://example.com/sitemap.xml"
],
"include_comments": false
}Also an MCP tool
Claude, Cursor, and any MCP client can call this tool as
findutils:robots_txt_generator after one connect command.
Connect once
claude mcp add findutils --transport http https://mcp.findutils.com/More Seo tools
Bot Crawl Checker
Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or
Domain Rating Checker
Return the Ahrefs Domain Rating (0-100) for up to 25 domains in one call, with a per-domain status.
Faq Schema Generator
Generate FAQPage JSON-LD structured data from question-answer pairs, and return the JSON-LD string, a ready <script> tag, the schema object,
Form To Json Schema
Returns a form schema object ({ formId, method, action, fields, submitText }) extracted from HTML form markup.
Geo Analyzer
Return a Generative Engine Optimization (GEO) report for a public web page: an overall 0-100 score, per-category scores (technical, content,
Graphql Playground
Check a GraphQL query for balanced braces and, when a schema is supplied, unknown fields on Query/Mutation/Subscription types.
Graphql Schema Validator
Validate a GraphQL SDL schema and return errors, naming-convention warnings, and counts of types, queries, mutations, and subscriptions.
Heading Structure Analyzer
Analyze the H1-H6 heading hierarchy of an HTML document and return every heading, SEO warnings (missing or multiple H1, skipped levels, empt