Return a robots.txt file built from user-agent rules (allow, disallow, crawl-delay), sitemap URLs and an optional Host directive, or from a preset: allow-all, block-all, wordpress, ecommerce.
This page documents the HTTP endpoint POST /api/tools/robots-txt-generator/execute. See the MCP reference →
Call it over REST
POST https://api.findutils.com/api/tools/robots-txt-generator/execute · no API keys · 60 requests/min per IP
Execute — verified arguments
curl -X POST https://api.findutils.com/api/tools/robots-txt-generator/execute \
-H "Content-Type: application/json" \
-d '{
"preset": "allow-all",
"sitemaps": [
"https://example.com/sitemap.xml"
],
"include_comments": false
}'Parameter schema for this endpoint
curl https://api.findutils.com/api/tools/robots-txt-generatorInput schema
| Argument | Type | Required | Description |
|---|---|---|---|
| rules | array | no | Rule groups: objects with user_agent (e.g. "*", "Googlebot", "GPTBot"), allow (array of paths), disallow (array of paths), crawl_delay (seconds). Overrides preset. |
| preset | string (allow-all | block-all | wordpress | ecommerce) | no | Starting rule set when rules is omitted. Default: "allow-all". |
| sitemaps | array | no | Absolute sitemap URLs. |
| host | string | no | Optional Host directive value (Yandex), e.g. "https://example.com". |
| include_comments | boolean | no | Prepend the generator comment header with the date. Default: true. |
Example arguments (verified)
{
"preset": "allow-all",
"sitemaps": [
"https://example.com/sitemap.xml"
],
"include_comments": false
}Also an MCP tool
Claude, Cursor, and any MCP client can call this tool as
findutils:robots_txt_generator after one connect command.
Connect once
claude mcp add findutils --transport http https://mcp.findutils.com/More Seo tools
Bot Crawl Checker
Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or
Domain Rating Checker
Return the Ahrefs Domain Rating (0-100) for up to 25 domains in one call, with a per-domain status.
Faq Schema Generator
Generate FAQPage JSON-LD structured data from question-answer pairs, and return the JSON-LD string, a ready <script> tag, the schema object,
Form To Json Schema
Returns a form schema object ({ formId, method, action, fields, submitText }) extracted from HTML form markup.
Geo Analyzer
Return a Generative Engine Optimization (GEO) report for a public web page: an overall 0-100 score, per-category scores (technical, content,
Graphql Schema Validator
Validate a GraphQL SDL schema and return errors, naming-convention warnings, and counts of types, queries, mutations, and subscriptions.
Heading Structure Analyzer
Analyze the H1-H6 heading hierarchy of an HTML document and return every heading, SEO warnings (missing or multiple H1, skipped levels, empt
Hreflang Tag Generator
Generate hreflang annotations for multilingual pages and return them as HTML <link> tags, an HTTP Link header, and an XML sitemap block, plu