# Robots Txt Generator — MCP tool `findutils:robots_txt_generator`

Return a robots.txt file built from user-agent rules (allow, disallow, crawl-delay), sitemap URLs and an optional Host directive, or from a preset: allow-all, block-all, wordpress, ecommerce.

- Category: seo
- MCP server: https://mcp.findutils.com/ (Streamable HTTP, no API keys, 120 req/min per IP)
- REST endpoint: POST https://api.findutils.com/api/tools/robots-txt-generator/execute (no API keys, 60 req/min per IP)
- Reference page: https://findutils.com/mcp/robots-txt-generator/
- Same tool on the other surface: https://findutils.com/api/robots-txt-generator/

## Connect

```bash
claude mcp add findutils --transport http https://mcp.findutils.com/
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "findutils": {
      "url": "https://mcp.findutils.com/"
    }
  }
}
```

## Call the tool (verified example)

```bash
curl -X POST https://mcp.findutils.com/ \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "robots_txt_generator",
      "arguments": {
        "preset": "allow-all",
        "sitemaps": [
          "https://example.com/sitemap.xml"
        ],
        "include_comments": false
      }
    }
  }'
```

## Input schema

| Argument | Type | Required | Description |
|---|---|---|---|
| `rules` | array | no | Rule groups: objects with user_agent (e.g. "*", "Googlebot", "GPTBot"), allow (array of paths), disallow (array of paths), crawl_delay (seconds). Overrides preset. |
| `preset` | string (allow-all \| block-all \| wordpress \| ecommerce) | no | Starting rule set when rules is omitted. Default: `"allow-all"`. |
| `sitemaps` | array | no | Absolute sitemap URLs. |
| `host` | string | no | Optional Host directive value (Yandex), e.g. "https://example.com". |
| `include_comments` | boolean | no | Prepend the generator comment header with the date. Default: `true`. |

Example arguments (verified):

```json
{
  "preset": "allow-all",
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ],
  "include_comments": false
}
```

## Also a REST endpoint

```bash
curl -X POST https://api.findutils.com/api/tools/robots-txt-generator/execute \
  -H "Content-Type: application/json" \
  -d '{
    "preset": "allow-all",
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ],
    "include_comments": false
  }'

# Parameter schema
curl https://api.findutils.com/api/tools/robots-txt-generator
```

Full REST reference: https://findutils.com/api/robots-txt-generator/ · OpenAPI 3.1 spec: https://findutils.com/api/openapi.json

---
Full catalog: POST https://mcp.findutils.com/ with method `tools/list` · https://findutils.com/mcp/ · https://findutils.com/llms.txt