Robots Txt Tester MCP tool
MCP
findutils:robots_txt_tester Test URLs or paths against robots.txt text for a given crawler, following RFC 9309 as Google implements it: the group naming the crawler's product token (all such groups merged, "*" only as a fallback), * and $ wildcards, the longest matching rule wins and Allow wins a tie. Returns allowed/blocked per path with the deciding rule and its line, plus sitemaps, crawl-delay and line-numbered warnings. Checks the pasted text only and fetches nothing.
Arguments
application/json-
robots
string required
The full robots.txt text.
-
paths
array required
Paths ("/admin?x=1") or full URLs to test, at most 500. For a URL only the path and query are tested.
-
user_agent
string optional
Crawler product token, e.g. "Googlebot", "GPTBot", "Bingbot". Default "*". Default
"*".
Example arguments
Verified{
"robots": "User-agent: *\nDisallow: /admin",
"paths": [
"/admin",
"/blog/"
]
}More SEO tools
- Bot Crawl Checker Return which of 24 well-known crawlers (Googlebot, BingBot, GPTBot, ClaudeBot, PerplexityBot, AhrefsBot, Twitterbot and more) are allowed or
- Domain Rating Checker Return the Ahrefs Domain Rating (0-100) for up to 25 domains in one call, with a per-domain status (provider_unavailable means Ahrefs is not
- Faq Schema Generator Generate FAQPage JSON-LD structured data from question-answer pairs, and return the JSON-LD string, a ready <script> tag, the schema object,
- Form To Json Schema Returns a form schema object ({ formId, method, action, fields, submitText }) extracted from HTML form markup.
- Geo Analyzer Return a Generative Engine Optimization (GEO) report for a public web page: an overall 0-100 score, per-category scores (technical, content,
- Graphql Playground Check a GraphQL query for balanced braces and, when a schema is supplied, unknown fields on Query/Mutation/Subscription types.