We'll send the crawl report to this email after verification.
How It Works
Enter URL & Email
Provide the website URL you want to check and your email address.
Verify Email
Click the verification link in your inbox to start the crawl check.
Get Report
Receive a detailed report showing which of the 24 bots can access your site.
24 Bots We Check
We simulate requests using each bot's real User-Agent header and check robots.txt, HTTP status, meta tags, and WAF challenges.
What We Check
robots.txt Rules
Parses robots.txt and checks each bot's specific User-Agent directives.
HTTP Response
Sends actual GET requests with each bot's real User-Agent header.
Meta Robots Tags
Detects noindex/nofollow directives in HTML meta tags.
X-Robots-Tag Headers
Checks HTTP response headers for indexing restrictions.
WAF/Firewall Detection
Detects Cloudflare JS challenges, CAPTCHAs, and 403 blocks.
Rate Limiting
Identifies HTTP 429 responses that throttle bot access.
How to Check Your Website's Bot Accessibility
- 1
Enter your website URL
Type or paste the URL you want to check into the Website URL field. You can enter a full URL likehttps://example.com/pageor justexample.com— we'll add HTTPS automatically. - 2
Provide your email address
Enter a valid email address where you want to receive the crawl report. We use email verification to prevent abuse and ensure you get your results. - 3
Click Check Crawlability
Hit the Check Crawlability button. We'll send a verification email to your inbox with a one-click confirmation link. - 4
Verify your email
Open the verification email and click Verify & Start Check. This triggers the full crawl check — all 24 bots are tested against your URL simultaneously. - 5
Receive your report
Within a few minutes, you'll get a detailed email report showing which bots can access your site, which are blocked, and exactly why — including robots.txt rules, HTTP errors, meta tags, and WAF challenge detection.
Why Check Bot & Crawler Accessibility?
Websites can accidentally block search engine crawlers, AI bots, and social media bots through misconfigured firewalls, robots.txt rules, or security settings. If Googlebot or BingBot can't access your pages, your content won't appear in search results. Blocking AI crawlers like GPTBot or ClaudeBot means your content won't be cited in AI-powered search experiences. Social media bots need access to generate link previews when your URLs are shared.
Common causes of accidental blocking include Cloudflare Bot Fight Mode, overly aggressive WAF rules, misconfigured robots.txt files, and meta robots tags set to noindex. These issues can go undetected for months, silently hurting your search rankings and traffic. This tool checks your URL against 24 well-known bots to identify these issues.
What This Tool Checks
For each of the 24 bots, we perform five independent checks:
- robots.txt Analysis — We fetch your site's robots.txt file and parse the User-agent rules specific to each bot. A bot might be allowed by the wildcard (
User-agent: *) but specifically blocked by its own directive (e.g.,User-agent: GPTBot / Disallow: /). - HTTP Response — We send an actual GET request using each bot's real User-Agent header string. This reveals whether your server, CDN, or WAF returns different responses to different bots (403 Forbidden, 429 Rate Limited, etc.).
- X-Robots-Tag Header — Some servers send indexing directives via HTTP headers instead of HTML meta tags. We check for
noindexandnofollowin the X-Robots-Tag response header. - Meta Robots Tag — We parse the first 8KB of HTML to find
<meta name="robots" content="...">tags that might block indexing. - WAF Challenge Detection — We detect Cloudflare JavaScript challenges, CAPTCHA pages, and other WAF challenge responses that bots cannot solve.
Bots We Test
Search Engine Crawlers (12)
Googlebot, BingBot, YandexBot, Baiduspider, DuckDuckBot, Yahoo Slurp, Applebot, SeznamBot, Qwantbot, Yeti (Naver), MojeekBot, and Kagi Bot. These crawlers index your pages for their respective search engines.
AI Crawlers (7)
GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended, CCBot (Common Crawl), and cohere-ai (Cohere). These bots access content for AI training and AI-powered search experiences.
SEO & Social Bots (5)
AhrefsBot, SemrushBot (SEO analysis), Twitterbot, Facebookbot, and LinkedInBot (social media link previews).
How It Compares
How to Fix Common Issues
Blocked by robots.txt: Edit your robots.txt file to add Allow: / for the blocked bot's User-agent.
HTTP 403 (WAF/Firewall): Check your CDN's security settings. In Cloudflare, go to Security > Bots and ensure Bot Fight Mode isn't blocking verified crawlers. Create a WAF rule with expression (cf.client.bot) and action Skip.
Cloudflare JS Challenge: Lower your Security Level from "High" to "Medium" or create a firewall rule that skips challenges for known bot user agents.
Meta noindex: Remove or update the <meta name="robots" content="noindex"> tag from your HTML. This is sometimes added during development and forgotten in production.