Security8 min read@codewitholgun

Why Your Website Is Invisible to Search Engines and AI Bots (And How to Fix It)

Tags:SecuritySEOCrawlersCloudflareAI Bots

We Accidentally Blocked Every Search Engine for 3 Months

Last week, we discovered that a single Cloudflare WAF rule had been silently blocking Googlebot, BingBot, and every AI crawler from accessing findutils.com since January 2026. Three months of zero Bing indexing, zero AI citations, and broken social media previews — all because of one checkbox in a firewall dashboard.

The worst part? There were no errors, no alerts, no warnings. Bing Webmaster Tools simply said "Discovered but not crawled." Google Search Console showed the pages as "known" but never fetched. We only caught it when we simulated bot requests using curl with different User-Agent headers and saw every single one getting blocked.

This story isn't unique. According to Cloudflare's own data, thousands of websites accidentally block search engine crawlers through misconfigured WAF rules, overly aggressive bot protection, and forgotten development settings. The difference between a site that gets crawled and one that doesn't can be a single configuration toggle.

The 5 Ways Your Site Silently Blocks Bots

Bot blocking happens at multiple layers, and each layer can independently prevent crawling. Your site might pass 4 checks and fail on the 5th — and that one failure is enough to kill your search visibility.

1. Cloudflare Bot Fight Mode and WAF Rules

This is the most common culprit we see. Cloudflare's Bot Fight Mode is designed to block malicious bots, but it doesn't distinguish between a DDoS bot and Googlebot. When enabled, it serves JavaScript challenges that legitimate crawlers cannot solve.

Custom WAF rules are even more dangerous. A rule like "challenge all requests that don't come from a browser" will block every search engine, every AI bot, and every social media crawler. The site looks perfectly fine to human visitors while being completely invisible to search engines.

How it looked for us: Bing Webmaster Tools showed "Discovered but not crawled — URL cannot appear on Bing." The URL was known but never fetched because Bingbot kept getting challenged.

The fix: Create a WAF exception rule with expression (cf.client.bot) and action Skip. This allows Cloudflare's verified bot list (including Googlebot, Bingbot, and others) through without challenges.

2. robots.txt Blocking Specific Bots

Many site owners add broad Disallow rules to their robots.txt without realizing the impact. A common pattern we see:

1
2
3
4
5
6
7
8
9
User-agent: *
Disallow: /api/
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

This blocks AI crawlers from your entire site while allowing search engines. Whether that's intentional depends on your AI content strategy — but many site owners add these rules after reading a scary headline about AI scraping without understanding that they're also blocking ChatGPT web search, Perplexity answers, and Claude citations.

3. HTTP 405 Method Not Allowed

This one caught us off guard. One of our Cloudflare Workers returned HTTP 405 for HEAD requests because the code only handled GET:

TS
1
2
3
if (request.method !== 'GET') {
    return new Response('Method not allowed', { status: 405 });
}

Bingbot uses HEAD requests to check if URLs are accessible before crawling them. Getting a 405 on HEAD told Bing the URL was broken, resulting in a "Blocked" status in Bing Webmaster Tools. The fix was two lines: allow HEAD requests and return the same headers without a body.

4. Meta Robots noindex Left from Development

During development, it's common practice to add <meta name="robots" content="noindex, nofollow"> to prevent search engines from indexing staging sites. The problem: this tag sometimes makes it into production.

Unlike robots.txt (which blocks crawling), a noindex meta tag allows crawling but prevents indexing. Search engines fetch the page, read the noindex directive, and then deliberately exclude it from results. The page exists, the server returns 200, but the content is invisible in search.

5. X-Robots-Tag HTTP Headers

The sneakiest blocking method. Some CDNs and server configurations add X-Robots-Tag: noindex as an HTTP response header. This has the same effect as a meta noindex tag but is completely invisible when viewing the page source. You'd only find it by inspecting HTTP response headers with curl -I or a security headers analysis tool.

How to Diagnose Bot Blocking Yourself

After our Cloudflare WAF incident, we developed a systematic approach to checking all 5 blocking layers. Here's how you can do it yourself using free tools.

Testing with curl

The most reliable way to simulate a bot visit is with curl using the bot's real User-Agent string:

1
2
3
4
5
6
7
8
9
10
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -I https://your-site.com

# Test as BingBot
curl -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" \
  -I https://your-site.com

# Test as GPTBot
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  -I https://your-site.com

Look for:

  • HTTP 200 = bot can access the page
  • HTTP 403 = blocked by WAF or firewall
  • HTTP 503 with Cloudflare challenge markers = JS challenge being served
  • X-Robots-Tag: noindex in the response headers = indexing blocked via header

Checking robots.txt

Fetch your robots.txt and look for bot-specific rules:

curl https://your-site.com/robots.txt

Check for Disallow: / under any bot's User-agent section. Pay special attention to AI crawler directives (GPTBot, ClaudeBot, CCBot, PerplexityBot).

Checking Meta Tags

Fetch the page and search for noindex directives:

curl -s https://your-site.com | grep -i "noindex"

Using Search Console and Webmaster Tools

  • Google Search Console: Check Coverage > Excluded for "Blocked by robots.txt" or "Noindex" entries
  • Bing Webmaster Tools: Check URL Inspection for "Discovered but not crawled" or "Blocked" status

The Bots That Matter Most in 2026

In 2026, bot accessibility isn't just about Google anymore. There are three tiers of bots you need to care about:

Tier 1: Search Engine Crawlers (Non-Negotiable)

Googlebot, BingBot, and Baiduspider are the foundation. Block these and your organic traffic drops to zero. Google processes over 8.5 billion searches per day, and every result starts with Googlebot successfully fetching your page.

Tier 2: AI Crawlers (Growing Fast)

GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot are the new frontier. AI-powered search experiences are growing rapidly. When someone asks ChatGPT "what's the best free JSON formatter?", your site can only be cited if ChatGPT-User has access to crawl it.

Many sites are making a nuanced choice here: allowing AI search bots (ChatGPT-User, PerplexityBot) while blocking AI training bots (GPTBot, CCBot, Google-Extended). This lets your content appear in AI search results without contributing to model training datasets.

Tier 3: Social Media Bots (Often Forgotten)

Twitterbot, Facebookbot, and LinkedInBot fetch your Open Graph meta tags to generate link preview cards. If you share a URL on Twitter and it shows as a plain text link instead of a rich card with title, description, and image — your site is blocking Twitterbot.

What to Do Right Now

Step 1: Test Your Site with curl

Run the curl commands above for Googlebot, BingBot, and GPTBot against your most important pages. Check the HTTP status codes and response headers.

Step 2: Check Cloudflare Settings (If Applicable)

If you use Cloudflare, check Security > Bots for Bot Fight Mode and Security > WAF for any custom rules that might challenge or block crawler traffic.

Step 3: Audit Your robots.txt

Review your robots.txt directives carefully. Make sure you're not accidentally blocking bots you want to allow. Use the Robots.txt Generator if you need to build one from scratch.

Step 4: Inspect Response Headers

Run the Security Headers Analyzer on your key pages to check for X-Robots-Tag headers. Or use curl -I to inspect headers manually.

Step 5: Set Up a Quarterly Check

Bot blocking can happen silently after any infrastructure change. Schedule a quarterly crawl check to catch issues before they cost you months of traffic.

Related Tools

FAQ

Q1: How do I know if my site is blocked by search engines? A: Use curl with different bot User-Agent strings to simulate crawler visits. Check the HTTP status codes and response headers. Also review Google Search Console and Bing Webmaster Tools for crawl errors like "Discovered but not crawled" or "Blocked by robots.txt."

Q2: Can Cloudflare block Googlebot? A: Yes. Cloudflare's Bot Fight Mode and custom WAF rules can accidentally block Googlebot by serving JavaScript challenges that crawlers cannot solve. Create a WAF rule with expression (cf.client.bot) and action Skip to allow verified bots through.

Q3: Should I allow AI crawlers on my website? A: If you want your content cited in AI-powered search results (ChatGPT, Claude, Perplexity), allow their crawlers. Many sites selectively allow search-focused AI bots (ChatGPT-User, PerplexityBot) while blocking training-only bots (GPTBot, CCBot).

Q4: How often should I check bot accessibility? A: After any infrastructure change (CDN settings, WAF rules, server configuration) and quarterly as routine maintenance. Bot blocking can happen silently without any visible errors.

Q5: What's the easiest way to check if bots can crawl my site? A: Use curl -A "User-Agent-String" -I https://your-site.com for quick spot checks. For a thorough audit, test multiple bot User-Agents across your key pages and check robots.txt, meta tags, response headers, and WAF behavior.