A site can be fully online, return 200 in a browser, and still lose Google traffic because a security layer in front of it is turning Googlebot away. Cloudflare is a common place for that to happen, but it is not the only one, and it is not always the cause. This guide shows how to confirm that crawling is actually blocked, which Cloudflare controls tend to cause it, and how to allow verified crawlers without opening the site to abuse.
First confirm it is a crawl block, not a ranking problem
Do not start changing firewall settings on a hunch. Gather evidence first:
- Search Console → Settings → Crawl stats. Look at response codes over time. A sudden rise in
403or5xxfor Googlebot points to a server-side block, not a content problem. - URL Inspection → Test live URL. If Google cannot fetch a page that loads fine in your browser, compare the details: the response code and any restrictions are shown in the page availability section.
- Cloudflare → Security → Events. Filter recent events by the request path or by bot category. Blocked or challenged requests from a verified bot category are the clearest signal.
- Distinguish "challenge" from "block". A challenge returns a page the crawler cannot use; a block returns a 4xx. Both hurt indexing, but they come from different settings.
If none of these show Googlebot being stopped, the problem is elsewhere — robots.txt, a noindex, a redirect loop, or indexing and quality signals. Fixing the firewall will not help.
Cloudflare features that commonly block legitimate crawlers
| Feature | How it can hurt | What to review |
|---|---|---|
| Security Level (High) or "I'm Under Attack" mode | Aggressive challenge thresholds or a global interstitial can stop crawlers | Lower the level when the attack has passed |
| Bot Fight Mode / Super Bot Fight Mode | May challenge or block automated clients, including crawlers, depending on configuration | Review bot settings; allow verified bots |
| WAF custom or managed rules | A rule that matches a bot signature or a path can block Googlebot | Check rule expressions for user-agent or path matches |
| Rate limiting | Fast crawling from Google can trip a low threshold | Ensure verified bots are exempt or the threshold is realistic |
| IP Access Rules / country blocks / Zone Lockdown | Blocking a region or range that includes crawler IPs | Check allow rules for Google's ranges |
| Browser Integrity Check | Can flag non-browser-like clients | Review whether it applies to verified bots |
Note that Cloudflare's own security posture is not the enemy here. The goal is a narrow allowance for verified search crawlers, not disabling protection for everyone.
The right way to allow Googlebot
There is a wrong way and a right way. The wrong way is a rule that allows anything whose User-Agent contains "Googlebot". User-Agent strings are trivially forged, so that rule also opens the door to anyone claiming to be a crawler.
The right way layers identity checks:
- Use verified-bot signals. Cloudflare classifies known crawlers and exposes fields such as
cf.client.botand a verified-bot category to rules. Allow the ones you actually want (for example, the search crawler, not every automated client). - Combine with IP ownership. Google publishes the IP ranges used by its crawlers as JSON. A rule that combines the verified-bot signal with Google's published ranges is far more resistant to spoofing than a user-agent match.
- Scope the exception by path where possible. If only public pages need crawling, you do not need to allow crawlers through admin or API paths.
- Do not disable everything. Skip or bypass rules for the crawler rather than turning off the WAF.
If you are unsure, make the smallest change that lets the specific verified bot through the specific rule that was blocking it, then measure.
Verify the fix properly
- Re-test live. Run URL Inspection → Test live URL on a page that was blocked and confirm the fetch is now successful.
- Watch Security Events. Make sure the crawler now shows as allowed, and that you did not create an obvious bypass for abusive traffic.
- Watch Crawl stats over several days. Response-code changes and crawl volume move slowly; a same-hour recovery is not the right expectation.
- Check the affected pages in the Page indexing report after a recrawl. "Crawled – currently not indexed" or "Discovered" states usually clear over time once fetching works again.
Common mistakes
- Changing five settings at once. You will not know which one mattered, and you may have weakened protection.
- Assuming a
200to your browser means a200to Googlebot. The two can be treated differently by any WAF. - Trusting a user-agent-only allowlist. It invites spoofing.
- Leaving "Under Attack" mode on indefinitely. It is an emergency measure; keeping it on keeps legitimate traffic out.
Guangsuan (光算) diagnoses crawling and indexing problems like this within its Google SEO service. If you are not sure whether the block is at the CDN, the origin or somewhere in between, see the companion piece on diagnosing why pages are not appearing.
Sources
- Google Search Central — Verifying Googlebot
- Google Search Central — Google common crawlers
- Search Console Help — Crawl stats report
- Cloudflare Docs — Verified bots
Cloudflare and Google both change interfaces and terminology; confirm the current settings and documentation before making changes in production.