跳至正文
技术 SEO 与抓取索引 · Google SEO

Is Cloudflare Blocking Googlebot? Verify the Request and the Rule

// / / 光算科技

To establish whether Cloudflare is blocking Googlebot, connect a real failed crawl with the matching edge event and rule. A browser 200, a robots.txt Allow, or a request that merely uses the Googlebot User-Agent cannot establish that the real crawler receives the intended page. Check the response body, crawler identity, event time and the product that acted.

Capture one failure with enough detail to investigate

  1. Select an affected public page or sitemap and a comparable working URL. Save the full URLs, time zone and observation time.
  2. Inspect the indexed and live results separately in URL Inspection. Record fetch details and rendered output where available. An inspection request and an ordinary scheduled Googlebot crawl are different observations.
  3. Use Crawl stats to look for response patterns, then correlate them with CDN and origin logs. A 403 can come from more than one layer; a 5xx can indicate an origin or upstream failure rather than a deliberate block.
  4. In Cloudflare Security Analytics/Events, inspect the matching request's Service, action and rule information. Preserve its request or Ray identifier when available. Absence from a filtered or limited event view does not prove no block occurred.

Cloudflare documents cf-mitigated: challenge and a text/html content type for Challenge Page responses. A challenge returned instead of an XML sitemap is not valid sitemap content. Do not classify challenge versus block from HTTP status alone; inspect the response and event.

Verify which crawler actually made the request

Google's request-verification guide describes reverse DNS followed by forward DNS back to the original IP, or matching the relevant published IP-range list. Use the list for the correct crawler/fetcher category. A Google-owned address alone does not identify every request as the ordinary Search crawler.

A local curl request with User-Agent: Googlebot still originates from your own machine. Comparing it with a browser-style request may reveal a User-Agent-dependent response, but does not test real Googlebot identity or locate the blocking layer by itself.

Cloudflare's cf.client.bot field indicates a known good bot or crawler, not Googlebot alone. Its Verified bots documentation also distinguishes the newer behavior taxonomy from legacy categories usable in WAF rules. Select the identity and public paths you intend to admit using fields available to the actual plan; do not invent an exact Google-only rule from a broad boolean.

Match the repair to the responsible product

Observed service or controlWhat the current documentation permitsCheck before applying a change
WAF custom ruleAdjust its expression, order or a specifically scoped exceptionAn earlier terminating block/challenge prevents later rules from helping
Managed rules or rate limitingUse the supported exception/Skip scope for the identified featureSelecting one phase does not disable every security product
Bot Fight ModeNo WAF custom-rule or Page Rule bypassIf confirmed as the cause, evaluate disabling this mode or changing to a product that supports exceptions
Super Bot Fight ModeSupports a custom Skip rule for its phaseKeep the match narrow and select SBFM; other checks can still act
Security Level / Under AttackCurrent controls turn Under Attack mode on or offDo not follow obsolete instructions to lower a High/Medium threshold
AI bot policiesReview Search, Agent and Training behavior policies and page scopeA mixed-purpose crawler can be affected by its Training classification

Security-feature interoperability lists the execution order and notes that terminating actions stop later evaluation. Available Skip options distinguishes current rulesets, phases and products outside the Ruleset Engine. Choosing “Skip SBFM” does not mean “allow through everything.” Keep request logging available when testing an exception.

The distinction between Bot Fight Mode and Super Bot Fight Mode matters: the former cannot be skipped with WAF custom rules; the latter has an explicit exception workflow. Check the exact Service label instead of applying the same recipe to both.

Check dated AI-bot and Security Level instructions

As checked on 22 October 2026, the AI bot policy page contains a dated 15 September 2026 change notice: new-domain defaults block Training and Agent behavior on ad-bearing pages while Search remains allowed, and Training blocks also affect mixed-purpose Search/Training crawlers. The page still contains a legacy section marked as deprecating on that date, with the older exclusion wording. Read the dated change notice together with the actual zone configuration and event; do not assume every existing account received the same defaults.

“Search: Allow” means that setting does not add a block; it is not proof that a Training policy, another security product or the origin cannot block the request. Review both behavior classification and whether the control applies to all pages or only pages with ads. If the account behavior is unclear, retain the event and obtain product-specific clarification before claiming a universal workaround.

The current Security Level documentation says the old dashboard shows “Always protected”; Under Attack is controlled separately there, while the new dashboard/API uses Security Level for that mode. It also says the former threat score is always zero. A tutorial that simply says “change High to Low” no longer describes these controls reliably.

Verify access first, then observe indexing

  1. Record the original rule and make the smallest justified change for the identified request. Preserve a way to restore the prior configuration.
  2. Repeat the affected and comparison requests. Confirm the intended page or XML body, not merely HTTP 200, and test ordinary users and sensitive routes too.
  3. Repeat the live inspection when appropriate, then inspect subsequent real crawler events and the origin response. One successful test does not establish all crawler paths.
  4. Observe later indexed data separately. A fetch repair does not guarantee that “Discovered” or “Crawled — currently not indexed” will clear.

Google's HTTP guidance distinguishes 200 processing, most 4xx responses and 429/server errors. Keep network recovery, sitemap processing, page indexing and rankings as separate outcomes. If access is working, continue with the new-site visibility diagnosis and sitemap-to-page indexing checks. Guangsuan's SEO service includes this type of technical investigation.