技术 SEO 与抓取索引 · Google SEO

Changed robots.txt: how long until Google updates it (and the index)?

// Don Jiang/ Updated 2026-10-11 / 光算科技

When people ask "how long does Google take to update robots.txt", they are usually asking one of two different questions, and the answers are on different clocks. The first is how long until Google refreshes its cached copy of the file — that is short and, if you want, you can ask Google to do it now. The second is how long until the search results for the affected URLs change — that is not controlled by robots.txt at all, and treating it as if it were is the most common mistake on the topic.

Google's official documentation page 'How to write and submit a robots.txt file', showing robots.txt rules and submission guidance — Google Search Central, "How to write and submit a robots.txt file" (captured 2026-10-01; page last updated 2025-11-21). The companion spec page documents the 24-hour cache and the 12-hour / 30-day behaviour for server errors.

Clock one: how long until Google re-reads the file

Google's own documentation is specific. In "How Google interprets the robots.txt specification", it states that Google generally caches the contents of robots.txt for up to 24 hours, and may cache it longer when refreshing is not possible (for example, timeouts or 5xx errors). It also notes that Google may increase or decrease the cache lifetime based on the max-age Cache-Control response header. So the default expectation is "within a day", not "instantly".

If waiting a day is not good enough, the "Update your robots.txt file" page says to use the Request a recrawl function of the robots.txt report in Search Console. That is the supported way to refresh the cached copy sooner. Because Google fetches robots.txt with a non-conditional GET, a plain page-cache purge at your CDN can also matter if your CDN is serving a stale copy to Googlebot — but the cache Google applies to the file itself is the one measured in the 24-hour statement.

What changes the timing

What your server returns for /robots.txt	How Google treats it
2xx with the new file	The file is parsed and used; the cached copy is refreshed
3xx redirect	Google follows at least five redirect hops, then treats the result as a 404
4xx (except 429)	Treated as if no robots.txt existed — i.e. no crawl restrictions. Do not use 401 or 403 to slow crawling
5xx or network error	Google stops crawling for the first 12 hours while retrying; then uses the last good version for up to 30 days, still retrying
Still broken after 30 days	If the site is otherwise reachable, Google behaves as if there is no robots.txt; if the site itself is unavailable, Google stops crawling but keeps checking

These are Google's documented behaviours, not a schedule you can negotiate. A max-age that is very long can extend how long the cached copy is reused, so do not set an aggressive cache lifetime on /robots.txt and then expect a quick change.

Clock two: the index is a different system

robots.txt controls crawling, not indexing. Google's spec page is explicit that when a page is disallowed for crawling, Google cannot index its content, but may still index the URL and show it in results without a snippet. In other words, blocking a URL in robots.txt is not a removal request. If your goal is "this page should not appear in search", robots.txt is the wrong tool and will often make the situation worse, because Google can no longer read the page.

The removal path is the opposite of intuition:

Let Google crawl the page. A blocked page cannot be read, so its noindex directive cannot be seen.
Return noindex (for example a X-Robots-Tag: noindex header or a meta robots tag) on that page and wait for a recrawl.
For urgent removal, use the Removals tool in Search Console for a temporary removal while the noindex takes effect — but do not leave it crawl-blocked.

The old version of this article implied that robots.txt itself deletes results within a fixed number of days. It does not; the crawl cache and the index have separate clocks, and the index updates as Google recrawls and re-evaluates the URLs it is still allowed to fetch.

How to verify what actually happened

robots.txt report (Search Console). Shows the copy Google last fetched and its fetch time, and lets you request a recrawl.
URL Inspection → Live test. Confirms the current crawl permission for a specific URL and whether the fetch succeeds.
Your server logs. If you were blocking a directory, requests for it should fall; if you unblocked one, new requests for it should appear as Google recrawls.
Crawl Stats. Distinguishes host availability problems (DNS, connection, robots) from ordinary crawl-volume changes.

Mistakes that reset the clock

Saving with a word processor. Smart quotes and a byte-order mark can invalidate lines. The file must be UTF-8 plain text; a BOM at the start is ignored, which can silently drop your first rule.
Using 401/403 to limit crawl rate. Google treats these as "no robots.txt", so the rules you wrote are ignored. Use Crawl-delay is not supported either; rate control is a separate topic.
Expecting deindexing. See above — use noindex or removals, with crawling allowed.
Accidental whole-site block. A stray Disallow: / under a group that matches your main crawler blocks everything until the file is fixed and re-fetched. Test with the report before uploading.
Wrong scope. A robots.txt on www.example.com does not apply to example.com, a subdomain, another protocol or a non-standard port.
Forgetting the 500 KiB limit. Content after 500 kibibytes is ignored, which can silently drop later rules.

A short, correct sequence

Edit the file locally and test it (the robots.txt report or Google's open-source parser).
Upload it to the correct host root, UTF-8, with a sane Cache-Control.
Confirm the live URL returns the new file, not a CDN copy.
If you need it sooner than about a day, request a recrawl in the robots.txt report.
If your goal is deindexing, change to allow + noindex (or use Removals) instead of relying on robots.txt.
Verify with URL Inspection and Crawl Stats; give the index its own time.

Guangsuan handles robots.txt, crawl and indexing diagnostics inside its Google SEO service. For conflicting directives on a single page, see what happens when an HTML tag and an X-Robots-Tag disagree (Chinese); for a sitemap that will not read, start from diagnosing the Sitemaps report errors.

Sources

The 24-hour cache, the 12-hour/30-day server-error behaviour and the 500 KiB limit are stated on Google's pages and can change; confirm the current wording there.