Crawl budget: what it is, who needs to care and how to stop wasting it

What crawl budget is

Google cannot fetch every URL that exists. In its own words, “The web is a nearly infinite space, exceeding Google's ability to explore every publicly accessible URL.” So it spends a limited amount of time and resources on each site. That allocation is what SEOs call crawl budget.

Three details change how you should think about it:

• A “site” is a hostname. Google says “Google's crawling infrastructure defines a site as a unique hostname.” www.example.com, shop.example.com and blog.example.com each have their own crawl budget. Moving a section to a subdomain does not borrow budget from the main host, and an overloaded subdomain does not slow the others.
• Crawled is not indexed. “For Google Search, not every page that is crawled will necessarily be indexed.” After a page is fetched, it is still evaluated, grouped with duplicates and judged for the index. More crawling does not guarantee more indexed pages.
• Crawling is not a ranking factor. Google is direct about this: “Improving your crawl rate won't necessarily lead to better positions in Google Search results.” Crawling is a precondition for appearing in search, nothing more.

What Google officially says

Note that Google has moved its crawling documentation: the crawl budget guide now lives under developers.google.com/crawling/, and the old Search Central URLs redirect there.

Do you need to worry about crawl budget?

Usually not. Google's guide is “intended primarily” for three kinds of sites:

• “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”
• “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”
• “Sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed”

Google immediately qualifies the numbers: “These are not exact thresholds.” They are a rough way to classify your site, not a line you cross. Size alone also does not decide how often you are crawled: “If a site has important content that changes often, we crawl it often, regardless of the size.”

The Crawl Stats report help page puts the lower bound plainly: “If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail.”

Signs that crawling really is the bottleneck on a large site: new or updated pages take days or weeks to be picked up, a growing share of URLs sits in “Discovered – currently not indexed”, the Crawl Stats report shows rising response times or server errors, or server logs show Googlebot spending most of its requests on parameter URLs instead of real pages. If none of these apply, look at content quality and internal linking first. The separate lesson on “Crawled” and “Discovered – currently not indexed” covers how to tell a crawling problem from a quality problem.

Crawl capacity limit (hostload)

“Google wants to crawl your site without overwhelming your servers.” To do that, its crawlers calculate a crawl capacity limit, which Google also calls hostload. It limits the total time your server spends holding connections open for Google, taking into account both the number of parallel connections and how long they last.

“Every site starts with the same default, conservative crawl capacity limit.” If there is demand to crawl more and the site stays healthy, Google raises it over time. The limit moves with two factors:

• Crawl health. If response times, including latency and Time to First Byte, stay stable or improve, the limit goes up. If the site slows down, or answers with server errors (5xx) or rate-limiting signals such as 429, “the limit goes down and Google crawls less.”
• Google's own limits. Google's resources are finite and are shared across the whole web.

One detail is easy to miss: “While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers.” Googlebot, AdsBot, the shopping crawler and the rest draw on the same capacity. A spike from one of them, for example AdsBot after you add many Dynamic Search Ad targets, leaves less for the others.

If URL Inspection shows Hostload exceeded, Google could not crawl a URL because of your server capacity. Google's advice for that case is to add more server resources if that makes sense for your business.

Crawl demand

Each crawler has its own demand. AdsBot generally wants more when a site runs dynamic ad targets, and Google Shopping wants more for products in your merchant feeds. For Googlebot, demand depends on a site's size, update frequency, page quality and relevance compared with other sites. Google names three factors you can influence:

• Perceived inventory. “Without guidance from you, Google tries to crawl all or most of the URLs that it knows about on your site.” If many of those are duplicates or pages you do not care about, crawling time is wasted on them. Google calls this “the factor that you can positively control the most.”
• Popularity. “URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.”
• Staleness. “Our systems want to recrawl documents frequently enough to pick up any changes.”

Site-wide events also raise demand. A site move, for instance, makes Google recrawl content under the new URLs, which is why crawling usually jumps after a migration.

Demand is also why a fast server alone does not guarantee heavy crawling. A healthy site with static, weakly linked content may be crawled lightly simply because Google sees little reason to come back.

What wastes crawling

Wasted crawling means Google spends requests on URLs that will never be useful in search, while real pages wait. Google lists the usual sources: faceted navigation and session identifiers, duplicate content, soft 404 pages, hacked pages, infinite spaces and proxies, low quality and spam content, and shopping cart, infinite scrolling and action pages such as “sign up” or “buy now”.

What actually helps

Everything that works falls into two groups: show Google fewer useless URLs, and make each request cheaper for your server.

1. Consolidate duplicates. Redirect URL variants nobody needs, and use canonical URLs in links, sitemaps and rel="canonical" for variants that must stay.
2. Block URLs you never want crawled with robots.txt. Sort orders, internal search, session URLs, infinite calendars. Google: “Blocking URLs with robots.txt prevents Google from crawling them, and significantly decreases the chance the URLs will be processed by other Google systems”. Syntax and pitfalls are in the robots.txt lesson.
3. Return 404 or 410 for removed pages. “Google won't forget a URL that it knows about, but a 404 status code is a strong signal not to crawl that URL again.” Blocked URLs, by contrast, stay in the crawl queue much longer and are recrawled once the block is removed.
4. Fix soft 404s. Find them in the Page indexing report and return a real 404, 410 or a 301 to a true replacement.
5. Keep sitemaps clean. List only canonical, indexable URLs that return 200, and include every page you want crawled. “If your site includes updated content, we recommend including the <lastmod> tag.” Google adds that it “uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate.” The XML sitemaps lesson covers formats and limits.
6. Link to what matters. Internal links are how Google finds and prioritises URLs. Google notes that “pages linked directly to the home page may be seen as more important, and therefore crawled more often.” Link to final 200 URLs, not to redirects or parameter versions, and make sure important pages are not orphans.
7. Make the server fast. “If you can serve us more pages in a limited time, we will be able to crawl more of them.” Google also adds that speed for users matters more than speed for crawl coverage.
8. Support HTTP caching (304). “If a page hasn't changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources.” See the example below.
9. Reuse resource URLs. Reference a shared image or script “from the same URL in each page, so that Google can cache and reuse the same resource without needing to request the same resource multiple times.”

GET /shoes/running/ HTTP/1.1
Host: example.com

HTTP/1.1 200 OK
ETag: "a7f3c9"
Last-Modified: Mon, 26 Oct 2026 09:12:00 GMT
… full HTML body …

GET /shoes/running/ HTTP/1.1
Host: example.com
If-None-Match: "a7f3c9"

HTTP/1.1 304 Not Modified
(no body)

Google supports both validator pairs: ETag with If-None-Match, and Last-Modified with If-Modified-Since. If both are present, Google's crawlers use the ETag. Two caveats: “Google's crawlers don't send the headers with all crawl attempts”, and a Last-Modified date must use the standard HTTP date format. Most CDNs and web servers can handle conditional requests for static files out of the box. For dynamic pages, your CMS or application has to generate the validators and compare them.

What does not help

Many popular crawl budget “tricks” do nothing for Google, and some backfire.

• ✕crawl-delay in robots.txt“The non-standard "crawl-delay" robots.txt rule is not processed by Google's crawlers.” Other search engines may read it; Google ignores it.
• ✕noindex to save crawlingGoogle says “Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time.” Once pages leave the index, noindex can free a little budget indirectly in the long run, but it is a tool for the index, not for crawling.
• ✕Pinging sitemapsGoogle deprecated the sitemaps ping endpoint in 2023, and requests to it now return 404. Submit sitemaps in Search Console or list them in robots.txt instead.
• ✕Rotating robots.txt or sitemapsBlocking sections for a while to “move” budget elsewhere does not work. Google “won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit.” Google also says “Don't rotate sitemaps or use other temporary hiding mechanisms to reallocate budget.”
• ✕Fake freshnessChanging dates or lastmod without real changes adds nothing. Google sees “no additional value in making pages artificially appear to be fresh by making trivial changes and updating the page date.”
• ✕Compressing sitemapsGoogle states that compressed sitemaps do not increase crawl budget: they still have to be fetched.
• ✕nofollow on internal linksA nofollowed URL “can still be crawled if another page on your site, or any page on the web, doesn't label the link as nofollow.” Fix the URLs instead of hiding links to them.
• ✕<priority> and <changefreq>“Google ignores <priority> and <changefreq> values.”
• ✕Asking Google for more“You cannot request an increase in crawl rate”. The only levers are more server capacity and content Google wants to crawl.

Reducing the crawl rate in an emergency

Sometimes the problem is the opposite: Google crawls so hard that your server struggles or your hosting bill spikes. Before you throttle anything, find the cause. Google lists the most common ones as faceted navigation or other filtering, “A calendar with a lot of URLs for specific dates”, and Dynamic Search Ad targets. Check your server logs and the Crawl Stats report (Googlebot type) to see which crawler and which URL patterns are responsible. A spike right after you unblock or add a large section of the site is expected.

If you need relief now, Google's documented method is to serve errors for a short time:

When Google sees many of these codes, it slows down across the whole hostname, including URLs that still return content, and it speeds up again automatically once the errors drop. With 503 or 429 you can add a Retry-After header, either as a delay in seconds or as an exact date and time:

HTTP/1.1 503 Service Unavailable
Retry-After: 120

HTTP/1.1 429 Too Many Requests
Retry-After: Fri, 30 Oct 2026 18:00:00 GMT

Two alternatives exist. You can block the overloading crawler in robots.txt, but the Crawl Stats help notes this “can take up to a day to take effect” and should not stay in place for long. If serving errors is not possible on your infrastructure, you can file a special request with Google to report an unusually high crawl rate, stating the rate that suits your site; Google says evaluating it may take several days. If your setup can detect load dynamically, Google suggests returning 503 or 429 only when you are nearing your serving limit, and for no more than two or three days.

Reading the Crawl Stats report

“The Crawl Stats report shows you statistics about Google's crawling history on your website.” Open it in Search Console under Settings → Crawl stats. It works only for root-level properties: a Domain property, or a URL-prefix property at the root such as https://example.com. Google describes it as a report for advanced users.

1. Totals and the chart. Total crawl requests, total download size and average response time over the period. Google says the crawl rate “should be relatively stable over the time span of a week or two”. A sudden drop often follows a broad new robots.txt rule, slower responses or more server errors. A spike often follows unblocking or adding a large section, or new Dynamic Search Ad targets. Watch average response time: when it climbs, the capacity limit tends to fall.
2. Host status. Covers three categories: robots.txt fetching, DNS resolution and server connectivity. The status tells you whether Google saw no significant availability issues in the past 90 days, an issue more than a week ago, or an issue within the last week. Each category has a chart with a dotted threshold line; for example, DNS failing for more than 5% of requests on a given day counts as an issue. A robots.txt file that returns 429 or 5xx is treated as unavailable and can make Google slow or stop crawling; a 404 for robots.txt is fine.
3. By response. “In normal circumstances, the vast majority of responses should be 200 responses.” 301, 302, 304 and meta refreshes are listed as good responses, and 404s are often correct. Fix server errors (5xx), unauthorised responses (401/407), timeouts, DNS errors and redirect errors. Percentages are per request, not per URL.
4. By file type. Shows what Google is fetching. If images or JSON dominate, ask whether those requests are needed, for example many small images that should be blocked, or resources hosted on a slower server. Redirects are counted under “Other file type”.
5. By purpose. Discovery means “The URL requested was never crawled by Google before.” Refresh is “A recrawl of a known page.” After you publish a lot of new content or submit a sitemap, you should see a bump in discovery crawls.
6. By Googlebot type. Smartphone, desktop, image, video, page resource load, AdsBot, StoreBot and others. When crawling spikes, this is where you see which crawler caused it. Google notes that AdsBot crawls URLs about every two weeks.

Keep the data quirks in mind. “All URLs shown and counted are the actual URLs requested by Google”, not canonical URLs, and each hop in a redirect chain counts as a separate request. Requests between subdomains are visible from the parent domain property, data covers both http and https, and example URLs are only a sample. Google also notes that some requests may not be counted, so small differences from your server logs are expected. If the report shows far more requests than your logs, robots.txt may have been unavailable: Google counts crawls it would have made but did not.

The report shows patterns, not every URL. To see exactly which URL patterns eat requests, combine it with server logs filtered to verified Googlebot requests, and group them by folder and parameter.

Common mistakes

• ✕Optimising crawl budget on a small siteOn small sites, slow indexing is almost always about quality, links or technical errors, not budget.
• ✕Using noindex to “save” crawlingnoindex pages are still requested. Use robots.txt for URLs that should never be crawled.
• ✕Blocking URLs you want removed from the indexA blocked page cannot show its noindex. Let Google crawl it until it drops out, or return 404/410.
• ✕Blocking CSS and JavaScript to save requestsGoogle needs them to render pages; blocking them can break how pages are understood.
• ✕Leaving soft 404s on discontinued productsEmpty pages with status 200 keep being crawled.
• ✕Internal links to redirects and parametersEvery menu link to an old URL costs an extra hop on every crawl.
• ✕Sitemaps full of non-canonical URLsRedirects, 404s and noindex pages in the sitemap tell Google to crawl what you do not want.
• ✕Serving 503 for daysShort emergencies only; longer than 1–2 days risks pages dropping from the index.
• ✕Relying on crawl-delay or sitemap pingsGoogle ignores the first and retired the second.

FAQ

Sources

• Google — Optimize your crawl budget
• Google — Reduce the Google crawl rate
• Google Search Console Help — Crawl Stats report
• Google — Myths and facts about crawling
• Google Search Central — Troubleshoot Google Search crawling errors
• Google — Overview of Google crawlers and fetchers (HTTP caching)
• Google Search Central — Build and submit a sitemap
• Google Search Central Blog — Sitemaps ping endpoint is going away

Quotes checked against the official pages on 30 October 2026.