What Is Crawl Budget? (And Does It Affect Your Site?)

Crawl budget is the number of URLs Googlebot will fetch from your site in a given period. It's set by two things: how much crawling your server can absorb without slowing down, and how much Google actually wants to crawl you. For most sites under a few thousand pages, crawl budget is not the reason anything isn't ranking.

The two halves of the number

Google doesn't publish a crawl budget figure. It emerges from two independent limits, and knowing which one is binding tells you whether there's anything to fix.

Crawl capacity limit — how fast Google can crawl you. Googlebot watches your response times and error rates. Fast, healthy responses mean it crawls more; slow responses or a run of 5xx errors mean it backs off automatically to avoid hurting your server. This limit is about your infrastructure.

Crawl demand — how much Google wants to crawl you. This is driven by how popular your URLs are, how often they genuinely change, and how much of your site Google judges worth revisiting. A page nobody links to and that hasn't changed in three years gets crawled rarely. This limit is about your content and your links.

The distinction matters because the fixes are opposites. If capacity is binding, you have a hosting and performance problem. If demand is binding — which is far more common — buying a faster server changes nothing, because Google was never trying to crawl harder in the first place.

Does it affect your site? Probably not

This is the part most crawl-budget advice skips. Google's own guidance on managing crawl budget is explicit that it's a large-site concern, and names two rough thresholds: sites with more than a million unique pages where content changes weekly, and sites with more than 10,000 unique pages where content changes daily.

Below that, Google states that keeping your sitemap current and reviewing index coverage is enough. If your new posts get crawled within a day or two of publishing, crawl budget is not your bottleneck — and time spent optimising it is time not spent on the thing that is.

The honest test: publish a page, link to it internally, and see how long it takes to appear in Search Console. If it's hours or days, you have plenty of budget. If a page sits undiscovered for weeks and you have tens of thousands of URLs, then the conversation is worth having.

When a small site's pages aren't getting indexed, the cause is almost always something else — a blocker, or a quality judgement. Why is my website not showing up on Google? walks through the far more likely explanations.

What actually wastes it

On sites big enough to care, budget rarely disappears in one dramatic way. It leaks through URLs that exist by accident:

Faceted navigation. Filter and sort combinations on an ecommerce or listing page can generate millions of URLs from a few hundred products. This is the single biggest source of crawl waste on large sites.
Session IDs and tracking parameters in URLs, each creating a fresh address for identical content.
Infinite spaces — calendar widgets that will happily serve the year 3025, "load more" endpoints, endlessly paginated archives.
Soft 404s. Pages that return HTTP 200 with a "nothing here" message. Google crawls them repeatedly because the server says they're fine.
Redirect chains. Every extra hop is an extra fetch. See what a 301 redirect is for how to collapse them.
Duplicate URLs — trailing-slash, uppercase, www/non-www, and HTTP/HTTPS variants all resolving separately.
Genuinely low-value pages — thin tag archives, near-empty category pages, auto-generated author pages for authors with one post.

How to see your actual crawl data

Stop guessing. Two sources give you real numbers:

Search Console → Settings → Crawl Stats. Shows total crawl requests over 90 days, average response time, and a breakdown by response code, file type, and purpose (discovery vs refresh). A rising response time with falling requests is the signature of a capacity limit. Lots of requests returning 404s or 301s is the signature of waste.

Server log files. The unabridged version. Filter for verified Googlebot requests and you can see exactly which URLs it's spending fetches on. Most large sites that run this exercise for the first time discover a substantial share of Googlebot's visits going to parameter URLs nobody knew existed.

How to free up budget

In rough order of impact:

Block the infinite spaces in robots.txt. Faceted parameters, internal search results, calendar endpoints. A Disallow stops the crawl before it starts — see what a robots.txt file is for how to write the rules without accidentally blocking something you need.
Return honest status codes. Real 404s or 410s for gone pages, not 200s with an apology.
Fix redirect chains so every old URL points directly at its final destination.
Consolidate duplicates with canonical tags and a single URL convention.
Keep your sitemap clean — canonical, indexable URLs only, with accurate lastmod dates. A sitemap full of redirects and 404s trains Google to trust it less. Do I need an XML sitemap? covers what belongs in one.
Speed up server response. This raises the capacity ceiling — but only helps if capacity was the binding limit.

Note what's not on the list: noindex. A noindex page still has to be crawled for Google to see the tag, so it saves no budget at all. It's an indexing tool, not a crawling one.

The link angle nobody mentions

Crawl demand is partly a popularity measurement, and popularity is measured in links.

Googlebot discovers and re-crawls URLs by following links. A page with external links pointing at it gets crawled sooner and revisited more often than an identical page with none — which is why a brand-new domain with zero backlinks can sit undiscovered for weeks while an established site's new posts appear within hours.

That makes link building a crawl-frequency lever, not just a ranking one. You're not increasing an allowance so much as giving Google more reasons to come back. For a new site, one or two genuine inbound links from sites Google already crawls regularly is the difference between "eventually" and "this week." Backlinkster exists for that first step — you're matched with a site owner in a related niche, each of you places the other's link inside real content, and the placement is verified live by code rather than promised.

Frequently asked questions

Does crawl budget affect rankings? Not directly. Crawl budget determines whether Google sees your pages and how quickly it notices changes. A page that never gets crawled can't rank, but among pages that are crawled, budget isn't a ranking factor.

Do noindex pages waste crawl budget? Yes. Googlebot has to fetch the page to read the noindex directive, so the request is spent either way. Use robots.txt if the goal is to stop the crawl; use noindex if the goal is to keep it out of the index.

Does page speed affect crawl budget? It affects the crawl capacity limit. Faster server responses let Googlebot crawl more per session without risking your site's stability. It has no effect on crawl demand, so speeding up a site nobody links to won't get it crawled more.

How do I increase my crawl budget? You can't request more. You raise capacity by improving server response times, and you raise demand by earning links and publishing content worth revisiting. Removing crawl waste is usually more effective than either, because it redirects existing budget to pages that matter.

Does crawl budget apply to Bing and other engines? Bingbot works on similar principles and exposes a crawl control setting in Bing Webmaster Tools. AI crawlers such as GPTBot and PerplexityBot have their own independent budgets, which is one reason a site can be well-indexed by Google and largely unseen by AI search.

The bottom line

Crawl budget is real, measurable in Search Console, and irrelevant to the overwhelming majority of sites. If you have fewer than a few thousand pages and your new content gets crawled within days, the correct amount of crawl-budget optimisation is none. If you run a large ecommerce or listing site, the work isn't asking Google for more — it's finding the parameter URLs and infinite spaces quietly absorbing what you already have, and pointing that attention at pages you actually want ranked.