Crawl Budget, explained properly.
What is crawl budget?
Crawl budget is the number of pages a search engine, typically Google, is willing and able to crawl on your website within a set period. It combines two ideas: crawl rate limit and crawl demand. If your site exceeds this budget, some pages may not get crawled or indexed as often, or at all.
Google defines it as the number of URLs Googlebot can and wants to crawl. Google's own guidance says it mainly matters for very large sites, roughly a million pages or more, or sites with 10,000 or more pages that change daily. For most small business sites it is not the constraint. For larger sites, especially ecommerce stores, blogs and marketplaces, inefficient use of crawl budget leads to wasted crawling, missed indexing and stagnant rankings.
Why crawl budget matters
Large sites often struggle with slow indexing, stale content in the results and unindexed products. Nine times out of ten, crawl budget is being wasted on:
- URL parameter variations from filters and sorts
- Out-of-stock or paginated content
- Duplicate or low-quality pages
- Broken links or redirect loops
This is why technical SEO is critical. If you want your most important pages crawled frequently and indexed correctly, you need to optimise your crawl budget.
How Google determines crawl budget
1. Crawl rate limit
This controls how many requests Googlebot can make without overwhelming your server. If your site slows down or returns errors, Google may crawl less aggressively.
2. Crawl demand
This is based on how important Google thinks your pages are, which depends on popularity, freshness, internal linking and how often content updates.
So crawl budget is not fixed; it fluctuates with your site's health and SEO signals.
Signs you are wasting crawl budget
- Pages take weeks or months to appear in search
- Google Search Console shows "Discovered, currently not indexed"
- Server logs show frequent crawling of unimportant URLs
- You have thousands of pages but few are ranking
If that sounds familiar, a full SEO audit with log file analysis can diagnose the issue.
How to improve crawl budget
1. Clean up low-value URLs
Use robots.txt to stop crawlers requesting URLs you never want crawled, such as internal search results or endless filter combinations. Do not rely on noindex to save crawl budget: Google still has to request the page to see the tag. Canonical tags help Google pick the right version of duplicates, but the duplicates still get crawled.
2. Fix redirect chains
Redirect chains slow crawling and waste resources. Fixing them improves efficiency and page authority flow.
3. Improve internal linking
Pages that are not linked internally are hard for bots to find. An internal link map prioritises important pages and ensures none are orphaned.
4. Consolidate thin or duplicate content
Trim or merge underperforming pages, and canonicalise duplicates to a single, stronger URL.
5. Improve site speed and server health
If Googlebot detects server errors or sluggish responses, it slows its crawl. Solid hosting, CDN and caching keep it moving.
6. Use XML sitemaps effectively
Submit sitemaps that include only your index-worthy URLs. Every page thrown in just clutters the crawl.
Tools to monitor and optimise crawl budget
- Google Search Console: check crawl stats, coverage and indexing status.
- Screaming Frog: identify duplicate pages, thin content and redirect chains.
- Server logs: see which URLs bots are actually hitting.
- Sitebulb: visualise crawl paths and prioritise pages.
Crawl budget and site size
- Under 10,000 URLs: crawl budget is almost never the problem. If pages are not being indexed, look at quality and internal linking first.
- 10,000+ URLs that change daily: crawl budget can start to matter.
- Around a million URLs or more: you need a crawl budget strategy.
The catch is that faceted filters on a Shopify or WooCommerce store can turn a few hundred products into hundreds of thousands of crawlable URLs. The number that counts is how many URLs bots can reach, not how many products you sell.
What crawl budget does not affect
- How well your page ranks: that is a content and backlink game.
- Whether Google discovers your site: that is more about links and sitemaps.
It governs how many pages Googlebot tries to visit in a timeframe, which on a very large site also means how quickly changes to buried pages get picked up. If your best pages are not crawled, that is a real problem.
Crawl Budget: common questions.
Does crawl budget matter for small sites?
Rarely. Google aims its crawl budget guidance at sites with around a million or more pages, or 10,000 or more pages that change daily. Below that, Google usually crawls everything it wants to. The exception is ecommerce with faceted navigation, where filters can create far more URLs than you have products.
How do I improve crawl budget efficiency?
Three approaches. Block low-value URLs in robots.txt (filter combinations, internal search results, admin pages); noindex does not help here, because Google still crawls the page to read the tag. Fix server response times, since Google reduces crawl rate when servers are slow. Improve internal linking so important pages are easy to discover; orphaned pages waste crawl signal.
Where do I see crawl budget data?
Google Search Console's Crawl Stats report (Settings, then Crawl Stats). It shows total crawl requests, average response time, and a breakdown by file type and response code. Spikes or drops reveal issues; steady patterns show healthy crawling.
Can I increase my crawl budget?
Indirectly. Improve site quality and authority, because Google crawls sites it considers important more frequently. Improve server performance, so faster responses let Google crawl more in the same window. Add new content regularly, which signals an active site worth checking.
“I've had the pleasure of working with Chris on several SEO projects for my websites in the health field. He was definitely been able to demystify things so that I felt able to make decisions with him and feel confident they were the right ones. For a non-techie, that is worth gold! I look forward to working with Chris again and warmly recommend him to anyone who - like me - is unsure about all things SEO.”
“I was recommended by my branding partner to use Chris and his team last year. I can’t recommend them highly enough. They are very responsive , easy to communicate with and great at translating techy stuff when needed. They are now an essential part of my business model”
“We've been working closely with Chris, Hannah and the team for over 12 months now. As a small company with ambitious growth plans, it was incredibly important that we made the right choice on who to trust with our SEO.<br><br>18 months later, we're totally thrilled with our experience so far and our growing online presence within our market. We'd happily recommend BTT and their services to other SMEs.<br><br>Drop them an enquiry!”
You work directly with the senior people who do the work, a small Bristol team you will actually get to know, not a call centre or an account manager relaying messages.
Meet the team →





