BrisTechTonic
GlossaryCrawl Budget

Crawl Budget

Crawl budget is the number of pages Google's bots will crawl on your site within a given period. For sites under about 10,000 URLs it is rarely the constraint. For larger sites, directing bots to important pages and away from low-value ones can meaningfully improve indexing and ranking.

Experienced3-month minimum, then 30-day rollingFree discovery call to start
In short
Crawl budget combines crawl rate (how hard Google can hit your server) and crawl demand (how important Google thinks your pages are).
Google says it mainly matters for very large sites, around a million pages, or sites with 10,000+ pages changing daily. Filter URLs can push a store into that range.
Wasted crawls on parameter and filter URLs can stop important pages being indexed.
Crawl budget does not affect how well a page ranks, only how many pages Googlebot tries to visit.
Book a call →
5.0109 Google reviewsElite Business Top 100, 2025 finalistExperienced Bristol team since 2019Hundreds of UK businesses helped
In depth

Crawl Budget, explained properly.

•Crawl budget combines crawl rate (how hard Google can hit your server) and crawl demand (how important Google thinks your pages are).
•Google says it mainly matters for very large sites, around a million pages, or sites with 10,000+ pages changing daily. Filter URLs can push a store into that range.
•Wasted crawls on parameter and filter URLs can stop important pages being indexed.
•Crawl budget does not affect how well a page ranks, only how many pages Googlebot tries to visit.

What is crawl budget?

Crawl budget is the number of pages a search engine, typically Google, is willing and able to crawl on your website within a set period. It combines two ideas: crawl rate limit and crawl demand. If your site exceeds this budget, some pages may not get crawled or indexed as often, or at all.

Google defines it as the number of URLs Googlebot can and wants to crawl. Google's own guidance says it mainly matters for very large sites, roughly a million pages or more, or sites with 10,000 or more pages that change daily. For most small business sites it is not the constraint. For larger sites, especially ecommerce stores, blogs and marketplaces, inefficient use of crawl budget leads to wasted crawling, missed indexing and stagnant rankings.

Why crawl budget matters

Large sites often struggle with slow indexing, stale content in the results and unindexed products. Nine times out of ten, crawl budget is being wasted on:

  • URL parameter variations from filters and sorts
  • Out-of-stock or paginated content
  • Duplicate or low-quality pages
  • Broken links or redirect loops

This is why technical SEO is critical. If you want your most important pages crawled frequently and indexed correctly, you need to optimise your crawl budget.

How Google determines crawl budget

1. Crawl rate limit

This controls how many requests Googlebot can make without overwhelming your server. If your site slows down or returns errors, Google may crawl less aggressively.

2. Crawl demand

This is based on how important Google thinks your pages are, which depends on popularity, freshness, internal linking and how often content updates.

So crawl budget is not fixed; it fluctuates with your site's health and SEO signals.

Signs you are wasting crawl budget

  • Pages take weeks or months to appear in search
  • Google Search Console shows "Discovered, currently not indexed"
  • Server logs show frequent crawling of unimportant URLs
  • You have thousands of pages but few are ranking

If that sounds familiar, a full SEO audit with log file analysis can diagnose the issue.

How to improve crawl budget

1. Clean up low-value URLs

Use robots.txt to stop crawlers requesting URLs you never want crawled, such as internal search results or endless filter combinations. Do not rely on noindex to save crawl budget: Google still has to request the page to see the tag. Canonical tags help Google pick the right version of duplicates, but the duplicates still get crawled.

2. Fix redirect chains

Redirect chains slow crawling and waste resources. Fixing them improves efficiency and page authority flow.

3. Improve internal linking

Pages that are not linked internally are hard for bots to find. An internal link map prioritises important pages and ensures none are orphaned.

4. Consolidate thin or duplicate content

Trim or merge underperforming pages, and canonicalise duplicates to a single, stronger URL.

5. Improve site speed and server health

If Googlebot detects server errors or sluggish responses, it slows its crawl. Solid hosting, CDN and caching keep it moving.

6. Use XML sitemaps effectively

Submit sitemaps that include only your index-worthy URLs. Every page thrown in just clutters the crawl.

Tools to monitor and optimise crawl budget

  • Google Search Console: check crawl stats, coverage and indexing status.
  • Screaming Frog: identify duplicate pages, thin content and redirect chains.
  • Server logs: see which URLs bots are actually hitting.
  • Sitebulb: visualise crawl paths and prioritise pages.

Crawl budget and site size

  • Under 10,000 URLs: crawl budget is almost never the problem. If pages are not being indexed, look at quality and internal linking first.
  • 10,000+ URLs that change daily: crawl budget can start to matter.
  • Around a million URLs or more: you need a crawl budget strategy.

The catch is that faceted filters on a Shopify or WooCommerce store can turn a few hundred products into hundreds of thousands of crawlable URLs. The number that counts is how many URLs bots can reach, not how many products you sell.

What crawl budget does not affect

- How well your page ranks: that is a content and backlink game.
- Whether Google discovers your site: that is more about links and sitemaps.
It governs how many pages Googlebot tries to visit in a timeframe, which on a very large site also means how quickly changes to buried pages get picked up. If your best pages are not crawled, that is a real problem.

How BrisTechTonic helps
Technical SEO →full SEO audit →SEO consultancy →SEO strategy service →
Common questions

Crawl Budget: common questions.

Does crawl budget matter for small sites?

Rarely. Google aims its crawl budget guidance at sites with around a million or more pages, or 10,000 or more pages that change daily. Below that, Google usually crawls everything it wants to. The exception is ecommerce with faceted navigation, where filters can create far more URLs than you have products.

How do I improve crawl budget efficiency?

Three approaches. Block low-value URLs in robots.txt (filter combinations, internal search results, admin pages); noindex does not help here, because Google still crawls the page to read the tag. Fix server response times, since Google reduces crawl rate when servers are slow. Improve internal linking so important pages are easy to discover; orphaned pages waste crawl signal.

Where do I see crawl budget data?

Google Search Console's Crawl Stats report (Settings, then Crawl Stats). It shows total crawl requests, average response time, and a breakdown by file type and response code. Spikes or drops reveal issues; steady patterns show healthy crawling.

Can I increase my crawl budget?

Indirectly. Improve site quality and authority, because Google crawls sites it considers important more frequently. Improve server performance, so faster responses let Google crawl more in the same window. Add new content regularly, which signals an active site worth checking.

More terms
Alt TextBacklinksCanonical TagClick-Through Rate (CTR)Conversion Rate Optimisation (CRO)Core Web VitalsCumulative Layout Shift (CLS)Digital PRDomain AuthorityE-E-A-TGoogle Tag Manager (GTM)Interaction to Next Paint (INP)Knowledge GraphLargest Contentful Paint (LCP)llms.txtMeta DescriptionROAS (Return on Ad Spend)Schema markupSEO (Search Engine Optimisation)Technical SEOTime to First Byte (TTFB)XML SitemapZero-Click Search
In their words
What clients say about working with us.
5.0 average · 109 reviews

“I've had the pleasure of working with Chris on several SEO projects for my websites in the health field. He was definitely been able to demystify things so that I felt able to make decisions with him and feel confident they were the right ones. For a non-techie, that is worth gold! I look forward to working with Chris again and warmly recommend him to anyone who - like me - is unsure about all things SEO.”

Biliana Avramova

“I was recommended by my branding partner to use Chris and his team last year. I can’t recommend them highly enough. They are very responsive , easy to communicate with and great at translating techy stuff when needed. They are now an essential part of my business model”

Sandra Webber

“We've been working closely with Chris, Hannah and the team for over 12 months now. As a small company with ambitious growth plans, it was incredibly important that we made the right choice on who to trust with our SEO.<br><br>18 months later, we're totally thrilled with our experience so far and our growing online presence within our market. We'd happily recommend BTT and their services to other SMEs.<br><br>Drop them an enquiry!”

Dan H
Read all 109 reviews →
The team
Real people on your account, not a faceless agency.

You work directly with the senior people who do the work, a small Bristol team you will actually get to know, not a call centre or an account manager relaying messages.

Meet the team →
Chris McDowell, Founder & CEO at BrisTechTonicDominique, Fractional COO at BrisTechTonicHayley, Senior SEO Strategist at BrisTechTonicGenevieve, Client Services Manager at BrisTechTonicHannah Gooding, Senior Marketing Executive/Copywriter at BrisTechTonicOlivia, Senior Marketing Executive/Designer at BrisTechTonicAmelia Cheek, Marketing Executive at BrisTechTonic
Chris, Dominique, Hayley, Genevieve and the rest of the team

Now see how your own site handles crawl budget.

Get a free check →Or book a call