Crawl budget is real, widely misdiagnosed, and irrelevant to most sites. Google will happily crawl a few hundred pages regardless of what you do; it cannot crawl five million filter combinations no matter what you do. Here is when budget genuinely bites, how to read the evidence, and the optimizations that redirect crawler attention to pages that earn money.
“crawl budget” — 260 searches/mo (Google, US)Average monthly US search volume. Source: Google autocomplete demand via AnswerThePublic, 2026.
Where budget actually comes from
Two curves intersect. Crawl capacityis how hard Google will hit your server without hurting it — it rises with fast, healthy responses and falls with slowness and 5xx errors. Crawl demandis how much Google wants your URLs — popular, frequently-updated content gets fetched often; stale corners get visited rarely. Your budget is whichever curve is lower. This is why the two best “optimizations” are unglamorous: a fast server and content worth re-fetching.
The problem is almost always URL inflation
Sites do not usually run out of budget; they manufacture URLs faster than any budget could cover. Faceted navigation is the classic engine — every filter × sort × page combination is a fetchable URL, so a 500-product store presents tens of thousands of near-duplicates. Internal search results, calendar pages, tracking parameters, and session IDs all do the same quieter. The crawler spends its day re-reading permutations while your new product page waits in the indexing queue.
The optimization playbook
Do first — high impact, many pages
- Block infinite spaces (search, calendars, params)
- Canonicalise faceted duplicates
- Fix 5xx errors and slow responses
Schedule — high impact, fewer pages
- Flatten redirect chains
- Prune or consolidate thin pages
- Review Crawl Stats monthly
Quick wins — low effort
- Sitemap with honest lastmod dates
- Fix broken internal links
- Indexing pings on publish/update
Deprioritise — low impact
- Crawl budget work on sites under ~10k pages
- Crawl-delay directives
- Blocking CSS/JS to "save" crawls
The supporting cast matters: a truthful sitemap with real lastmod dates lets Google skip unchanged pages instead of re-verifying them; robots.txt walls off the infinite spaces; canonicalsconsolidate what filters split. And that last deprioritised item is a live footgun — blocked CSS and JavaScript do not save budget, they break rendering.
Wondering where your crawls go? Run the free audit — redirect chains, broken links, and duplicate clusters are exactly what it maps.
Frequently asked questions
- What is crawl budget in SEO?
- Crawl budget is the number of URLs a search engine will fetch from your site in a given period. It emerges from two limits: crawl capacity (how hard Google will hit your server without degrading it) and crawl demand (how much Google wants your content, based on popularity and update frequency). When your crawlable URLs exceed the budget, some pages wait — sometimes indefinitely.
- Does crawl budget matter for my site?
- Google’s own guidance: it concerns sites with over roughly a million pages that change daily, or over ten thousand pages changing very frequently — plus any site whose faceted navigation or parameters generate URLs endlessly. A few hundred pages? Your issue is almost certainly indexing quality or directives, not budget.
- How do I check my crawl budget?
- Search Console’s Crawl Stats report shows requests per day, response times, and the split by purpose and file type. The diagnostic ratio: crawls per day against URLs you actually want indexed. If Google fetches 5,000 URLs daily but coverage of your product pages lags for weeks, the budget exists — it is being spent on the wrong URLs.
- How do I optimize crawl budget?
- Stop manufacturing URLs: control faceted navigation and parameters, canonicalise duplicates, and noindex or block the infinite spaces (calendars, internal search). Then make crawling cheaper and better-directed: fast responses, accurate sitemaps with real lastmod dates, flattened redirect chains, fixed broken internal links, and indexing pings so engines fetch changed pages instead of re-checking unchanged ones.
- Does page speed affect crawl budget?
- Directly. Crawl capacity adapts to your server: fast, healthy responses invite more parallel fetching, while slow responses and 5xx errors make Google back off to protect the site. Speed work therefore pays twice — rankings via Core Web Vitals, and coverage via a crawler that can afford more of your pages.
