Does crawl budget matter for your site?
For most websites, no. Google says it mainly matters for large sites, with around a million pages or more, and for sites with 10,000 or more pages that change every day. If your new pages are usually crawled within a few days, you don’t need to worry about it.
What wastes crawl budget
- Endless URL variations from filters, sorting options and tracking parameters
- Duplicate pages and thin, low-value pages
- Redirect chains, broken links and soft 404 pages
- A slow server, which makes Google crawl less
How to optimize crawl budget
- Block crawling of URLs with no search value in robots.txt.
- Use canonical tags for duplicates, and remove or merge thin pages.
- Keep your XML sitemap up to date with the URLs you want indexed.
- Link clearly to your important pages and fix redirect chains.
- Check the Crawl Stats report in Google Search Console and your server logs.
Crawl budget example
A fashion shop has 5,000 products, but filters for size, colour, price and sort order produce more than 400,000 URLs. Search Console shows that most crawl requests go to filter URLs, while new products wait weeks to be crawled. The shop adds canonical tags, blocks the sort and price-range filters in robots.txt and removes broken links. New products are then crawled in days.
How to check your crawl activity
- Open Settings and the Crawl stats report in Google Search Console.
- Look at total requests, response codes and average response time.
- Check which file types and URL groups take the most requests.
- Compare with your server logs for a detailed view.
- Look for spikes of 404s or 5xx errors, which reduce crawling.
Crawl budget mistakes
- Spending time on crawl budget for a site with a few hundred pages.
- Using noindex to save budget. Google still has to crawl the page to see the tag.
- Blocking resources that pages need to display.
- Leaving long redirect chains and soft 404s in place.
Crawl budget FAQs
How can I get pages crawled faster?
Keep the server fast, link to important pages, keep the XML sitemap current and fix errors. See bot herding and crawlers.
Does crawl budget affect rankings?
Not directly. It affects how fast pages are found and refreshed. See robots.txt.