Does crawl budget matter for your site?
For most websites, no. Google says it mainly matters for large sites, with around a million pages or more, and for sites with 10,000 or more pages that change every day. If your new pages are usually crawled within a few days, you don’t need to worry about it.
What wastes crawl budget
- Endless URL variations from filters, sorting options and tracking parameters
- Duplicate pages and thin, low-value pages
- Redirect chains, broken links and soft 404 pages
- A slow server, which makes Google crawl less
How to optimize crawl budget
- Block crawling of URLs with no search value in robots.txt.
- Use canonical tags for duplicates, and remove or merge thin pages.
- Keep your XML sitemap up to date with the URLs you want indexed.
- Link clearly to your important pages and fix redirect chains.
- Check the Crawl Stats report in Google Search Console and your server logs.