SEO glossary · C

Crawl Budget

What is Crawl Budget?

Crawl budget is the number of URLs a search engine can and wants to crawl on your website in a given period. It depends on how much crawling your server can handle without slowing down (crawl capacity) and how much the search engine wants to crawl your pages (crawl demand).

Does crawl budget matter for your site?

For most websites, no. Google says it mainly matters for large sites, with around a million pages or more, and for sites with 10,000 or more pages that change every day. If your new pages are usually crawled within a few days, you don’t need to worry about it.

What wastes crawl budget

  • Endless URL variations from filters, sorting options and tracking parameters
  • Duplicate pages and thin, low-value pages
  • Redirect chains, broken links and soft 404 pages
  • A slow server, which makes Google crawl less

How to optimize crawl budget

  • Block crawling of URLs with no search value in robots.txt.
  • Use canonical tags for duplicates, and remove or merge thin pages.
  • Keep your XML sitemap up to date with the URLs you want indexed.
  • Link clearly to your important pages and fix redirect chains.
  • Check the Crawl Stats report in Google Search Console and your server logs.

Crawl budget example

A fashion shop has 5,000 products, but filters for size, colour, price and sort order produce more than 400,000 URLs. Search Console shows that most crawl requests go to filter URLs, while new products wait weeks to be crawled. The shop adds canonical tags, blocks the sort and price-range filters in robots.txt and removes broken links. New products are then crawled in days.

How to check your crawl activity

  1. Open Settings and the Crawl stats report in Google Search Console.
  2. Look at total requests, response codes and average response time.
  3. Check which file types and URL groups take the most requests.
  4. Compare with your server logs for a detailed view.
  5. Look for spikes of 404s or 5xx errors, which reduce crawling.

Crawl budget mistakes

  • Spending time on crawl budget for a site with a few hundred pages.
  • Using noindex to save budget. Google still has to crawl the page to see the tag.
  • Blocking resources that pages need to display.
  • Leaving long redirect chains and soft 404s in place.

Crawl budget FAQs

How can I get pages crawled faster?

Keep the server fast, link to important pages, keep the XML sitemap current and fix errors. See bot herding and crawlers.

Does crawl budget affect rankings?

Not directly. It affects how fast pages are found and refreshed. See robots.txt.

Know the term. Now rank for it.

Tell us the search you want to win. A specialist reads it and replies by email, and if it helps, we set up a short call.

  1. We reply by emailA specialist reads your message and answers your questions.
  2. A short call, if usefulAbout your business, your market and your goals.
  3. A written proposalThe work, the goals, the reporting and our fee.

    A specialist reads your message and replies by email.