Why bot herding matters
Search engines crawl a limited number of URLs on each site in a given period, often called the crawl budget. On large websites, such as online shops with filters and sorting options, crawlers can spend that budget on thousands of near-identical URLs and reach important pages late or not at all.
Common bot herding techniques
- Internal linking: link often and clearly to key pages, and keep them only a few clicks from the home page.
- Robots.txt: block crawling of URLs with no search value, such as internal search results or endless filter combinations.
- Canonical tags: point duplicate versions of a page to the main version.
- XML sitemaps: list the URLs you want crawled and indexed.
- Clean-up: fix broken links, redirect chains and orphan pages.
Adding nofollow to your own internal links is not a reliable way to steer crawlers; the methods above work better. To see where crawlers actually spend their time, check the Crawl Stats report in Google Search Console and your server log files.
Bot herding example
An online shop with 2,000 products has 60,000 URLs because every combination of size, colour and sort order has its own address. Google crawls many of them and visits new products slowly. The shop adds canonical tags to the filtered versions, blocks the sort-order URLs in robots.txt and links to its best-selling categories from the home page. Within weeks the server log shows more crawl activity on product pages.
How to herd bots: a short plan
- Export crawl data from your server logs or Search Console to see what is being crawled.
- Find URLs that get crawled a lot but should never rank, such as internal search results and endless filters.
- Block or consolidate them with robots.txt rules and canonical tags.
- Strengthen internal links to the pages you want crawled first.
- Check the logs again after a few weeks.
Bot herding mistakes
- Blocking a page in robots.txt that you also want to be deindexed, since Google then cannot see the noindex tag.
- Blocking CSS or JavaScript files that pages need to display.
- Worrying about crawl budget on a small site, where it is rarely a problem.
Bot herding FAQs
Is bot herding the same as crawl budget optimization?
They are close. See crawl budget for how crawl capacity and demand work.
Does it help small sites?
Mostly, no. A site with a few hundred pages is usually crawled fully. Good robots.txt rules and internal links are enough.