Why bot herding matters
Search engines crawl a limited number of URLs on each site in a given period, often called the crawl budget. On large websites, such as online shops with filters and sorting options, crawlers can spend that budget on thousands of near-identical URLs and reach important pages late or not at all.
Common bot herding techniques
- Internal linking: link often and clearly to key pages, and keep them only a few clicks from the home page.
- Robots.txt: block crawling of URLs with no search value, such as internal search results or endless filter combinations.
- Canonical tags: point duplicate versions of a page to the main version.
- XML sitemaps: list the URLs you want crawled and indexed.
- Clean-up: fix broken links, redirect chains and orphan pages.
Adding nofollow to your own internal links is not a reliable way to steer crawlers; the methods above work better. To see where crawlers actually spend their time, check the Crawl Stats report in Google Search Console and your server log files.