How to recognize Bingbot
Bingbot identifies itself in your server logs with a user agent that contains “bingbot”. Because anyone can fake a user agent, you can check that a visit really came from Bing with the Verify Bingbot tool in Bing Webmaster Tools or a reverse DNS lookup.
Controlling how Bingbot crawls
- Add robots.txt rules for
User-agent: bingbotto keep it out of sections you don’t want crawled. - Set crawl rate preferences in Bing Webmaster Tools if crawling puts load on your server.
- Use IndexNow to tell Bing and other participating search engines straight away when you add, change or delete a page.
Bingbot example
A site owner sees “bingbot” in the server log requesting 3,000 URLs a day, many of them filter URLs from a product list. The owner blocks the filter parameters in robots.txt, keeps the category pages open and lists the main products in the sitemap. A few weeks later the log shows Bingbot fetching the pages that matter, and fewer duplicate URLs.
Bingbot checklist
- Make sure robots.txt does not block Bingbot from pages you want in the index.
- Keep the sitemap current and error-free.
- Check server logs for 5xx errors and slow responses to Bingbot.
- Verify suspicious “Bingbot” visits with a reverse DNS lookup before you block or allow them.
- Use IndexNow for fast updates on busy sites.
Bingbot mistakes
- Blocking all bots in robots.txt during a redesign and forgetting to remove the rule.
- Trusting the user agent alone, since it can be faked.
- Setting a very low crawl rate and delaying indexing of new pages.
Bingbot FAQs
Is Bingbot different from Googlebot?
They are separate crawlers for separate search engines, with similar rules. Read about the general idea of a crawler.
Where can I see what Bingbot does on my site?
In Bing Webmaster Tools and in your server logs.