Examples
- A browser:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/… Safari/537.36 - Googlebot: its strings contain
Googlebot; the smartphone crawler also identifies as a mobile browser. - Bing: its strings contain
bingbot. - AI crawlers: for example
GPTBot(OpenAI) andClaudeBot(Anthropic).
How user agents are used in SEO
- Robots.txt rules are written per user agent, such as
User-agent: Googlebot, to allow or block specific crawlers. - Server logs show which crawlers visit which pages, and how often.
- Testing: you can switch the user agent in Chrome’s developer tools to see how a page responds to Googlebot or a phone.
Good to know
User agent strings are easy to fake, so a visitor claiming to be Googlebot may not be. Google and Bing let you verify their crawlers with a reverse DNS lookup or their published IP ranges. Serving crawlers different content from users based on the user agent is cloaking.
User agent example
A server log has a line from a request whose user agent contains “Googlebot/2.1” and an address that does not belong to Google. A reverse DNS lookup shows it is not a real Google crawler, so the site owner blocks that address at the firewall. A second request, with a user agent containing “bingbot” and an address that verifies as Microsoft’s, is allowed and indexed.
Robots.txt example
Rules are grouped by user agent: User-agent: * applies to all crawlers that follow the rules, while User-agent: GPTBot followed by Disallow: / asks that one AI crawler to stay out of the whole site. Well-behaved crawlers obey these rules, others may ignore them.
User agent checklist
- Use your server logs to see which crawlers visit and how often.
- Verify claims to be Googlebot or Bingbot with a reverse DNS lookup.
- Decide which AI crawlers you want to allow, and write the rules down.
- Do not serve different content to different user agents, since that is cloaking.
- Test your pages as Googlebot with the URL Inspection tool.
User agent mistakes
- Trusting a user agent string as proof of identity.
- Blocking all unknown bots and shutting out useful ones.
- Using user agent sniffing for design, instead of responsive design.
User agent FAQs
Can I change my user agent?
Yes, in browser developer tools, to test how a site responds. See crawlers and robots.txt.
Where is it shown?
In server logs and analytics. See Bingbot and IP addresses.