SEO glossary · C

Crawler

What is a Crawler?

A crawler, also called a spider or bot, is a program that search engines use to find and read web pages. It fetches a page, follows the links on it to find more pages, and passes what it finds on so the pages can be indexed.

Well-known crawlers

  • Googlebot: Google’s crawler, which mainly crawls as a smartphone.
  • Bingbot: Microsoft’s crawler for Bing.
  • AI crawlers: such as GPTBot (OpenAI) and ClaudeBot (Anthropic), which collect content for AI models and assistants.
  • SEO crawlers: tools like Screaming Frog or Sitebulb that crawl your own site to find technical problems.

Helping crawlers find your pages

  • Link to every important page from other pages on your site.
  • Submit an XML sitemap in Google Search Console and Bing Webmaster Tools.
  • Use robots.txt to keep crawlers out of areas with no search value.
  • Keep the site fast and free of server errors, so crawlers can fetch more pages.

Crawlers identify themselves with a user agent name, which you can see in your server logs.

Crawler example

A new page is linked from the blog home page. Googlebot visits the home page, finds the link, fetches the new page, reads its text and links, and adds it to the list of URLs to process. Some days later the page appears in the index. If the page had no internal links and was not in the sitemap, the crawler might not have found it at all.

How to see what crawlers do on your site

  • Check the Crawl stats report in Search Console for Google’s activity.
  • Read the server log to see every visit, with the user agent and response code.
  • Run your own crawl with an SEO crawler tool to see the site as a bot sees it.
  • Use URL Inspection to see a single page as Google rendered it.

Crawler mistakes

  • Blocking all crawlers in robots.txt during development and forgetting to remove the rule at launch.
  • Hiding important content behind forms, logins or scripts that crawlers cannot run.
  • Blocking AI crawlers by mistake, or allowing all without deciding. Review your policy.
  • Believing every request that says “Googlebot” is real: verify with a reverse DNS lookup.

Crawler FAQs

Is a crawler the same as an indexer?

No. The crawler fetches pages. Indexing is the separate step of processing and storing them. See index page.

How do I control a crawler?

With robots.txt, meta robots tags and server settings. See Bingbot and user agent.

Get in touch

Know the term. Now rank for it.

Tell us the search you want to win. A specialist reads it and replies by email, and if it helps, we set up a short call.

    A specialist reads your message and replies by email.

    Protected by reCAPTCHA: Google's Privacy Policy and Terms of Service apply.