What a robots.txt file looks like
Each group of rules names a crawler (the user agent) and lists the paths it may not crawl:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xmlWhat robots.txt can and can’t do
- It controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. To keep a page out of the index, use a noindex meta tag and let the page be crawled.
- It is not a security measure. The file is public, and badly behaved bots can ignore it.
- Blocking CSS or JavaScript files can stop Google from rendering your pages properly.
- You can use it to block AI crawlers, such as GPTBot, if you don’t want your content used for AI training.
Test changes carefully: one wrong Disallow: / line can block your whole site from search.
Common robots.txt rules
| Goal | Rule |
|---|---|
| Allow everything | User-agent: * and an empty Disallow: |
| Block a folder | Disallow: /private/ |
| Block one crawler | A group for that user agent, such as User-agent: GPTBot then Disallow: / |
| Point to the sitemap | Sitemap: https://www.example.com/sitemap.xml |
Rules to remember
- The file must sit at the root of the host (example.com/robots.txt). Each subdomain needs its own file.
- Google reads about the first 500 KiB of the file.
- Google does not support a
noindexrule or a crawl delay in robots.txt. - A page blocked in robots.txt cannot show its noindex tag, because the crawler cannot read it.
- Rules are case-sensitive for paths.
Robots.txt and AI crawlers
Search and AI crawlers read robots.txt, so a blanket block can also keep you out of AI answers and search features that rely on those crawlers. Decide which bots you want, name them explicitly and check any CDN or firewall rule that may block them without your knowing. Our guide to answer engine optimization covers this.
How to test robots.txt
- Open example.com/robots.txt in a browser and read it.
- Use the robots.txt report in Google Search Console and the URL Inspection tool to see whether a page is blocked.
- Test changes on a staging site first.
Robots.txt FAQs
Does robots.txt remove a page from Google?
No. It blocks crawling, and the URL can still appear if other pages link to it. Use noindex or remove the page.
Do I need a robots.txt file?
Not always. Without one, crawlers may crawl everything. Most sites use one to block admin areas and point to the sitemap.