What is robots.txt?
robots.txt is a plain text file at the root of a site that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing: a blocked page can still appear in Google, just without a description. And it is a request, not a lock: well-behaved crawlers follow it, and anyone else can still open the URL. Of the 98 large sites in Check Engine's study, 9 had no robots.txt at all.
What it looks like
User-agent: *
Disallow: /cart/
Disallow: /search
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlEach group starts with one or more User-agent lines, naming the crawler, then the rules for it. The star means any crawler that isn't named elsewhere. The file must sit at the top of the host, as /robots.txt, and it covers only that host, protocol and port. A robots.txt on www.example.com says nothing about shop.example.com.
The lines Google reads
- User-agent: which crawler the group is for.
- Disallow: a path the crawler should not fetch.
- Allow: a path it may fetch, even inside a disallowed folder.
- Sitemap: where your sitemap is. This one can go anywhere in the file.
Google ignores everything else. Crawl-delay is not supported by Google, though Bing honours it. A noindex line in robots.txt has done nothing in Google since 1 September 2019.
How the rules match
Three rules decide what a crawler does, and they trip people up more than the syntax does.
A crawler follows one group: the most specific one that names it. If there is a group for Googlebot, Googlebot ignores the star group entirely. Groups don't add up. One of the sites I run handles this correctly: it names each AI crawler in its own group and repeats the same Disallow lines in every one, because a named group would otherwise lose the rules written under the star.
Within a group, the longest matching rule wins. With Disallow: /shop and Allow: /shop/sale, the sale pages are allowed, because the longer rule is the closer match. When an Allow and a Disallow are the same length, Google picks the less restrictive one, the Allow.
Paths are case-sensitive, and two wildcards work. /Shop and /shop are different paths. A star matches any run of characters, and a dollar sign marks the end of the URL, so Disallow: /*.pdf$ blocks every PDF. The words Disallow and User-agent themselves are not case-sensitive.
What robots.txt can't do
It can't keep a page out of search results. Google's own guide says robots.txt is not a way to keep a page out of Google. If other pages link to a blocked URL, Google may still list it, with no description, because it never read the page. To keep a page out, let it be crawled and give it a noindex tag, or put it behind a password. A noindex on a page that robots.txt blocks is never seen, so the two together fail.
It can't hide anything. The RFC that standardised robots.txt says its rules are not a form of access authorisation. One of my own sites learned this the obvious way: its robots.txt disallows 4 internal files, and on 9 November 2026 all 4 still loaded for anyone who asked for them. Nothing in them was secret, but the robots.txt line protected none of it.
How Google treats the file itself
| Situation | What Google does |
|---|---|
| The file is over 500 KiB | Reads the first 500 KiB and ignores the rest |
| robots.txt returns 404 or another 4xx (not 429) | Crawls as if there were no robots.txt |
| robots.txt returns a 5xx | Stops crawling the site for 12 hours, then uses the last good copy for up to 30 days |
| Any successful read | Caches the file for up to 24 hours |
From Google's robots.txt specification, read 9 November 2026.
That 5xx row is the one to watch. A server error on robots.txt doesn't mean "no rules"; for Google it means "stop".
AI crawlers in robots.txt
Each AI company documents its own crawler names, and they do different jobs. Blocking the training crawler is not the same as leaving an AI's search:
| Company | Name in robots.txt | What blocking it does |
|---|---|---|
| OpenAI | GPTBot | Asks that your content not be used for training |
| OpenAI | OAI-SearchBot | Keeps your pages out of ChatGPT's search results |
| Anthropic | ClaudeBot | Asks that your future content not be used for training |
| Anthropic | Claude-SearchBot | Stops Claude indexing your content for search |
| Anthropic | Claude-User | Stops Claude fetching your page when a user asks about it |
| Google-Extended | Opts out of Gemini training and grounding. It does not affect Google Search |
Read on each company's own crawler page, 9 November 2026.
Check Engine's study shows how sites use these. Of the 98 large, well-known sites it read (a hand-picked list, not a random sample), 42 mentioned at least one AI crawler, 37 blocked at least one training crawler, 29 blocked at least one crawler that feeds AI answers, and 4 blocked a search engine. The most-blocked name was CCBot, Common Crawl's crawler, on 33 sites. The AI crawler directives guide covers which to keep and which to block.
What Check Engine checks
Check Engine reads your robots.txt before it crawls and follows the rules written for Googlebot, and for Check Engine's own crawler if you name it, matching them the way the standard describes: longest match, wildcards and the dollar sign. It reports:
- a robots.txt that blocks a crawler from the whole site;
- a robots.txt with syntax problems, or one that returns a server error;
- AI search crawlers and AI training crawlers that are blocked, listed separately because they do different jobs;
- a CDN or firewall that blocks a crawler your robots.txt allows.
You can test a single URL against your file with the free robots.txt tester.
Sources
- Google, robots.txt introduction
- Google, how Google interprets robots.txt
- Google, block indexing with noindex
- RFC 9309
- Bing, crawl control
- OpenAI crawlers
- Anthropic crawlers
- Google common crawlers
$59once
For Macs with Apple silicon or Intel, macOS 12 or later, and 64-bit Windows
Priced in US dollars. Checkout shows your currency and any local tax.
On a Mac? Get the Mac version.
On your phone?
Launch price: $29.50 once, through 30 November. From 1 December: $59. Buy applies the code FOUNDINGDOCKET for you; if the checkout shows $59, enter it under ‘Have a discount code?’.
Screaming Frog is $279 a year. Ahrefs plans with Site Audit run $129–$449 a month. Check Engine is $59 once. Screaming Frog’s price read 4 November 2026, Ahrefs’ 9 November 2026.
30 days, no conditions, no questions asked. Refund policy
Common questions
Does robots.txt stop a page from being indexed?
No. It stops crawling. Google can still list a blocked URL, without a description, if other pages link to it. To keep a page out of search, allow crawling and add a noindex tag.
Do I need a robots.txt file?
Not strictly. With no robots.txt, crawlers treat every URL as allowed. Many sites have one anyway, to point to the sitemap and keep crawlers out of cart, search and admin pages.
Does Google support crawl-delay?
No. Google ignores it. Bing honours it. To slow Googlebot down, your server can answer with a 503 or 429 status for a while.
Is robots.txt case-sensitive?
The paths are: /Shop and /shop are different. The field names, such as Disallow, and the crawler names are not.