GrokBot
GrokBot is listed as a search crawler, and the community reference lists its operator as xAI. In October 2026, 73 sites block it, a 1.2% share of readable files.
Who runs GrokBot?
| Fact | GrokBot reference entry |
|---|---|
| Operator | “xAI” |
| Purpose | “AI Search Crawlers” |
| Does GrokBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for GrokBot | No |
These facts quote the stored community ai.robots.txt reference.
“Used to collect publicly available web content to support the training of Grok AI models.”
The listed use here is search. Someone asking an assistant to open a page is using it for an immediate answer. An owner may permit that request and block automatic training collection.
How many sites block it?
The 73 files blocking the whole site account for 1.2% of 6,171 readable robots.txt files. We found its name in 68 User-agent lists, a 1.1% share. Without a whole-site Disallow: / block, 6,098 files remain.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 18 files carry the name, or 3.3%, with 22 files blocking the whole site. Blocking is more common among the leading sites, at 4.0%. Farther down, 51 of 5,619 files record a block, 0.9% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| GrokBot, all ranks | 68 | 1.1% | 73 | 1.2% |
| GrokBot, top 1,000 | 18 | 3.3% | 22 | 4.0% |
| GrokBot, remaining ranks | 50 | 0.9% | 51 | 0.9% |
Here, the total counts whole-site blocks recorded for this crawler. It leaves out general User-agent: * instructions. Of 73 blocks, 16 do not also appear in our count of files naming it. The split among files that name it is 57 with a whole-site block and 11 without one. Its name is absent from 6,103 readable files. We ignore capitalization and repeated names.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a GrokBot group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: GrokBot
Disallow: /Allow access
User-agent: GrokBot
Allow: /It's the 90th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For GrokBot, our 19 October 2026 collection used Tranco’s top 10,000 sites. Files had an 8-second timeout and a 262,144-byte limit. Shares exclude unreachable, unavailable and oversized files. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
This survey includes 11 files with the name and no whole-site ban, out of 68 naming files.
Are leading sites different?
Its name appears in 3.3% of readable leading files and 1.1% overall. Blocking affects 4.0% in the leading group and 1.2% in all ranks.
How does the nearby crawler compare?
For comparison, AddSearchBot has 67 files naming it and 68 blocks, or 1.1%. Our counts for this entry are 68 files with the name and 73 blocking the whole site, or 1.2%.