xAI-Grok
xAI-Grok is blocked by 64 sites in the October 2026 survey, or 1.0% of readable files. The community reference lists its operator as xAI.
Who runs xAI-Grok?
| Fact | xAI-Grok reference entry |
|---|---|
| Operator | “xAI” |
| Purpose | “xAI-Grok is an xAI crawler token associated with Grok content collection.” |
| Does xAI-Grok obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for xAI-Grok | No |
These facts quote the stored community ai.robots.txt reference.
“xAI-Grok is an xAI crawler token associated with Grok content collection. It is classified as training because it performs automated collection rather than a user-triggered fetch.”
This entry’s stated use is quoted above. Search indexing is aimed at helping people find pages. Feeding text into model training is another use an owner may decline.
How many sites block it?
Instructions to stay out of the whole site appear in 64 files, or 1.0% of 6,171 readable files. We counted 46 with its name in a User-agent line, a 0.7% share. Another 6,107 lack a recorded Disallow: / ban.
At the top 1,000 ranks, we could read 552 files. Naming covers 16, a 2.9% share, and blocking covers 21. Blocking is more common among the leading sites, at 3.8%. Elsewhere, 5,619 readable files give 43 blocks, or 0.8%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| xAI-Grok, all ranks | 46 | 0.7% | 64 | 1.0% |
| xAI-Grok, top 1,000 | 16 | 2.9% | 21 | 3.8% |
| xAI-Grok, remaining ranks | 30 | 0.5% | 43 | 0.8% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 64 blocks, 22 do not also appear in our count of files naming it. Names and bans overlap in 42 files. We also found 4 files with a name and no whole-site ban, plus 6,125 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming xAI-Grok under the robots.txt standard.
Block the whole site
User-agent: xAI-Grok
Disallow: /Allow access
User-agent: xAI-Grok
Allow: /It's the 122nd most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For xAI-Grok, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
A name identifies the crawler addressed by a group. The survey found 4 files with that name and no recorded whole-site ban, out of 46.
What do the top-ranked sites show?
Leading files record 2.9% with the name and 3.8% blocking it. Across all readable files, the corresponding shares are 0.7% and 1.0%.
Which nearby entry can I compare?
The nearby Anomura entry reports 45 files that name it and 44 whole-site blocks, 0.7% of readable files. Here, 46 files mention the name and 64 block access, or 1.0%.