AI2Bot
AI2Bot collects content for model training, and the community reference lists its operator as Ai2. Our October 2026 survey found 251 sites blocking it, or 4.1% of the files we could read.
Who runs AI2Bot?
| Fact | AI2Bot reference entry |
|---|---|
| Operator | “Ai2” |
| Purpose | “Content is used to train open language models.” |
| Does AI2Bot obey robots.txt? | “Yes” |
| Frequency | “No information provided.” |
| Cloudflare managed list for AI2Bot | No |
These facts quote the stored community ai.robots.txt reference.
A matching description is used for 1 other entry. Counts below are specific to this crawler.
Model training is this crawler’s listed use. An owner may want search visitors and answers that link back to a page while keeping its text out of model-training datasets.
How many sites block it?
The name occurs in 278 out of 6,171 readable files, a 4.5% share. Across the full sample, the whole-site Disallow: / block count is 251, or 4.1%. Files without that ban number 5,920.
At the top 1,000 ranks, we could read 552 files. Naming covers 44, a 8.0% share, and blocking covers 39. Blocking is more common among the leading sites, at 7.1%. Elsewhere, 5,619 readable files give 212 blocks, or 3.8%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| AI2Bot, all ranks | 278 | 4.5% | 251 | 4.1% |
| AI2Bot, top 1,000 | 44 | 8.0% | 39 | 7.1% |
| AI2Bot, remaining ranks | 234 | 4.2% | 212 | 3.8% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. Of 251 blocks, 6 do not also appear in our count of files naming it. The counted overlap is 245 files, with 33 files naming it without banning all paths. Missing names account for 5,893 files. Repeating a name or changing its capitalization does not increase the naming total.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for AI2Bot take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: AI2Bot
Disallow: /Allow access
User-agent: AI2Bot
Allow: /It's the 31st most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For AI2Bot, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
The 278 files naming it include 33 files without a whole-site ban.
Are leading sites different?
At the top of the ranking, 8.0% of readable files name it and 7.1% block it. Across all ranks, those shares are 4.5% and 4.1%.
How does the nearby crawler compare?
For comparison, Applebot has 276 files naming it and 264 blocks, or 4.3%. Our counts for this entry are 278 files with the name and 251 blocking the whole site, or 4.1%.