Ai2Bot-Dolma
Ai2Bot-Dolma collects content for model training, and the community reference lists its operator as Ai2. Our October 2026 survey found 271 sites blocking it, or 4.4% of the files we could read.
Who runs Ai2Bot-Dolma?
| Fact | Ai2Bot-Dolma reference entry |
|---|---|
| Operator | “Ai2” |
| Purpose | “Content is used to train open language models.” |
| Does Ai2Bot-Dolma obey robots.txt? | “Yes” |
| Frequency | “No information provided.” |
| Cloudflare managed list for Ai2Bot-Dolma | No |
These facts quote the stored community ai.robots.txt reference.
A matching description is used for 1 other entry. Counts below are specific to this crawler.
Model training is this crawler’s listed use. An owner may permit an assistant to read a page for someone and deny bulk collection that would use it as training material.
How many sites block it?
Of 6,171 robots.txt files we could read, 271 block the whole site with Disallow: /, or 4.4%. Another way to read the sample is by name, with 177 files naming it in a User-agent line, or 2.9%. The 5,900 remaining files have no recorded whole-site block.
Tranco’s top 1,000 sites yielded 552 readable files, including 34 files with its name, or 6.2%, and 40 files blocking it. Blocking is more common among the leading sites, at 7.2%. Outside those ranks, 231 of 5,619 files block it, or 4.1%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Ai2Bot-Dolma, all ranks | 177 | 2.9% | 271 | 4.4% |
| Ai2Bot-Dolma, top 1,000 | 34 | 6.2% | 40 | 7.2% |
| Ai2Bot-Dolma, remaining ranks | 143 | 2.5% | 231 | 4.1% |
This is a count of blocks attached to the named crawler, covering the whole site. Blanket rules for every crawler do not enter it. Of 271 blocks, 111 do not also appear in our count of files naming it. The overlap between naming and blocking is 160 files. Naming without a whole-site ban accounts for 17, and 5,994 files never name it. We ignore capitalization and repeated names.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for Ai2Bot-Dolma take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: Ai2Bot-Dolma
Disallow: /Allow access
User-agent: Ai2Bot-Dolma
Allow: /It's the 43rd most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For Ai2Bot-Dolma, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Requests could run for 8 seconds, with a 262,144-byte file limit. Only available files we could read enter the percentages. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
The 177 files that mention its name do not all record a whole-site ban. Those without one total 17.
Are leading sites different?
Leading files record 6.2% with the name and 7.2% blocking it. Across all readable files, the corresponding shares are 2.9% and 4.4%.
How does the nearby crawler compare?
For comparison, Cotoyogi has 177 files naming it and 170 blocks, or 2.8%. Our counts for this entry are 177 files with the name and 271 blocking the whole site, or 4.4%.