cohere-training-data-crawler
cohere-training-data-crawler is blocked by 246 sites in the October 2026 survey, or 4.0% of readable files. The community reference lists its operator as Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products.
Who runs cohere-training-data-crawler?
| Fact | cohere-training-data-crawler reference entry |
|---|---|
| Operator | “Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products” |
| Purpose | “AI Data Scrapers” |
| Does cohere-training-data-crawler obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for cohere-training-data-crawler | No |
These facts quote the stored community ai.robots.txt reference.
“cohere-training-data-crawler is a web crawler operated by Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products.”
This entry is classed as a crawler collecting data for AI. A site can welcome a page fetch prompted by a person while declining the collection of that text to build future models.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 250, or 4.1%. Instructions blocking every path, Disallow: /, appear in 246, a 4.0% share. No whole-site ban is recorded for 5,925 files.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 39 files carry the name, or 7.1%, with 34 files blocking the whole site. Blocking is more common among the leading sites, at 6.2%. Farther down, 212 of 5,619 files record a block, 3.8% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| cohere-training-data-crawler, all ranks | 250 | 4.1% | 246 | 4.0% |
| cohere-training-data-crawler, top 1,000 | 39 | 7.1% | 34 | 6.2% |
| cohere-training-data-crawler, remaining ranks | 211 | 3.8% | 212 | 3.8% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 246 blocks, 21 do not also appear in our count of files naming it. The split among files that name it is 225 with a whole-site block and 25 without one. Its name is absent from 5,921 readable files. We ignore capitalization and repeated names.
How to block or allow it
Add these lines to robots.txt at your site root. For cohere-training-data-crawler, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: cohere-training-data-crawler
Disallow: /Allow access
User-agent: cohere-training-data-crawler
Allow: /It's the 34th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For cohere-training-data-crawler, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Requests could run for 8 seconds, with a 262,144-byte file limit. Shares exclude unreachable, unavailable and oversized files. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
A name identifies the crawler addressed by a group. The survey found 25 files with that name and no recorded whole-site ban, out of 250.
Are leading sites different?
Leading files record 7.1% with the name and 6.2% blocking it. Across all readable files, the corresponding shares are 4.1% and 4.0%.
How does the nearby crawler compare?
The nearby MistralAI-User entry reports 244 files that name it and 426 whole-site blocks, 6.9% of readable files. Here, 250 files mention the name and 246 block access, or 4.0%.