cohere-ai
cohere-ai fetches pages in response to users, according to the reference, and the community reference lists its operator as Cohere. In October 2026, 722 sites block it, or 11.7% of readable robots.txt files.
Who runs cohere-ai?
| Fact | cohere-ai reference entry |
|---|---|
| Operator | “Cohere” |
| Purpose | “Retrieves data to provide responses to user-initiated prompts.” |
| Does cohere-ai obey robots.txt? | “Unclear at this time.” |
| Frequency | “Takes action based on user prompts.” |
| Cloudflare managed list for cohere-ai | No |
These facts quote the stored community ai.robots.txt reference.
“Retrieves data based on user prompts.”
Here, the reference describes visits triggered by a user. Keeping a site available to readers may include allowing search and requested fetches. An owner may still keep the text out of training collections.
How many sites block it?
Our readable sample has 722 whole-site blocks, 11.7% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 587 name this crawler, or 9.5%. The count without a whole-site block is 5,449.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 74 files carry the name, or 13.4%, with 94 files blocking the whole site. Blocking is more common among the leading sites, at 17.0%. Farther down, 628 of 5,619 files record a block, 11.2% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| cohere-ai, all ranks | 587 | 9.5% | 722 | 11.7% |
| cohere-ai, top 1,000 | 74 | 13.4% | 94 | 17.0% |
| cohere-ai, remaining ranks | 513 | 9.1% | 628 | 11.2% |
Rules for every crawler, written User-agent: *, count here too. Of 722 blocks, 225 come only from that general rule, without naming this crawler. We found 497 files naming it and blocking the whole site, along with 90 files naming it without that ban. Another 5,584 files omit the name. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 29 sites that had blocked it stopped and 75 started, so 46 more of the same 5,889 sites block it. A further 603 blocked it in both months, and 5,182 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 658 blocks among 6,236 readable files. October’s was 722 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 75 |
| Stopped blocking | 29 |
| Blocked in both | 603 |
| Blocked in neither | 5,182 |
| Net change | +46 |
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for cohere-ai take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: cohere-ai
Disallow: /Allow access
User-agent: cohere-ai
Allow: /It's the 13th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For cohere-ai, our 19 October 2026 collection used Tranco’s top 10,000 sites. An 8-second wait or a file over 262,144 bytes ended a read. Missing, unreachable and oversized files do not contribute to these shares. Reading files again used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) as well. See the full survey method.
Common questions
What does naming this crawler tell me?
We found 90 files without a whole-site ban among 587 that name it.
Are leading sites different?
Its name appears in 13.4% of readable leading files and 9.5% overall. Blocking affects 17.0% in the leading group and 11.7% in all ranks.
How does the nearby crawler compare?
The nearby Diffbot entry reports 553 files that name it and 723 whole-site blocks, 11.7% of readable files. Here, 587 files mention the name and 722 block access, or 11.7%.