SemrushBot-OCOB
SemrushBot-OCOB is blocked by 275 sites in the October 2026 survey, or 4.5% of readable files. The community reference lists its operator as Semrush.
Who runs SemrushBot-OCOB?
| Fact | SemrushBot-OCOB reference entry |
|---|---|
| Operator | “Semrush” |
| Purpose | “Crawls your site for ContentShake AI tool.” |
| Does SemrushBot-OCOB obey robots.txt? | “Yes” |
| Frequency | “Roughly once every 10 seconds.” |
| Cloudflare managed list for SemrushBot-OCOB | No |
These facts quote the stored community ai.robots.txt reference.
“Data collected is used for the ContentShake AI tool reports.”
This entry’s stated use is quoted above. Using a page as a search result can send people to it. Training models uses its contents for another purpose, which an owner may refuse.
How many sites block it?
Instructions to stay out of the whole site appear in 275 files, or 4.5% of 6,171 readable files. We counted 80 with its name in a User-agent line, a 1.3% share. Another 5,896 lack a recorded Disallow: / ban.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 15, or 2.7%, and the block count is 22. The leading sites have a lower block share, 4.0%. Compare 253 blocks among the 5,619 readable files below those ranks, or 4.5%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| SemrushBot-OCOB, all ranks | 80 | 1.3% | 275 | 4.5% |
| SemrushBot-OCOB, top 1,000 | 15 | 2.7% | 22 | 4.0% |
| SemrushBot-OCOB, remaining ranks | 65 | 1.2% | 253 | 4.5% |
For this entry, we counted whole-site bans attached to its own name, leaving general rules for all crawlers out of the total. Of 275 blocks, 208 do not also appear in our count of files naming it. Names and bans overlap in 67 files. We also found 13 files with a name and no whole-site ban, plus 6,091 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for SemrushBot-OCOB take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: SemrushBot-OCOB
Disallow: /Allow access
User-agent: SemrushBot-OCOB
Allow: /It's the 73rd most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For SemrushBot-OCOB, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Missing, unreachable and oversized files do not contribute to these shares. Collection and repeat readings used scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 13 files without a whole-site ban among 80 that name it.
How do the shares compare by rank?
At the top of the ranking, 2.7% of readable files name it and 4.0% block it. Across all ranks, those shares are 1.3% and 4.5%.
What other entry is close in the index?
The nearby Devin entry reports 79 files that name it and 69 whole-site blocks, 1.1% of readable files. Here, 80 files mention the name and 275 block access, or 4.5%.