Sidetrade indexer bot
Sidetrade indexer bot collects content for model training, and the community reference lists its operator as Sidetrade. Our October 2026 survey found 0 sites blocking it, or 0.0% of the files we could read.
Who runs Sidetrade indexer bot?
| Fact | Sidetrade indexer bot reference entry |
|---|---|
| Operator | “Sidetrade” |
| Purpose | “Extracts data for a variety of uses including training AI.” |
| Does Sidetrade indexer bot obey robots.txt? | “Unclear at this time.” |
| Frequency | “No information.” |
| Cloudflare managed list for Sidetrade indexer bot | No |
These facts quote the stored community ai.robots.txt reference.
“AI product training.”
Model training is this crawler’s listed use. An owner may permit an assistant to read a page for someone and deny bulk collection that would use it as training material.
How many sites block it?
We have 6,171 readable files for this entry, 83 of them naming it, a 1.3% share. Whole-site blocks written Disallow: / total 0. That puts the count without a recorded ban at 6,171.
Its name appears in 24 files near the top, compared with 59 farther down. The shares are 4.3% of 552 readable files in the top 1,000, and 1.1% of 5,619 in remaining ranks. Both samples have no recorded whole-site blocks.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Sidetrade indexer bot, all ranks | 83 | 1.3% | 0 | 0.0% |
| Sidetrade indexer bot, top 1,000 | 24 | 4.3% | 0 | 0.0% |
| Sidetrade indexer bot, remaining ranks | 59 | 1.1% | 0 | 0.0% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. Names and bans overlap in 0 files. We also found 83 files with a name and no whole-site ban, plus 6,088 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a Sidetrade indexer bot group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: Sidetrade indexer bot
Disallow: /Allow access
User-agent: Sidetrade indexer bot
Allow: /It's the 72nd most-named crawler of the 171 we counted in the crawler index.
How we counted
For Sidetrade indexer bot, we surveyed Tranco’s top 10,000 sites on 19 October 2026. An 8-second wait or a file over 262,144 bytes ended a read. Shares exclude unreachable, unavailable and oversized files. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
This survey includes 83 files with the name and no whole-site ban, out of 83 naming files.
What do the top-ranked sites show?
Its name appears in 4.3% of readable leading files and 1.3% overall. Blocking affects 0.0% in the leading group and 0.0% in all ranks.
Which nearby entry can I compare?
The nearby SemrushBot-OCOB entry reports 80 files that name it and 275 whole-site blocks, 4.5% of readable files. Here, 83 files mention the name and 0 block access, or 0.0%.