AgentDataBot
Few readable files block AgentDataBot in October 2026, at 16 sites or 0.3%. Its operator is unclear in the community reference.
Who runs AgentDataBot?
| Fact | AgentDataBot reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Providers” |
| Does AgentDataBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for AgentDataBot | No |
These facts quote the stored community ai.robots.txt reference.
“AgentDataBot visits public web pages to identify technology stacks, company details, and team members. The data builds AgentData's company-intelligence database for AI ag…”
Its listed role is supplying data for AI tools. An owner may want search visitors and answers that link back to a page while keeping its text out of model-training datasets.
How many sites block it?
Of 6,171 robots.txt files we could read, 16 block the whole site with Disallow: /, or 0.3%. Another way to read the sample is by name, with 16 files naming it in a User-agent line, or 0.3%. The 6,155 remaining files have no recorded whole-site block.
Tranco’s top 1,000 sites yielded 552 readable files, including 1 file with its name, or 0.2%, and 1 file blocking it. The leading sites have a lower block share, 0.2%. Outside those ranks, 15 of 5,619 files block it, or 0.3%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| AgentDataBot, all ranks | 16 | 0.3% | 16 | 0.3% |
| AgentDataBot, top 1,000 | 1 | 0.2% | 1 | 0.2% |
| AgentDataBot, remaining ranks | 15 | 0.3% | 15 | 0.3% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 16 blocks, 1 does not also appear in our count of files naming it. The split among files that name it is 15 with a whole-site block and 1 without one. Its name is absent from 6,155 readable files. We ignore capitalization and repeated names.
How to block or allow it
Add these lines to robots.txt at your site root. For AgentDataBot, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: AgentDataBot
Disallow: /Allow access
User-agent: AgentDataBot
Allow: /It's the 160th most-named crawler of the 171 we counted in the crawler index.
How we counted
For AgentDataBot, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Requests could run for 8 seconds, with a 262,144-byte file limit. Only available files we could read enter the percentages. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 1 file without a whole-site ban among 16 that name it.
How do the shares compare by rank?
For naming rules, 0.2% of the leading sample mentions it, compared with 0.3% overall. Whole-site blocking is 0.2% at leading ranks and 0.3% across the full sample.
What other entry is close in the index?
The nearby Aranet-SearchBot entry reports 16 files that name it and 16 whole-site blocks, 0.3% of readable files. Here, 16 files mention the name and 16 block access, or 0.3%.