ExaBot
ExaBot is blocked by 334 sites in the October 2026 survey, or 5.4% of readable files. Its operator is unclear in the community reference.
Who runs ExaBot?
| Fact | ExaBot reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Providers” |
| Does ExaBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for ExaBot | No |
These facts quote the stored community ai.robots.txt reference.
“ExaBot is a web crawler that indexes web content to power Exa's AI search engine and semantic search APIs for AI applications.”
Its listed role is supplying data for AI tools. An owner may want a page included in search and available on request without offering its contents for future model development.
How many sites block it?
The 334 files blocking the whole site account for 5.4% of 6,171 readable robots.txt files. We found its name in 141 User-agent lists, a 2.3% share. Without a whole-site Disallow: / block, 5,837 files remain.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 17, or 3.1%, and the block count is 53. Blocking is more common among the leading sites, at 9.6%. Compare 281 blocks among the 5,619 readable files below those ranks, or 5.0%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ExaBot, all ranks | 141 | 2.3% | 334 | 5.4% |
| ExaBot, top 1,000 | 17 | 3.1% | 53 | 9.6% |
| ExaBot, remaining ranks | 124 | 2.2% | 281 | 5.0% |
Both named groups and general User-agent: * groups can contribute to blocking the whole site. Of 334 blocks, 224 come only from that general rule, without naming this crawler. The counted overlap is 110 files, with 31 files naming it without banning all paths. Missing names account for 6,030 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 16 sites that had blocked it stopped and 22 started, so 6 more of the same 5,889 sites block it. A further 276 blocked it in both months, and 5,575 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 316 blocks among 6,236 readable files. October’s was 334 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 22 |
| Stopped blocking | 16 |
| Blocked in both | 276 |
| Blocked in neither | 5,575 |
| Net change | +6 |
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming ExaBot under the robots.txt standard.
Block the whole site
User-agent: ExaBot
Disallow: /Allow access
User-agent: ExaBot
Allow: /It's the 50th most-named crawler of the 171 we counted in the crawler index.
How we counted
For ExaBot, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Only available files we could read enter the percentages. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
What does naming this crawler tell me?
We found 31 files without a whole-site ban among 141 that name it.
Are leading sites different?
Leading files record 3.1% with the name and 9.6% blocking it. Across all readable files, the corresponding shares are 2.3% and 5.4%.
How does the nearby crawler compare?
The nearby QualifiedBot entry reports 141 files that name it and 136 whole-site blocks, 2.2% of readable files. Here, 141 files mention the name and 334 block access, or 5.4%.