ExaSearchBot
ExaSearchBot is listed as a search crawler, and the community reference lists its operator as Exa. In October 2026, 40 sites block it, a 0.6% share of readable files.
Who runs ExaSearchBot?
| Fact | ExaSearchBot reference entry |
|---|---|
| Operator | “Exa” |
| Purpose | “AI Search Crawlers” |
| Does ExaSearchBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for ExaSearchBot | No |
These facts quote the stored community ai.robots.txt reference.
“ExaSearchBot is a web crawler operated by Exa that discovers and indexes public web pages so their content can be found, retrieved, and cited through Exa.”
The listed use here is search. Helping people find a page through search is a different use from using its contents to train a model, so owners may allow those visits separately.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 42, or 0.7%. Instructions blocking every path, Disallow: /, appear in 40, a 0.6% share. No whole-site ban is recorded for 6,131 files.
Breaking out the leading 1,000 sites gives 552 readable files, with 5 naming the crawler, or 0.9%. Whole-site blocking appears in 6 of them. Blocking is more common among the leading sites, at 1.1%. Beyond those ranks, 5,619 files have a blocking total of 34, 0.6% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| ExaSearchBot, all ranks | 42 | 0.7% | 40 | 0.6% |
| ExaSearchBot, top 1,000 | 5 | 0.9% | 6 | 1.1% |
| ExaSearchBot, remaining ranks | 37 | 0.7% | 34 | 0.6% |
Here, the total counts whole-site blocks recorded for this crawler. It leaves out general User-agent: * instructions. Of 40 blocks, 2 do not also appear in our count of files naming it. The naming total includes 38 files with a ban on every path and 4 files without that ban. Elsewhere, 6,129 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for ExaSearchBot take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: ExaSearchBot
Disallow: /Allow access
User-agent: ExaSearchBot
Allow: /It's the 127th most-named crawler of the 171 we counted in the crawler index.
How we counted
For ExaSearchBot, our 19 October 2026 collection used Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
We found 4 files without a whole-site ban among 42 that name it.
What do the top-ranked sites show?
Leading files record 0.9% with the name and 1.1% blocking it. Across all readable files, the corresponding shares are 0.7% and 0.6%.
Which nearby entry can I compare?
For comparison, Poseidon Research Crawler has 42 files naming it and 0 blocks, or 0.0%. Our counts for this entry are 42 files with the name and 40 blocking the whole site, or 0.6%.