TavilyBot
TavilyBot is blocked by 292 sites in the October 2026 survey, or 4.7% of readable files. Its operator is unclear in the community reference.
Who runs TavilyBot?
| Fact | TavilyBot reference entry |
|---|---|
| Operator | “Unclear at this time.” |
| Purpose | “AI Data Providers” |
| Does TavilyBot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for TavilyBot | No |
These facts quote the stored community ai.robots.txt reference.
“TavilyBot is a web crawler by Tavily that indexes and extracts content from billions of pages, providing real-time search, extraction, and research data to ground AI agen…”
Its listed role is supplying data for AI tools. Reading a page to answer a user has a different purpose from building a model with it. Owners may allow the requested visit.
How many sites block it?
The 292 files blocking the whole site account for 4.7% of 6,171 readable robots.txt files. We found its name in 72 User-agent lists, a 1.2% share. Without a whole-site Disallow: / block, 5,879 files remain.
Breaking out the leading 1,000 sites gives 552 readable files, with 22 naming the crawler, or 4.0%. Whole-site blocking appears in 60 of them. Blocking is more common among the leading sites, at 10.9%. Beyond those ranks, 5,619 files have a blocking total of 232, 4.1% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| TavilyBot, all ranks | 72 | 1.2% | 292 | 4.7% |
| TavilyBot, top 1,000 | 22 | 4.0% | 60 | 10.9% |
| TavilyBot, remaining ranks | 50 | 0.9% | 232 | 4.1% |
The survey includes relevant User-agent: * instructions alongside instructions addressed to this crawler. Of 292 blocks, 227 come only from that general rule, without naming this crawler. The counted overlap is 65 files, with 7 files naming it without banning all paths. Missing names account for 6,099 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 15 sites that had blocked it stopped and 18 started, so 3 more of the same 5,889 sites block it. A further 242 blocked it in both months, and 5,614 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 277 blocks among 6,236 readable files. October’s was 292 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 18 |
| Stopped blocking | 15 |
| Blocked in both | 242 |
| Blocked in neither | 5,614 |
| Net change | +3 |
How to block or allow it
Add these lines to robots.txt at your site root. For TavilyBot, the robots.txt standard gives a named group priority over User-agent: *.
Block the whole site
User-agent: TavilyBot
Disallow: /Allow access
User-agent: TavilyBot
Allow: /It's the 82nd most-named crawler of the 171 we counted in the crawler index.
How we counted
For TavilyBot, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Missing, unreachable and oversized files do not contribute to these shares. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
This survey includes 7 files with the name and no whole-site ban, out of 72 naming files.
What do the top-ranked sites show?
Its name appears in 4.0% of readable leading files and 1.2% overall. Blocking affects 10.9% in the leading group and 4.7% in all ranks.
Which nearby entry can I compare?
YandexAdditionalBot is named by 71 files and blocked by 174, or 2.8%. For this crawler, 72 files name it and 292 block it, a 4.7% share.