OAI-SearchBot
OAI-SearchBot is listed as a search crawler, and the community reference lists its operator as OpenAI. In October 2026, 427 sites block it, a 6.9% share of readable files.
Who runs OAI-SearchBot?
| Fact | OAI-SearchBot reference entry |
|---|---|
| Operator | “OpenAI” |
| Purpose | “Search result generation.” |
| Does OAI-SearchBot obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for OAI-SearchBot | No |
These facts quote the stored community ai.robots.txt reference.
“Crawls sites to surface as results in SearchGPT.”
The listed use here is search. Someone asking an assistant to open a page is using it for an immediate answer. An owner may permit that request and block automatic training collection.
How many sites block it?
A User-agent line names it in 608 files, 9.9% of 6,171 readable files. Whole-site bans, written Disallow: /, total 427, or 6.9%. That leaves 5,744 without a recorded ban.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 89 files carry the name, or 16.1%, with 58 files blocking the whole site. Blocking is more common among the leading sites, at 10.5%. Farther down, 369 of 5,619 files record a block, 6.6% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| OAI-SearchBot, all ranks | 608 | 9.9% | 427 | 6.9% |
| OAI-SearchBot, top 1,000 | 89 | 16.1% | 58 | 10.5% |
| OAI-SearchBot, remaining ranks | 519 | 9.2% | 369 | 6.6% |
Counts include a whole-site ban in a User-agent: * group if no named group overrides it. Of 427 blocks, 182 come only from that general rule, without naming this crawler. The counted overlap is 245 files, with 363 files naming it without banning all paths. Missing names account for 5,563 files. Repeating a name or changing its capitalization does not increase the naming total.
Has that changed since August?
Between August and October, 25 sites that had blocked it stopped and 22 started, so 3 fewer of the same 5,889 sites block it. A further 376 blocked it in both months, and 5,466 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 425 blocks among 6,236 readable files. October’s was 427 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 22 |
| Stopped blocking | 25 |
| Blocked in both | 376 |
| Blocked in neither | 5,466 |
| Net change | -3 |
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a OAI-SearchBot group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: OAI-SearchBot
Disallow: /Allow access
User-agent: OAI-SearchBot
Allow: /It's the 12th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For OAI-SearchBot, our 19 October 2026 collection used Tranco’s top 10,000 sites. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
This survey includes 363 files with the name and no whole-site ban, out of 608 naming files.
How do the shares compare by rank?
At the top of the ranking, 16.1% of readable files name it and 10.5% block it. Across all ranks, those shares are 9.9% and 6.9%.
What other entry is close in the index?
The nearby cohere-ai entry reports 587 files that name it and 722 whole-site blocks, 11.7% of readable files. Here, 608 files mention the name and 427 block access, or 6.9%.