OpenAI
OpenAI has 32 recorded blocks in October 2026, or 0.5%, and 30 files naming it. The community reference lists its operator as OpenAI.
Who runs OpenAI?
| Fact | OpenAI reference entry |
|---|---|
| Operator | “OpenAI” |
| Purpose | “Unclear at this time.” |
| Does OpenAI obey robots.txt? | “Yes” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for OpenAI | No |
These facts quote the stored community ai.robots.txt reference.
“The purpose of this bot is unclear at this time but it is a member of OpenAI's suite of crawlers.”
This entry’s stated use is quoted above. Search indexing is aimed at helping people find pages. Feeding text into model training is another use an owner may decline.
How many sites block it?
The 32 files blocking the whole site account for 0.5% of 6,171 readable robots.txt files. We found its name in 30 User-agent lists, a 0.5% share. Without a whole-site Disallow: / block, 6,139 files remain.
Among 552 readable files from the top 1,000 sites, we found 3 files naming it, a 0.5% share, with 2 files blocking the whole site. The leading sites have a lower block share, 0.4%. Outside this group, 5,619 files contribute 30 blocks, or 0.5%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| OpenAI, all ranks | 30 | 0.5% | 32 | 0.5% |
| OpenAI, top 1,000 | 3 | 0.5% | 2 | 0.4% |
| OpenAI, remaining ranks | 27 | 0.5% | 30 | 0.5% |
Here, the total counts whole-site blocks recorded for this crawler. It leaves out general User-agent: * instructions. Of 32 blocks, 5 do not also appear in our count of files naming it. The overlap between naming and blocking is 27 files. Naming without a whole-site ban accounts for 3, and 6,141 files never name it. We ignore capitalization and repeated names.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a OpenAI group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: OpenAI
Disallow: /Allow access
User-agent: OpenAI
Allow: /It's the 138th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For OpenAI, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Each file had 8 seconds to arrive, up to 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. Repeat readings are included, using scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
We found 3 files without a whole-site ban among 30 that name it.
How do the shares compare by rank?
At the top of the ranking, 0.5% of readable files name it and 0.4% block it. Across all ranks, those shares are 0.5% and 0.5%.
What other entry is close in the index?
The nearby AzureAI-SearchBot entry reports 29 files that name it and 259 whole-site blocks, 4.2% of readable files. Here, 30 files mention the name and 32 block access, or 0.5%.