GPTBot
GPTBot collects content for model training, and the community reference lists its operator as OpenAI. Our October 2026 survey found 906 sites blocking it, or 14.7% of the files we could read.
Who runs GPTBot?
| Fact | GPTBot reference entry |
|---|---|
| Operator | “OpenAI” |
| Purpose | “Scrapes data to train OpenAI's products.” |
| Does GPTBot obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for GPTBot | Yes |
These facts quote the stored community ai.robots.txt reference.
“Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.”
Model training is this crawler’s listed use. Search referrals or a user’s question can give an owner a reason to allow access while declining collection of page text for training.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 1,108, or 18.0%. Instructions blocking every path, Disallow: /, appear in 906, a 14.7% share. No whole-site ban is recorded for 5,265 files.
For the top 1,000 ranks, its name appears in 134 files out of 552, or 24.3%. Whole-site blocks total 109. Blocking is more common among the leading sites, at 19.7%. Lower in the ranking, blocking reaches 797 files out of 5,619, a 14.2% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| GPTBot, all ranks | 1,108 | 18.0% | 906 | 14.7% |
| GPTBot, top 1,000 | 134 | 24.3% | 109 | 19.7% |
| GPTBot, remaining ranks | 974 | 17.3% | 797 | 14.2% |
A general User-agent: * ban contributes to this total whenever it applies to this crawler. Of 906 blocks, 174 come only from that general rule, without naming this crawler. Looking just at named files gives 732 blocks and 376 without a whole-site block. For 5,063 readable files, we found no mention by name. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 171 sites that had blocked it stopped and 49 started, so 122 fewer of the same 5,889 sites block it. A further 804 blocked it in both months, and 4,865 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 1,093 blocks among 6,236 readable files. October’s was 906 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 49 |
| Stopped blocking | 171 |
| Blocked in both | 804 |
| Blocked in neither | 4,865 |
| Net change | -122 |
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for GPTBot take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: GPTBot
Disallow: /Allow access
User-agent: GPTBot
Allow: /It's the most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For GPTBot, our 19 October 2026 collection used Tranco’s top 10,000 sites. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
The 1,108 files that mention its name do not all record a whole-site ban. Those without one total 376.
Are leading sites different?
At the top of the ranking, 24.3% of readable files name it and 19.7% block it. Across all ranks, those shares are 18.0% and 14.7%.
How does the nearby crawler compare?
The nearby ClaudeBot entry reports 946 files that name it and 835 whole-site blocks, 13.5% of readable files. Here, 1,108 files mention the name and 906 block access, or 14.7%.