Thinkbot
Thinkbot has 63 sites asking it to stay out in October 2026, or 1.0% of the files we could read. However, the reference says it does not obey robots.txt, and the community reference lists its operator as Thinkbot.
Who runs Thinkbot?
| Fact | Thinkbot reference entry |
|---|---|
| Operator | “Thinkbot” |
| Purpose | “Insights on AI integration and automation.” |
| Does Thinkbot obey robots.txt? | “No” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for Thinkbot | No |
These facts quote the stored community ai.robots.txt reference.
“Collects data for analysis on AI usage and automation.”
The reference says this crawler ignores robots.txt. A site can welcome a page fetch prompted by a person while declining the collection of that text to build future models.
How many sites block it?
The 63 files blocking the whole site account for 1.0% of 6,171 readable robots.txt files. We found its name in 62 User-agent lists, a 1.0% share. Without a whole-site Disallow: / block, 6,108 files remain.
Breaking out the leading 1,000 sites gives 552 readable files, with 17 naming the crawler, or 3.1%. Whole-site blocking appears in 17 of them. Blocking is more common among the leading sites, at 3.1%. Beyond those ranks, 5,619 files have a blocking total of 46, 0.8% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Thinkbot, all ranks | 62 | 1.0% | 63 | 1.0% |
| Thinkbot, top 1,000 | 17 | 3.1% | 17 | 3.1% |
| Thinkbot, remaining ranks | 45 | 0.8% | 46 | 0.8% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 63 blocks, 1 does not also appear in our count of files naming it. Among files listing the crawler, 62 record a ban on all paths. The number not naming it is 6,109. For naming, repeats and different case are merged.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming Thinkbot takes precedence over User-agent: *.
Block the whole site
User-agent: Thinkbot
Disallow: /Allow access
User-agent: Thinkbot
Allow: /It's the 101st most-named crawler of the 171 we counted in the crawler index.
How we counted
For Thinkbot, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Shares exclude unreachable, unavailable and oversized files. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
Every one of the 62 files naming it also has a whole-site ban.
What do the top-ranked sites show?
Leading files record 3.1% with the name and 3.1% blocking it. Across all readable files, the corresponding shares are 1.0% and 1.0%.
Which nearby entry can I compare?
Crawlspace is named by 61 files and blocked by 58, or 0.9%. For this crawler, 62 files name it and 63 block it, a 1.0% share.