Brightbot
Brightbot collects content for model training, and the community reference lists its operator as https://brightdata.com/brightbot. Our October 2026 survey found 86 sites blocking it, or 1.4% of the files we could read.
Who runs Brightbot?
| Fact | Brightbot reference entry |
|---|---|
| Operator | “Unclear at this time.” / “https://brightdata.com/brightbot” |
| Purpose | “AI Data Providers” / “LLM/AI training.” |
| Does Brightbot obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” / “At least one per minute.” |
| Cloudflare managed list for Brightbot | No |
These facts quote the stored community ai.robots.txt reference.
“Scrapes data to train LLMs and AI products focused on website customer support, uses residential IPs and legit-looking user-agents to disguise itself.”
Model training is this crawler’s listed use. Using a page as a search result can send people to it. Training models uses its contents for another purpose, which an owner may refuse.
How many sites block it?
A User-agent line names it in 167 files, 2.7% of 6,171 readable files. Whole-site bans, written Disallow: /, total 86, or 1.4%. That leaves 6,085 without a recorded ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 43 naming the crawler, or 7.8%. Whole-site blocking appears in 33 of them. Blocking is more common among the leading sites, at 6.0%. Beyond those ranks, 5,619 files have a blocking total of 53, 0.9% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Brightbot, all ranks | 167 | 2.7% | 86 | 1.4% |
| Brightbot, top 1,000 | 43 | 7.8% | 33 | 6.0% |
| Brightbot, remaining ranks | 124 | 2.2% | 53 | 0.9% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. Of 86 blocks, 1 does not also appear in our count of files naming it. The naming total includes 85 files with a ban on every path and 82 files without that ban. Elsewhere, 6,004 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
The examples include the spelling variants counted together here. Edit the root robots.txt file on your site. The robots.txt standard gives a Brightbot group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: Brightbot
User-agent: Brightbot 1.0
Disallow: /Allow access
User-agent: Brightbot
User-agent: Brightbot 1.0
Allow: /It's the 47th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Brightbot, on 19 October 2026, we requested robots.txt from Tranco’s top 10,000 sites. Each file had 8 seconds to arrive, up to 262,144 bytes. Only available files we could read enter the percentages. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Does a User-agent line mean it is blocked?
This survey includes 82 files with the name and no whole-site ban, out of 167 naming files.
What do the top-ranked sites show?
At the top of the ranking, 7.8% of readable files name it and 6.0% block it. Across all ranks, those shares are 2.7% and 1.4%.
Which nearby entry can I compare?
SemrushBot-SWA is named by 145 files and blocked by 343, or 5.6%. For this crawler, 167 files name it and 86 block it, a 1.4% share.