CragCrawler
CragCrawler has 24 recorded blocks in October 2026, or 0.4%, and 19 files naming it. The community reference lists its operator as CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services.
Who runs CragCrawler?
| Fact | CragCrawler reference entry |
|---|---|
| Operator | “CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services” |
| Purpose | “AI Data Providers” |
| Does CragCrawler obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for CragCrawler | No |
These facts quote the stored community ai.robots.txt reference.
“CragCrawler is a web scraping bot operated by CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services. The bot is used…”
Its listed role is supplying data for AI tools. Helping people find a page through search is a different use from using its contents to train a model, so owners may allow those visits separately.
How many sites block it?
Instructions to stay out of the whole site appear in 24 files, or 0.4% of 6,171 readable files. We counted 19 with its name in a User-agent line, a 0.3% share. Another 6,147 lack a recorded Disallow: / ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 3 naming the crawler, or 0.5%. Whole-site blocking appears in 3 of them. Blocking is more common among the leading sites, at 0.5%. Beyond those ranks, 5,619 files have a blocking total of 21, 0.4% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| CragCrawler, all ranks | 19 | 0.3% | 24 | 0.4% |
| CragCrawler, top 1,000 | 3 | 0.5% | 3 | 0.5% |
| CragCrawler, remaining ranks | 16 | 0.3% | 21 | 0.4% |
This total covers whole-site bans recorded for this name. A blanket User-agent: * rule alone is outside this measure. Of 24 blocks, 5 do not also appear in our count of files naming it. 19 files have the name plus a whole-site ban. Outside that set, 6,152 files never name it. Repeated user-agent mentions or capitalization variants are counted once.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a CragCrawler group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: CragCrawler
Disallow: /Allow access
User-agent: CragCrawler
Allow: /It's the 150th most-named crawler of the 171 we counted in the crawler index.
How we counted
For CragCrawler, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. An 8-second wait or a file over 262,144 bytes ended a read. Shares exclude unreachable, unavailable and oversized files. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
A ban accompanies each appearance of the name across 19 files.
How do the shares compare by rank?
Leading files record 0.5% with the name and 0.5% blocking it. Across all readable files, the corresponding shares are 0.3% and 0.4%.
What other entry is close in the index?
For comparison, Querit-SearchBot has 19 files naming it and 20 blocks, or 0.3%. Our counts for this entry are 19 files with the name and 24 blocking the whole site, or 0.4%.