Scrapy
Scrapy collects content for model training, and the community reference lists its operator as Zyte. Our October 2026 survey found 277 sites blocking it, or 4.5% of the files we could read.
Who runs Scrapy?
| Fact | Scrapy reference entry |
|---|---|
| Operator | “Zyte” |
| Purpose | “Scrapes data for a variety of uses including training AI.” |
| Does Scrapy obey robots.txt? | “Unclear at this time.” |
| Frequency | “No information.” |
| Cloudflare managed list for Scrapy | No |
These facts quote the stored community ai.robots.txt reference.
“"AI and machine learning applications often need large amounts of quality data, and web data extraction is a fast, efficient way to build structured data sets."”
Model training is this crawler’s listed use. Helping people find a page through search is a different use from using its contents to train a model, so owners may allow those visits separately.
How many sites block it?
A User-agent line names it in 287 files, 4.7% of 6,171 readable files. Whole-site bans, written Disallow: /, total 277, or 4.5%. That leaves 5,894 without a recorded ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 54 naming the crawler, or 9.8%. Whole-site blocking appears in 53 of them. Blocking is more common among the leading sites, at 9.6%. Beyond those ranks, 5,619 files have a blocking total of 224, 4.0% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| Scrapy, all ranks | 287 | 4.7% | 277 | 4.5% |
| Scrapy, top 1,000 | 54 | 9.8% | 53 | 9.6% |
| Scrapy, remaining ranks | 233 | 4.1% | 224 | 4.0% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. The naming total includes 277 files with a ban on every path and 10 files without that ban. Elsewhere, 5,884 files do not name it. We ignore capitalization and repeated names.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming Scrapy takes precedence over User-agent: *.
Block the whole site
User-agent: Scrapy
Disallow: /Allow access
User-agent: Scrapy
Allow: /It's the 29th most-named crawler of the 171 we counted in the crawler index.
How we counted
For Scrapy, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. An 8-second wait or a file over 262,144 bytes ended a read. Only available files we could read enter the percentages. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 287 files naming it include 10 files without a whole-site ban.
How do the shares compare by rank?
Its name appears in 9.8% of readable leading files and 4.7% overall. Blocking affects 9.6% in the leading group and 4.5% in all ranks.
What other entry is close in the index?
For comparison, Webzio-Extended has 281 files naming it and 270 blocks, or 4.4%. Our counts for this entry are 287 files with the name and 277 blocking the whole site, or 4.5%.