VelenPublicWebCrawler
VelenPublicWebCrawler is blocked by 194 sites in the October 2026 survey, or 3.1% of readable files. The community reference lists its operator as Velen Crawler.
Who runs VelenPublicWebCrawler?
| Fact | VelenPublicWebCrawler reference entry |
|---|---|
| Operator | “Velen Crawler” |
| Purpose | “Scrapes data for business data sets and machine learning models.” |
| Does VelenPublicWebCrawler obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for VelenPublicWebCrawler | No |
These facts quote the stored community ai.robots.txt reference.
“"Our goal with this crawler is to build business datasets and machine learning models to better understand the web."”
This entry’s stated use is quoted above. Indexing helps readers find pages, and a user’s request can fetch one. Training reuses the material in models, which an owner may choose to block.
How many sites block it?
Within 6,171 readable robots.txt files, the name appears in 200, or 3.2%. Instructions blocking every path, Disallow: /, appear in 194, a 3.1% share. No whole-site ban is recorded for 5,977 files.
From the highest 1,000 ranks, 552 files could be read. Its naming count is 33, or 6.0%, and the block count is 31. Blocking is more common among the leading sites, at 5.6%. Compare 163 blocks among the 5,619 readable files below those ranks, or 2.9%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| VelenPublicWebCrawler, all ranks | 200 | 3.2% | 194 | 3.1% |
| VelenPublicWebCrawler, top 1,000 | 33 | 6.0% | 31 | 5.6% |
| VelenPublicWebCrawler, remaining ranks | 167 | 3.0% | 163 | 2.9% |
Blocking here counts whole-site bans recorded for it by name. A general ban for all crawlers is excluded. Of 194 blocks, 3 do not also appear in our count of files naming it. Names and bans overlap in 191 files. We also found 9 files with a name and no whole-site ban, plus 5,971 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
Put these in robots.txt at the site root. Per the robots.txt standard, a group naming VelenPublicWebCrawler takes precedence over User-agent: *.
Block the whole site
User-agent: VelenPublicWebCrawler
Disallow: /Allow access
User-agent: VelenPublicWebCrawler
Allow: /It's the 41st most-named crawler of the 171 we counted in the crawler index.
How we counted
For VelenPublicWebCrawler, our 19 October 2026 collection used Tranco’s top 10,000 sites. Files had an 8-second timeout and a 262,144-byte limit. Shares exclude unreachable, unavailable and oversized files. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
This survey includes 9 files with the name and no whole-site ban, out of 200 naming files.
How do the shares compare by rank?
Its name appears in 6.0% of readable leading files and 3.2% overall. Blocking affects 5.6% in the leading group and 3.1% in all ranks.
What other entry is close in the index?
The nearby DeepSeekBot entry reports 180 files that name it and 387 whole-site blocks, 6.3% of readable files. Here, 200 files mention the name and 194 block access, or 3.1%.