omgilibot
omgilibot is blocked by 523 sites in the October 2026 survey, or 8.5% of readable files. The community reference lists its operator as Webz.io.
Who runs omgilibot?
| Fact | omgilibot reference entry |
|---|---|
| Operator | “Webz.io” |
| Purpose | “Data is sold.” |
| Does omgilibot obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for omgilibot | No |
These facts quote the stored community ai.robots.txt reference.
“Legacy user agent initially used for Omgili search engine. Unknown if still used, `omgili` agent still used by Webz.io.”
This entry’s stated use is quoted above. An owner may permit an assistant to read a page for someone and deny bulk collection that would use it as training material.
How many sites block it?
Instructions to stay out of the whole site appear in 523 files, or 8.5% of 6,171 readable files. We counted 395 with its name in a User-agent line, a 6.4% share. Another 5,648 lack a recorded Disallow: / ban.
At the top 1,000 ranks, we could read 552 files. Naming covers 54, a 9.8% share, and blocking covers 70. Blocking is more common among the leading sites, at 12.7%. Elsewhere, 5,619 readable files give 453 blocks, or 8.1%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| omgilibot, all ranks | 395 | 6.4% | 523 | 8.5% |
| omgilibot, top 1,000 | 54 | 9.8% | 70 | 12.7% |
| omgilibot, remaining ranks | 341 | 6.1% | 453 | 8.1% |
Here, the total counts whole-site blocks recorded for this crawler. It leaves out general User-agent: * instructions. Of 523 blocks, 157 do not also appear in our count of files naming it. Names and bans overlap in 366 files. We also found 29 files with a name and no whole-site ban, plus 5,776 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for omgilibot overrides User-agent: *.
Block the whole site
User-agent: omgilibot
Disallow: /Allow access
User-agent: omgilibot
Allow: /It's the 21st most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For omgilibot, we surveyed Tranco’s top 10,000 sites on 19 October 2026. Files had an 8-second timeout and a 262,144-byte limit. Shares exclude unreachable, unavailable and oversized files. The file readings, including repeats, use scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
This survey includes 29 files with the name and no whole-site ban, out of 395 naming files.
Are leading sites different?
For naming rules, 9.8% of the leading sample mentions it, compared with 6.4% overall. Whole-site blocking is 12.7% at leading ranks and 8.5% across the full sample.
How does the nearby crawler compare?
For comparison, Claude-SearchBot has 394 files naming it and 417 blocks, or 6.8%. Our counts for this entry are 395 files with the name and 523 blocking the whole site, or 8.5%.