omgili
omgili is blocked by 703 sites in the October 2026 survey, or 11.4% of readable files. The community reference lists its operator as Webz.io.
Who runs omgili?
| Fact | omgili reference entry |
|---|---|
| Operator | “Webz.io” |
| Purpose | “Data is sold.” |
| Does omgili obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for omgili | No |
These facts quote the stored community ai.robots.txt reference.
“Crawls sites for APIs used by Hootsuite, Sprinklr, NetBase, and other companies. Data also sold for research purposes or LLM training.”
This entry’s stated use is quoted above. Allowing people to find and read a page need not include permission to gather it for model training. Owners can express those choices in crawler-specific groups.
How many sites block it?
The 703 files blocking the whole site account for 11.4% of 6,171 readable robots.txt files. We found its name in 519 User-agent lists, a 8.4% share. Without a whole-site Disallow: / block, 5,468 files remain.
For the top 1,000 ranks, its name appears in 71 files out of 552, or 12.9%. Whole-site blocks total 103. Blocking is more common among the leading sites, at 18.7%. Lower in the ranking, blocking reaches 600 files out of 5,619, a 10.7% share.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| omgili, all ranks | 519 | 8.4% | 703 | 11.4% |
| omgili, top 1,000 | 71 | 12.9% | 103 | 18.7% |
| omgili, remaining ranks | 448 | 8.0% | 600 | 10.7% |
Rules for every crawler, written User-agent: *, count here too. Of 703 blocks, 214 come only from that general rule, without naming this crawler. We found 489 files naming it and blocking the whole site, along with 30 files naming it without that ban. Another 5,652 files omit the name. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 27 sites that had blocked it stopped and 80 started, so 53 more of the same 5,889 sites block it. A further 579 blocked it in both months, and 5,203 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 632 blocks among 6,236 readable files. October’s was 703 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 80 |
| Stopped blocking | 27 |
| Blocked in both | 579 |
| Blocked in neither | 5,203 |
| Net change | +53 |
How to block or allow it
Place the group in your root robots.txt file. Per the robots.txt standard, a named group for omgili overrides User-agent: *.
Block the whole site
User-agent: omgili
Disallow: /Allow access
User-agent: omgili
Allow: /It's the 15th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For omgili, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. A request timed out after 8 seconds, with files capped at 262,144 bytes. Shares exclude unreachable, unavailable and oversized files. scripts/robots_2026_08_06.py (robots.py at app commit 07104ff) parsed both initial and repeat readings. See the full survey method.
Common questions
What does naming this crawler tell me?
A name identifies the crawler addressed by a group. The survey found 30 files with that name and no recorded whole-site ban, out of 519.
Are leading sites different?
Leading files record 12.9% with the name and 18.7% blocking it. Across all readable files, the corresponding shares are 8.4% and 11.4%.
How does the nearby crawler compare?
For comparison, Claude-Web has 511 files naming it and 619 blocks, or 10.0%. Our counts for this entry are 519 files with the name and 703 blocking the whole site, or 11.4%.