YouBot
YouBot is listed as a search crawler, and the community reference lists its operator as You. In October 2026, 583 sites block it, a 9.4% share of readable files.
Who runs YouBot?
| Fact | YouBot reference entry |
|---|---|
| Operator | “You” |
| Purpose | “Scrapes data for search engine and LLMs.” |
| Does YouBot obey robots.txt? | “Yes” |
| Frequency | “No information.” |
| Cloudflare managed list for YouBot | No |
These facts quote the stored community ai.robots.txt reference.
“Retrieves data used for You.com web search engine and LLMs.”
The listed use here is search. An owner may want a page included in search and available on request without offering its contents for future model development.
How many sites block it?
Instructions to stay out of the whole site appear in 583 files, or 9.4% of 6,171 readable files. We counted 451 with its name in a User-agent line, a 7.3% share. Another 5,588 lack a recorded Disallow: / ban.
Breaking out the leading 1,000 sites gives 552 readable files, with 64 naming the crawler, or 11.6%. Whole-site blocking appears in 85 of them. Blocking is more common among the leading sites, at 15.4%. Beyond those ranks, 5,619 files have a blocking total of 498, 8.9% of that sample.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| YouBot, all ranks | 451 | 7.3% | 583 | 9.4% |
| YouBot, top 1,000 | 64 | 11.6% | 85 | 15.4% |
| YouBot, remaining ranks | 387 | 6.9% | 498 | 8.9% |
A general User-agent: * ban contributes to this total whenever it applies to this crawler. Of 583 blocks, 211 come only from that general rule, without naming this crawler. The split among files that name it is 372 with a whole-site block and 79 without one. Its name is absent from 5,720 readable files. We ignore capitalization and repeated names.
Has that changed since August?
Between August and October, 21 sites that had blocked it stopped and 53 started, so 32 more of the same 5,889 sites block it. A further 493 blocked it in both months, and 5,322 did so in neither. Keeping the sites the same lets us count changes in their rules.
August’s full sample had 540 blocks among 6,236 readable files. October’s was 583 among 6,171. Those monthly totals also reflect which files we could read on each visit.
| August to October | Files |
|---|---|
| Started blocking | 53 |
| Stopped blocking | 21 |
| Blocked in both | 493 |
| Blocked in neither | 5,322 |
| Net change | +32 |
How to block or allow it
Use your site’s root file, robots.txt. Specific groups for YouBot take precedence over User-agent: *, according to the robots.txt standard.
Block the whole site
User-agent: YouBot
Disallow: /Allow access
User-agent: YouBot
Allow: /It's the 19th most-named crawler of the 171 we counted in the crawler index.
How we counted
For YouBot, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Each file had 8 seconds to arrive, up to 262,144 bytes. Only available files we could read enter the percentages. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
Can a file name it and still allow it?
The 451 files naming it include 79 files without a whole-site ban.
How do the shares compare by rank?
Within readable robots.txt files near the top of the ranking, 11.6% name it and 15.4% block it. Overall, naming reaches 7.3% and blocking 9.4%.
What other entry is close in the index?
For comparison, facebookexternalhit has 435 files naming it and 23 blocks, or 0.4%. Our counts for this entry are 451 files with the name and 583 blocking the whole site, or 9.4%.