kagi-fetcher
kagi-fetcher is blocked by 90 sites in the October 2026 survey, or 1.5% of readable files. The community reference lists its operator as Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res….
Who runs kagi-fetcher?
| Fact | kagi-fetcher reference entry |
|---|---|
| Operator | “Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…” |
| Purpose | “AI Assistants” |
| Does kagi-fetcher obey robots.txt? | “Unclear at this time.” |
| Frequency | “Unclear at this time.” |
| Cloudflare managed list for kagi-fetcher | No |
These facts quote the stored community ai.robots.txt reference.
“kagi-fetcher is an AI Assistant operated by Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…”
The stated role here is an AI assistant. Indexing helps readers find pages, and a user’s request can fetch one. Training reuses the material in models, which an owner may choose to block.
How many sites block it?
Of 6,171 robots.txt files we could read, 90 block the whole site with Disallow: /, or 1.5%. Another way to read the sample is by name, with 29 files naming it in a User-agent line, or 0.5%. The 6,081 remaining files have no recorded whole-site block.
Across Tranco’s top 1,000 ranks, there were 552 readable files. Among them, 3 files carry the name, or 0.5%, with 10 files blocking the whole site. Blocking is more common among the leading sites, at 1.8%. Farther down, 80 of 5,619 files record a block, 1.4% of that group.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| kagi-fetcher, all ranks | 29 | 0.5% | 90 | 1.5% |
| kagi-fetcher, top 1,000 | 3 | 0.5% | 10 | 1.8% |
| kagi-fetcher, remaining ranks | 26 | 0.5% | 80 | 1.4% |
Here, the total counts whole-site blocks recorded for this crawler. It leaves out general User-agent: * instructions. Of 90 blocks, 65 do not also appear in our count of files naming it. Names and bans overlap in 25 files. We also found 4 files with a name and no whole-site ban, plus 6,142 files omitting the name. Case changes and repeated spellings are merged.
How to block or allow it
These lines go in robots.txt at the root. User-agent: * yields to a group naming kagi-fetcher under the robots.txt standard.
Block the whole site
User-agent: kagi-fetcher
Disallow: /Allow access
User-agent: kagi-fetcher
Allow: /It's the 140th most-named crawler of the 171 we counted in the crawler index.
How we counted
For kagi-fetcher, the survey covered 10,000 top Tranco sites, collected on 19 October 2026. Each file had 8 seconds to arrive, up to 262,144 bytes. Only available files we could read enter the percentages. Parsing includes files read again, through scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
We found 4 files without a whole-site ban among 29 that name it.
Are leading sites different?
At the top of the ranking, 0.5% of readable files name it and 1.8% block it. Across all ranks, those shares are 0.5% and 1.5%.
How does the nearby crawler compare?
Claude-Code is named by 28 files and blocked by 256, or 4.1%. For this crawler, 29 files name it and 90 block it, a 1.5% share.