AI crawlers
AI crawler index
AI crawler rules in the October 2026 survey of Tranco’s top 10,000 sites, with counts from 6,171 readable robots.txt files.
Select a column heading to sort. Each row merges case and version spellings and needs at least 10 files naming it. Existing crawler articles are linked at their current addresses. Purpose and operator are quoted from the saved community ai.robots.txt reference.
| AddSearchBot | Unclear at this time.· named by 67 (1.1%) · blocked by 68 (1.1%) | AI Search Crawlers | 67 | 68 |
| AgentDataBot | Unclear at this time.· named by 16 (0.3%) · blocked by 16 (0.3%) | AI Data Providers | 16 | 16 |
| AI2Bot | Ai2· named by 278 (4.5%) · blocked by 251 (4.1%) | Content is used to train open language models. | 278 | 251 |
| AI2Bot-DeepResearchEval | Ai2, a non-profit AI research institute· named by 70 (1.1%) · blocked by 258 (4.2%) | AI Assistants | 70 | 258 |
| Ai2Bot-Dolma | Ai2· named by 177 (2.9%) · blocked by 271 (4.4%) | Content is used to train open language models. | 177 | 271 |
| aiHitBot | aiHit· named by 98 (1.6%) · blocked by 94 (1.5%) | A massive, artificial intelligence/machine learning, automated system. | 98 | 94 |
| AIWebIndex | Lyrenth· named by 18 (0.3%) · blocked by 24 (0.4%) | AI Search Crawlers | 18 | 24 |
| amazon-kendra | Amazon· named by 63 (1.0%) · blocked by 58 (0.9%) | Collects data for AI natural language search | 63 | 58 |
| amazon-QBusiness | Unclear at this time.· named by 37 (0.6%) · blocked by 44 (0.7%) | AI Assistants | 37 | 44 |
| Amazonbot | Amazon· named by 666 (10.8%) · blocked by 719 (11.7%) | Service improvement and enabling answers for Alexa users. | 666 | 719 |
| AmazonBuyForMe | Amazon· named by 56 (0.9%) · blocked by 52 (0.8%) | AI Agents | 56 | 52 |
| Amzn-SearchBot | Unclear at this time.· named by 109 (1.8%) · blocked by 307 (5.0%) | AI Search Crawlers | 109 | 307 |
| Amzn-User | Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services· named by 94 (1.5%) · blocked by 76 (1.2%) | AI Assistants | 94 | 76 |
| Andibot | Andi· named by 70 (1.1%) · blocked by 54 (0.9%) | Search engine using generative AI, AI Search Assistant | 70 | 54 |
| Anomura | Direqt· named by 45 (0.7%) · blocked by 44 (0.7%) | Collects data for AI search | 45 | 44 |
| anthropic-ai | Anthropic· named by 642 (10.4%) · blocked by 718 (11.6%) | Scrapes data to train Anthropic's AI products. | 642 | 718 |
| ApifyBot | Unclear at this time.· named by 49 (0.8%) · blocked by 46 (0.7%) | AI Data Providers | 49 | 46 |
| ApifyWebsiteContentCrawler | Unclear at this time.· named by 46 (0.7%) · blocked by 51 (0.8%) | AI Data Providers | 46 | 51 |
| Applebot | Unclear at this time.· named by 276 (4.5%) · blocked by 264 (4.3%) | AI Search Crawlers | 276 | 264 |
| Applebot-Extended | Apple· named by 658 (10.7%) · blocked by 694 (11.2%) | Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others. | 658 | 694 |
| Aranet-SearchBot | Unclear at this time.· named by 16 (0.3%) · blocked by 16 (0.3%) | Undocumented AI Agents | 16 | 16 |
| atlassian-bot | Atlassian· named by 110 (1.8%) · blocked by 103 (1.7%) | AI search, assistants and agents | 110 | 103 |
| Awario | Awario· named by 32 (0.5%) · blocked by 26 (0.4%) | AI Data Scrapers | 32 | 26 |
| AzureAI-SearchBot | Unclear at this time.· named by 29 (0.5%) · blocked by 259 (4.2%) | AI Search Crawlers | 29 | 259 |
| bedrockbot | Amazon· named by 60 (1.0%) · blocked by 53 (0.9%) | Data scraping for custom AI applications. | 60 | 53 |
| bigsur.ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi…· named by 50 (0.8%) · blocked by 52 (0.8%) | AI Assistants | 50 | 52 |
| BixelBot | Unclear at this time.· named by 16 (0.3%) · blocked by 16 (0.3%) | AI Data Providers | 16 | 16 |
| Bravebot | https://safe.search.brave.com/help/brave-search-crawler· named by 90 (1.5%) · blocked by 81 (1.3%) | AI Data Providers | 90 | 81 |
| Brightbot | https://brightdata.com/brightbot· named by 167 (2.7%) · blocked by 86 (1.4%) | LLM/AI training. | 167 | 86 |
| BuddyBot | BuddyBotLearning· named by 35 (0.6%) · blocked by 43 (0.7%) | AI Learning Companion | 35 | 43 |
| Bytespider | ByteDance· named by 826 (13.4%) · blocked by 930 (15.1%) | LLM training. | 826 | 930 |
| CCBot | Common Crawl Foundation· named by 934 (15.1%) · blocked by 945 (15.3%) | Provides open crawl dataset, used for many purposes, including Machine Learning/AI. | 934 | 945 |
| Channel3Bot | Unclear at this time.· named by 57 (0.9%) · blocked by 55 (0.9%) | AI Search Crawlers | 57 | 55 |
| ChatGLM-Spider | Unclear at this time.· named by 117 (1.9%) · blocked by 123 (2.0%) | AI Data Scrapers | 117 | 123 |
| ChatGPT Agent | OpenAI· named by 36 (0.6%) · blocked by 0 (0.0%) | AI Agents | 36 | 0 |
| ChatGPT-User | OpenAI· named by 728 (11.8%) · blocked by 541 (8.8%) | AI Assistants | 728 | 541 |
| Claude-Code | Unclear at this time.· named by 28 (0.5%) · blocked by 256 (4.1%) | AI Coding Agents | 28 | 256 |
| Claude-SearchBot | Anthropic· named by 394 (6.4%) · blocked by 417 (6.8%) | Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses. | 394 | 417 |
| Claude-User | Anthropic· named by 388 (6.3%) · blocked by 426 (6.9%) | AI Assistants | 388 | 426 |
| Claude-Web | Anthropic· named by 511 (8.3%) · blocked by 619 (10.0%) | Undocumented AI Agents | 511 | 619 |
| ClaudeBot | Anthropic· named by 946 (15.3%) · blocked by 835 (13.5%) | Scrapes data to train Anthropic's AI products. | 946 | 835 |
| Cloudflare-AutoRAG | Cloudflare· named by 54 (0.9%) · blocked by 52 (0.8%) | Collects data for AI search | 54 | 52 |
| CloudflareBrowserRenderingCrawler | Cloudflare that returns rendered website content for research, monitoring, and AI data workflows· named by 65 (1.1%) · blocked by 68 (1.1%) | AI Data Providers | 65 | 68 |
| CloudVertexBot | Unclear at this time.· named by 58 (0.9%) · blocked by 57 (0.9%) | AI Data Scrapers | 58 | 57 |
| Code | Unclear at this time.· named by 16 (0.3%) · blocked by 15 (0.2%) | AI Coding Agents | 16 | 15 |
| cohere-ai | Cohere· named by 587 (9.5%) · blocked by 722 (11.7%) | Retrieves data to provide responses to user-initiated prompts. | 587 | 722 |
| cohere-training-data-crawler | Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products· named by 250 (4.1%) · blocked by 246 (4.0%) | AI Data Scrapers | 250 | 246 |
| Cotoyogi | ROIS· named by 177 (2.9%) · blocked by 170 (2.8%) | AI LLM Scraper. | 177 | 170 |
| CragCrawler | CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services· named by 19 (0.3%) · blocked by 24 (0.4%) | AI Data Providers | 19 | 24 |
| Crawl4AI | Unclear at this time.· named by 69 (1.1%) · blocked by 71 (1.2%) | Undocumented AI Agents | 69 | 71 |
| Crawlspace | Crawlspace· named by 61 (1.0%) · blocked by 58 (0.9%) | AI Data Providers | 61 | 58 |
| Cursor | Unclear at this time.· named by 17 (0.3%) · blocked by 14 (0.2%) | AI Coding Agents | 17 | 14 |
| Datenbank Crawler | Datenbank· named by 88 (1.4%) · blocked by 0 (0.0%) | AI Data Scrapers | 88 | 0 |
| DeepSeekBot | DeepSeek· named by 180 (2.9%) · blocked by 387 (6.3%) | Training language models and improving AI products | 180 | 387 |
| Devin | Devin AI· named by 79 (1.3%) · blocked by 69 (1.1%) | AI Coding Agents | 79 | 69 |
| Diffbot | Diffbot· named by 553 (9.0%) · blocked by 723 (11.7%) | AI Data Providers | 553 | 723 |
| Diffbot-User | Diffbot· named by 28 (0.5%) · blocked by 513 (8.3%) | AI Assistants | 28 | 513 |
| DoubaoBot | ByteDance· named by 15 (0.2%) · blocked by 28 (0.5%) | AI crawler for ByteDance's Doubao AI assistant. | 15 | 28 |
| DuckAssistBot | Unclear at this time.· named by 320 (5.2%) · blocked by 237 (3.8%) | AI Assistants | 320 | 237 |
| Echobot Bot | Echobox· named by 62 (1.0%) · blocked by 0 (0.0%) | AI Data Scrapers | 62 | 0 |
| EchoboxBot | Echobox· named by 74 (1.2%) · blocked by 70 (1.1%) | Data collection to support AI-powered products. | 74 | 70 |
| ERNIEBot | Baidu· named by 38 (0.6%) · blocked by 39 (0.6%) | Collects public web content for Baidu's ERNIE large language models. | 38 | 39 |
| ExaBot | Unclear at this time.· named by 141 (2.3%) · blocked by 334 (5.4%) | AI Data Providers | 141 | 334 |
| ExaSearchBot | Exa· named by 42 (0.7%) · blocked by 40 (0.6%) | AI Search Crawlers | 42 | 40 |
| FacebookBot | Meta/Facebook· named by 452 (7.3%) · blocked by 370 (6.0%) | Training language models | 452 | 370 |
| facebookexternalhit | Meta/Facebook· named by 435 (7.0%) · blocked by 23 (0.4%) | Ostensibly only for sharing, but likely used as an AI crawler as well | 435 | 23 |
| Factset_spyderbot | Factset· named by 96 (1.6%) · blocked by 92 (1.5%) | AI model training. | 96 | 92 |
| FirecrawlAgent | Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications· named by 143 (2.3%) · blocked by 347 (5.6%) | AI Data Providers | 143 | 347 |
| FriendlyCrawler | Unknown· named by 201 (3.3%) · blocked by 196 (3.2%) | We are using the data from the crawler to build datasets for machine learning experiments. | 201 | 196 |
| GeistHaus-PageFetcher | GeistHaus, a company developing AI systems for therapy and psychological assessment· named by 22 (0.4%) · blocked by 85 (1.4%) | AI Assistants | 22 | 85 |
| Gemini-Deep-Research | Unclear at this time.· named by 108 (1.8%) · blocked by 312 (5.1%) | AI Assistants | 108 | 312 |
| Google-Agent | Unclear at this time.· named by 30 (0.5%) · blocked by 19 (0.3%) | AI Agents | 30 | 19 |
| Google-CloudVertexBot | Google· named by 275 (4.5%) · blocked by 462 (7.5%) | Build and manage AI models for businesses employing Vertex AI | 275 | 462 |
| Google-Extended | Google· named by 875 (14.2%) · blocked by 748 (12.1%) | LLM training. | 875 | 748 |
| Google-Firebase | Google· named by 43 (0.7%) · blocked by 38 (0.6%) | Used as part of AI apps developed by users of Google's Firebase AI products. | 43 | 38 |
| Google-NotebookLM | Unclear at this time.· named by 79 (1.3%) · blocked by 278 (4.5%) | AI Assistants | 79 | 278 |
| GoogleAgent-Mariner | Google· named by 67 (1.1%) · blocked by 57 (0.9%) | AI Agents | 67 | 57 |
| GoogleOther | Google· named by 232 (3.8%) · blocked by 371 (6.0%) | Scrapes data. | 232 | 371 |
| GoogleOther-Image | Google· named by 57 (0.9%) · blocked by 177 (2.9%) | Scrapes data. | 57 | 177 |
| GoogleOther-Video | Google· named by 56 (0.9%) · blocked by 177 (2.9%) | Scrapes data. | 56 | 177 |
| GPTBot | OpenAI· named by 1,108 (18.0%) · blocked by 906 (14.7%) | Scrapes data to train OpenAI's products. | 1,108 | 906 |
| GrokBot | xAI· named by 68 (1.1%) · blocked by 73 (1.2%) | AI Search Crawlers | 68 | 73 |
| HenkBot | Unclear at this time.· named by 18 (0.3%) · blocked by 19 (0.3%) | AI Data Providers | 18 | 19 |
| Hunyuan | Tencent· named by 15 (0.2%) · blocked by 14 (0.2%) | Hunyuan is a training crawler operated by Tencent. | 15 | 14 |
| iAskBot | Unclear at this time.· named by 88 (1.4%) · blocked by 83 (1.3%) | Undocumented AI Agents | 88 | 83 |
| iaskspider | iAsk· named by 138 (2.2%) · blocked by 120 (1.9%) | Crawls sites to provide answers to user queries. | 138 | 120 |
| ICC-Crawler | NICT· named by 230 (3.7%) · blocked by 229 (3.7%) | Scrapes data to train and support AI technologies. | 230 | 229 |
| ImagesiftBot | ImageSift· named by 332 (5.4%) · blocked by 532 (8.6%) | ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support their suite of web intelligence products | 332 | 532 |
| imageSpider | Unclear at this time.· named by 90 (1.5%) · blocked by 104 (1.7%) | AI Data Scrapers | 90 | 104 |
| img2dataset | img2dataset· named by 200 (3.2%) · blocked by 181 (2.9%) | Scrapes images for use in LLMs. | 200 | 181 |
| ISSCyberRiskCrawler | ISS-Corporate· named by 109 (1.8%) · blocked by 111 (1.8%) | Scrapes data to train machine learning models. | 109 | 111 |
| kagi-fetcher | Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…· named by 29 (0.5%) · blocked by 90 (1.5%) | AI Assistants | 29 | 90 |
| Kangaroo Bot | Unclear at this time.· named by 172 (2.8%) · blocked by 0 (0.0%) | AI Data Scrapers | 172 | 0 |
| KeenableBot | Keenable· named by 17 (0.3%) · blocked by 17 (0.3%) | AI Search Crawlers | 17 | 17 |
| Kimi-Agent | Moonshot AI that browses websites while carrying out user-directed research and other tasks· named by 12 (0.2%) · blocked by 13 (0.2%) | AI Agents | 12 | 13 |
| Kimi-SearchBot | Moonshot AI· named by 17 (0.3%) · blocked by 18 (0.3%) | AI Search Crawlers | 17 | 18 |
| Kimi-User | Moonshot AI that fetches web content on behalf of users interacting with Kimi· named by 30 (0.5%) · blocked by 253 (4.1%) | AI Assistants | 30 | 253 |
| KimiBot | Moonshot AI· named by 76 (1.2%) · blocked by 77 (1.2%) | AI Data Scrapers | 76 | 77 |
| KlaviyoAIBot | Klaviyo· named by 59 (1.0%) · blocked by 69 (1.1%) | AI Assistants | 59 | 69 |
| KunatoCrawler | Unclear at this time.· named by 66 (1.1%) · blocked by 70 (1.1%) | Undocumented AI Agents | 66 | 70 |
| laion-huggingface-processor | Unclear at this time.· named by 72 (1.2%) · blocked by 93 (1.5%) | AI Data Scrapers | 72 | 93 |
| LAIONDownloader | Large-scale Artificial Intelligence Open Network· named by 28 (0.5%) · blocked by 34 (0.6%) | AI tools and models for machine learning research. | 28 | 34 |
| LCC | Unclear at this time.· named by 48 (0.8%) · blocked by 46 (0.7%) | AI Data Scrapers | 48 | 46 |
| LinerBot | Unclear at this time.· named by 83 (1.3%) · blocked by 77 (1.2%) | AI Assistants | 83 | 77 |
| Linguee Bot | Linguee· named by 41 (0.7%) · blocked by 0 (0.0%) | AI powered translation service | 41 | 0 |
| LinkupBot | Unclear at this time.· named by 69 (1.1%) · blocked by 68 (1.1%) | AI Search Crawlers | 69 | 68 |
| Manus-User | Butterfly Effect, a company based in China· named by 60 (1.0%) · blocked by 56 (0.9%) | AI Agents | 60 | 56 |
| Meta-ExternalAgent | Meta· named by 676 (11.0%) · blocked by 738 (12.0%) | Used to train models and improve products. | 676 | 738 |
| Meta-ExternalFetcher | Unclear at this time.· named by 298 (4.8%) · blocked by 509 (8.2%) | AI Assistants | 298 | 509 |
| meta-webindexer | Meta· named by 174 (2.8%) · blocked by 154 (2.5%) | AI Assistants | 174 | 154 |
| MistralAI-Index | Mistral AI· named by 43 (0.7%) · blocked by 54 (0.9%) | Indexes web content for Mistral AI's search, used to answer questions in Le Chat. Per Mistral, not used for model training. | 43 | 54 |
| MistralAI-Training | Mistral AI· named by 70 (1.1%) · blocked by 86 (1.4%) | AI Data Scrapers | 70 | 86 |
| MistralAI-User | Mistral· named by 244 (4.0%) · blocked by 426 (6.9%) | AI Assistants | 244 | 426 |
| MoonshotBot | Kimi· named by 16 (0.3%) · blocked by 16 (0.3%) | MoonshotBot is a training crawler. | 16 | 16 |
| Mozilla-Tabstack | Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack· named by 15 (0.2%) · blocked by 17 (0.3%) | AI Data Providers | 15 | 17 |
| MyCentralAIScraperBot | Unclear at this time.· named by 51 (0.8%) · blocked by 52 (0.8%) | AI data scraper | 51 | 52 |
| netEstate Imprint Crawler | netEstate· named by 86 (1.4%) · blocked by 0 (0.0%) | AI Data Scrapers | 86 | 0 |
| NotebookLM | Unclear at this time.· named by 11 (0.2%) · blocked by 8 (0.1%) | AI Assistants | 11 | 8 |
| NovaAct | Unclear at this time.· named by 65 (1.1%) · blocked by 56 (0.9%) | AI Agents | 65 | 56 |
| OAI-AdsBot | OpenAI· named by 36 (0.6%) · blocked by 20 (0.3%) | Validates and targets ads on ChatGPT. | 36 | 20 |
| OAI-SearchBot | OpenAI· named by 608 (9.9%) · blocked by 427 (6.9%) | Search result generation. | 608 | 427 |
| omgili | Webz.io· named by 519 (8.4%) · blocked by 703 (11.4%) | Data is sold. | 519 | 703 |
| omgilibot | Webz.io· named by 395 (6.4%) · blocked by 523 (8.5%) | Data is sold. | 395 | 523 |
| OpenAI | OpenAI· named by 30 (0.5%) · blocked by 32 (0.5%) | Unclear at this time. | 30 | 32 |
| opencode | Unclear at this time.· named by 18 (0.3%) · blocked by 16 (0.3%) | AI Coding Agents | 18 | 16 |
| Operator | Unclear at this time.· named by 44 (0.7%) · blocked by 36 (0.6%) | AI Agents | 44 | 36 |
| PanguBot | the Chinese company Huawei· named by 205 (3.3%) · blocked by 193 (3.1%) | AI Data Scrapers | 205 | 193 |
| Panscient | Panscient· named by 77 (1.2%) · blocked by 75 (1.2%) | Data collection and analysis using machine learning and AI. | 77 | 75 |
| Perplexity-User | Perplexity· named by 383 (6.2%) · blocked by 440 (7.1%) | AI Assistants | 383 | 440 |
| PerplexityBot | Perplexity· named by 784 (12.7%) · blocked by 622 (10.1%) | Search result generation. | 784 | 622 |
| PetalBot | Huawei· named by 487 (7.9%) · blocked by 433 (7.0%) | Used to provide recommendations in Hauwei assistant and AI search services. | 487 | 433 |
| PhindBot | phind· named by 96 (1.6%) · blocked by 76 (1.2%) | AI Assistants | 96 | 76 |
| Poggio-Citations | Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan…· named by 58 (0.9%) · blocked by 57 (0.9%) | AI Assistants | 58 | 57 |
| Poseidon Research Crawler | Poseidon Research· named by 42 (0.7%) · blocked by 0 (0.0%) | AI research crawler | 42 | 0 |
| qodercli | Qoder that helps developers build, debug, and modify software from the terminal· named by 12 (0.2%) · blocked by 11 (0.2%) | AI Coding Agents | 12 | 11 |
| QualifiedBot | Qualified· named by 141 (2.3%) · blocked by 136 (2.2%) | AI Assistants | 141 | 136 |
| Querit-SearchBot | Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg…· named by 19 (0.3%) · blocked by 20 (0.3%) | AI Data Providers | 19 | 20 |
| QueritBot | Querit, a company providing a search API for large language model integration· named by 19 (0.3%) · blocked by 20 (0.3%) | AI Data Providers | 19 | 20 |
| QuillBot | Quillbot· named by 62 (1.0%) · blocked by 62 (1.0%) | Company offers AI detection, writing tools and other services. | 62 | 62 |
| QwenBot | Alibaba· named by 20 (0.3%) · blocked by 20 (0.3%) | Collects public web content for Alibaba's Qwen (Tongyi Qianwen) large language models. | 20 | 20 |
| Reflectionbot | Reflection· named by 22 (0.4%) · blocked by 22 (0.4%) | Undocumented AI Agents | 22 | 22 |
| SBIntuitionsBot | SB Intuitions· named by 107 (1.7%) · blocked by 105 (1.7%) | Uses data gathered in AI development and information analysis. | 107 | 105 |
| Scrapy | Zyte· named by 287 (4.7%) · blocked by 277 (4.5%) | Scrapes data for a variety of uses including training AI. | 287 | 277 |
| SemrushBot-OCOB | Semrush· named by 80 (1.3%) · blocked by 275 (4.5%) | Crawls your site for ContentShake AI tool. | 80 | 275 |
| SemrushBot-SWA | Semrush· named by 145 (2.3%) · blocked by 343 (5.6%) | Checks URLs on your site for SEO Writing Assistant. | 145 | 343 |
| Shap-User | Unclear at this time.· named by 61 (1.0%) · blocked by 60 (1.0%) | AI Assistants | 61 | 60 |
| ShapBot | Parallel· named by 123 (2.0%) · blocked by 120 (1.9%) | AI Data Providers | 123 | 120 |
| Sidetrade indexer bot | Sidetrade· named by 83 (1.3%) · blocked by 0 (0.0%) | Extracts data for a variety of uses including training AI. | 83 | 0 |
| Spider | Unclear at this time.· named by 52 (0.8%) · blocked by 50 (0.8%) | AI Data Scrapers | 52 | 50 |
| TavilyBot | Unclear at this time.· named by 72 (1.2%) · blocked by 292 (4.7%) | AI Data Providers | 72 | 292 |
| TerraCotta | Ceramic AI· named by 73 (1.2%) · blocked by 70 (1.1%) | AI Data Providers | 73 | 70 |
| Thinkbot | Thinkbot· named by 62 (1.0%) · blocked by 63 (1.0%) | Insights on AI integration and automation. | 62 | 63 |
| TikTokSpider | ByteDance· named by 99 (1.6%) · blocked by 334 (5.4%) | LLM training. | 99 | 334 |
| Timpibot | Timpi· named by 349 (5.7%) · blocked by 550 (8.9%) | Scrapes data for use in training LLMs. | 349 | 550 |
| TongyiBot | Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers· named by 24 (0.4%) · blocked by 27 (0.4%) | AI Assistants | 24 | 27 |
| Trae | Unclear at this time.· named by 16 (0.3%) · blocked by 15 (0.2%) | AI Coding Agents | 16 | 15 |
| TwinAgent | Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa…· named by 55 (0.9%) · blocked by 53 (0.9%) | AI Agents | 55 | 53 |
| UseAI | Unclear at this time.· named by 21 (0.3%) · blocked by 25 (0.4%) | AI Assistants | 21 | 25 |
| VelenPublicWebCrawler | Velen Crawler· named by 200 (3.2%) · blocked by 194 (3.1%) | Scrapes data for business data sets and machine learning models. | 200 | 194 |
| WARDBot | WEBSPARK· named by 92 (1.5%) · blocked by 92 (1.5%) | AI Data Scrapers | 92 | 92 |
| Webzio-Extended | Unclear at this time.· named by 281 (4.6%) · blocked by 270 (4.4%) | AI Data Scrapers | 281 | 270 |
| wpbot | QuantumCloud· named by 63 (1.0%) · blocked by 64 (1.0%) | Live chat support and lead generation. | 63 | 64 |
| WRTNBot | Unclear at this time.· named by 70 (1.1%) · blocked by 68 (1.1%) | Undocumented AI Agents | 70 | 68 |
| xAI-Grok | xAI· named by 46 (0.7%) · blocked by 64 (1.0%) | xAI-Grok is an xAI crawler token associated with Grok content collection. | 46 | 64 |
| YaK | Meltwater· named by 77 (1.2%) · blocked by 71 (1.2%) | According to the Meltwater Consumer Intelligence page 'By applying AI, data science, and market research expertise to a live feed of global data sources, we transform unstructured data into actionable insights allowing better decision-making'. | 77 | 71 |
| YandexAdditional | Yandex· named by 63 (1.0%) · blocked by 164 (2.7%) | Scrapes/analyzes data for the YandexGPT LLM. | 63 | 164 |
| YandexAdditionalBot | Yandex· named by 71 (1.2%) · blocked by 174 (2.8%) | Scrapes/analyzes data for the YandexGPT LLM. | 71 | 174 |
| YiBot | 01.ai, an AI company founded by Kai-Fu Lee.· named by 17 (0.3%) · blocked by 16 (0.3%) | This bot crawls the web to gather data and content for indexation and AI model training. | 17 | 16 |
| YiyanBot | Baidu that fetches web content for the yiyan· named by 22 (0.4%) · blocked by 22 (0.4%) | AI Assistants | 22 | 22 |
| YouBot | You· named by 451 (7.3%) · blocked by 583 (9.4%) | Scrapes data for search engine and LLMs. | 451 | 583 |
| ZanistaBot | Unclear at this time.· named by 58 (0.9%) · blocked by 57 (0.9%) | AI Search Crawlers | 58 | 57 |
Named by counts files naming the crawler in a User-agent line. Blocked by counts instructions to stay out of the whole site. Some totals include rules for every crawler, while others count named rules alone. Each linked page explains which count it uses, how many files we could read, and how the leading sites compare. Read the survey method.