SemrushBot-SWA
SemrushBot-SWA is blocked by 343 sites in the October 2026 survey, or 5.6% of readable files. The community reference lists its operator as Semrush.
Who runs SemrushBot-SWA?
| Fact | SemrushBot-SWA reference entry |
|---|---|
| Operator | “Semrush” |
| Purpose | “Checks URLs on your site for SEO Writing Assistant.” |
| Does SemrushBot-SWA obey robots.txt? | “Yes” |
| Frequency | “Roughly once every 10 seconds.” |
| Cloudflare managed list for SemrushBot-SWA | No |
These facts quote the stored community ai.robots.txt reference.
“Data collected is used for the SEO Writing Assistant tool to check if URL is accessible.”
The stated role here is an AI assistant. Search indexing is aimed at helping people find pages. Feeding text into model training is another use an owner may decline.
How many sites block it?
Our readable sample has 343 whole-site blocks, 5.6% of 6,171 files, with Disallow: / as the instruction. Of that full sample, 145 name this crawler, or 2.3%. The count without a whole-site block is 5,828.
Among 552 readable files from the top 1,000 sites, we found 23 files naming it, a 4.2% share, with 33 files blocking the whole site. Blocking is more common among the leading sites, at 6.0%. Outside this group, 5,619 files contribute 310 blocks, or 5.5%.
| Readable files | Named by | Named share | Blocked by | Blocked share |
|---|---|---|---|---|
| SemrushBot-SWA, all ranks | 145 | 2.3% | 343 | 5.6% |
| SemrushBot-SWA, top 1,000 | 23 | 4.2% | 33 | 6.0% |
| SemrushBot-SWA, remaining ranks | 122 | 2.2% | 310 | 5.5% |
Only whole-site blocks recorded against this crawler’s name count here. Rules addressed to every crawler are outside this count. Of 343 blocks, 205 do not also appear in our count of files naming it. Looking just at named files gives 138 blocks and 7 without a whole-site block. For 6,026 readable files, we found no mention by name. We ignore capitalization and repeated names.
How to block or allow it
Edit the root robots.txt file on your site. The robots.txt standard gives a SemrushBot-SWA group precedence over User-agent: * for this crawler.
Block the whole site
User-agent: SemrushBot-SWA
Disallow: /Allow access
User-agent: SemrushBot-SWA
Allow: /It's the 48th most-named crawler of the 171 we counted in the crawler index.
Which other crawlers does this operator run?
How we counted
For SemrushBot-SWA, our 19 October 2026 collection used Tranco’s top 10,000 sites. A request timed out after 8 seconds, with files capped at 262,144 bytes. Only available files we could read enter the percentages. Later readings are included in results from scripts/robots_2026_08_06.py (robots.py at app commit 07104ff). See the full survey method.
Common questions
What does naming this crawler tell me?
The 145 files naming it include 7 files without a whole-site ban.
Are leading sites different?
For naming rules, 4.2% of the leading sample mentions it, compared with 2.3% overall. Whole-site blocking is 6.0% at leading ranks and 5.6% across the full sample.
How does the nearby crawler compare?
The nearby FirecrawlAgent entry reports 143 files that name it and 347 whole-site blocks, 5.6% of readable files. Here, 145 files mention the name and 343 block access, or 5.6%.