Does robots.txt block a whole search crawler?
It checks whole-site search restrictions and problems reading the robots policy.
What Check Engine measures
The check uses the saved robots file's access rules for search crawlers. A response with HTTP status 500 or above leaves the policy unreadable and produces a critical server-error finding. Blocking Googlebot across the site is critical. Bingbot blocks are high, and another supported crawler block is low. Syntax problems are low. A crawl-delay wait skipped by the audit produces a notice. For example, a robots response with HTTP status 500 produces the server-error result and leaves the policy unreadable.
This site-wide test reads the robots response, its rules and the saved crawl-delay result. It does not need article pages to run. A failed robots request can leave the intended policy unknown.
How to read the result
The report distinguishes policy restrictions, parsing problems and an audit that did not honour the requested wait. A failed robots request cannot prove which individual allow or disallow rule would apply if the file were readable.
Why it matters
Robots rules state where a crawler may go. A site-wide restriction can prevent crawling when that was never intended. Check that the file loaded successfully before treating its absence or unreadability as a deliberate policy.
How to fix it
Inspect the root robots response and the exact applicable group. Remove a whole-site block only when it conflicts with your intended policy. Repair server errors before interpreting missing rule data. Correct syntax in place, then check the saved access result again. Review crawl-delay handling as an audit setting rather than changing your public policy solely to speed a test.
What we found in field audits
Problems in Crawlability & indexing were found on 12 of the 14 sites we audited. These are area totals. Read the field-audit method.
Method note
backend/seo_engine/checks/indexability.py, robots_blocks_all, revision 0bb8e8d3.
Common questions
Is blocking Googlebot across the site more serious than blocking Bingbot?
Yes. The Googlebot case is critical, while Bingbot blocks are high. Review the intended crawler policy before changing either.
Can a robots.txt server error leave search policy unknown?
Yes. HTTP status 500 or higher leaves robots policy unreadable and gets a critical server-error result.
Does an ignored crawl-delay request mean the site blocks every crawler?
No. The crawl-delay wait concerns a notice about audit behaviour. It is separate from a policy blocking search crawlers.