Do your AI crawler rules still name a real crawler?

By Matt Kerr·Published 11 November 2026

It finds retired or unmatchable crawler names in the robots file.

What Check Engine measures

The test needs a nonempty robots file. It compares crawler names with lists of retired names and names no crawler can match. Restrictions are compared with the wildcard group and any live replacement in the same or a separate group. An old name causes no lost restriction when its replacement already has the same restrictions or stricter ones.

The check reads the saved robots groups. It uses the policy text to identify crawler names.

How to read the result

A lost restriction is medium severity. Inert names whose restrictions are already covered, or which sit above no restriction, are notices. The report explains whether the old spelling changes the effective policy. It does not assume every retired name makes private pages readable.

Why it matters

A rule can look restrictive while naming a token no crawler sends. The useful distinction is whether your intended restriction still takes effect through another group, which is why the check compares coverage before choosing severity.

How to fix it

Find the exact token named in the report and identify its current replacement. Copy the restrictions you intend to keep to the live name, checking for an existing group first. Remove obsolete lines after you have checked the effective policy. Avoid blindly adding another group with a different set of rules.

AI crawler directives: which to block, which to keep

What we found in field audits

Problems in AI search visibility were found on 14 of the 14 sites we audited. These are area totals. Read the field-audit method.

Method note

backend/seo_engine/checks/ai_visibility.py, dead_crawler_directive, revision 0bb8e8d3.