Free robots.txt tester and checker

By Matt Kerr·Published 6 October 2026·Updated 9 November 2026

This free robots.txt tester checks whether a page is allowed or blocked for Googlebot, Bingbot and the AI crawlers. Paste the page address to see which robots.txt rule decides. No sign-up is needed. It tests crawl rules, not whether a page is indexed or whether a firewall lets a crawler through.

Page address

Test this page

Another crawler (optional)

Free, no sign-up required. It works on any site, with nothing to verify first.

1. Paste the address of the page that dropped out of search, not only the homepage.
2. Add another crawler's name if you want one, then press Test this page.
3. Read each crawler's verdict beside the line that decides it.

You will see:

• Allowed or blocked for Googlebot, Bingbot and the AI crawlers, one row each
• The exact robots.txt line that decides, so you know which line to change
• The sitemaps the file lists

One file, read for one page. Check Engine checks robots.txt rules for 22 AI crawlers across your site. A separate check requests your homepage as 9 AI crawlers plus Googlebot to catch server refusals the file cannot show, among its 99 checks.

How to read your robots.txt checker result

Allowed or Blocked is what the file tells that crawler about this page. The rule beside it is the line that decided, and "named in your file" means a group addresses that crawler by name, so your * rules no longer apply to it. If no line matches, the page is allowed.

The rules follow RFC 9309, the robots.txt standard, and match the parser inside Check Engine. A crawler obeys every group that names it; only if none does, it obeys the * groups. The longest matching rule wins, and Allow wins a tie. * matches any run of characters and a $ at the end anchors it. If the server answers robots.txt with a server error, crawlers stay out of the whole site; if it answers "not found", everything is allowed.

Where did Google's robots.txt tester go?

Google no longer runs a robots.txt Tester in Search Console. In its place is the robots.txt report. It shows which robots.txt files Google found for the top 20 hosts of a property, when each was last crawled, and any warnings or errors, and it lets you request a recrawl of a file in an emergency. It is available only for properties at the domain level, and Google's help page describes no way to type in a page address and see whether that page is blocked.

This tester does that part. It fetches the file the way a crawler would, for the page you give it, and shows which rule decided. Use the report to see what Google last fetched, and this to see what a rule does to a page.

What robots.txt does not do

• It does not remove a page from search. A blocked page can still be indexed from links pointing at it, without its content. To keep a page out, allow crawling and add a noindex tag.
• It is a request, not a lock. Well-behaved crawlers obey it. A firewall or CDN rule is what actually refuses a request, and this tester cannot see one.
• It is per host. www. and the bare domain, and every subdomain, each have their own file. The tester reads the file for the host you type.

Checking every page

Check Engine also checks the noindex tags and headers this tester cannot see, then ranks what to fix with the rest of its 99 checks.

Related

• What an SEO audit checks beyond robots.txt
• Free website audit tools and their limits
• AI crawler checker: can ChatGPT read your site?
• Which AI crawlers to block and which to keep
• Does noindex stop AI crawlers?
• Ahrefs Site Audit: what a paid crawler checks, and how a crawl is paid for

Checked 5 November 2026. The factual claims about this product were read from its own pages on that date: Google's robots.txt report shows which robots.txt files Google found for the top 20 hosts of a property, the last time they were crawled, and any warnings or errors, and lets you request a recrawl of a file for emergencies; it is available only for properties at the domain level (source); Google's guide to updating a robots.txt file points to the robots.txt report to copy the file's content and to request a recrawl (source); Search Engine Land reported when Google added the robots.txt report and decided to sunset the robots.txt tester (its site refuses automated requests, so this line comes from its search listing) (source). Google's pages change; if one has moved on from what is described here, tell us and it will be corrected.

Common questions

Is this robots.txt checker free?

Yes. Paste a page address with no sign-up. The checker reads the robots.txt for that host and tests the rules for Googlebot, Bingbot, AI crawlers and a crawler you name. It does not check indexing or firewall access.

Where is the robots.txt tester in Google Search Console?

Google replaced it with the robots.txt report, which lists the files Google found, when it last crawled them and any errors. The report does not take a page address, so to see whether one page is blocked, paste its address into the tester on this page.

How do I test my robots.txt?

Paste the address of a page into the tester above. It fetches the site's robots.txt, applies the rules the way crawlers do, and shows for each crawler whether that page is allowed and which line decided.

Does Disallow in robots.txt remove a page from Google?

No. It stops Googlebot fetching the page, but the page can still be indexed from links to it, without its content. To keep a page out of search, allow crawling and add a noindex tag.

Why is a page blocked when my * rules allow it?

Because a group names that crawler. A crawler that finds a group addressed to it by name obeys only that group and ignores the * rules entirely.