Does GitHub Pages block AI bots by default?

By Matt Kerr·Published 12 November 2026

GitHub Pages setup guide says: GitHub’s Pages setup guide does not say whether these six bots are blocked or allowed by default. It describes publishing static files. It gives no AI crawler toggle or named-bot firewall control for Pages.

Check the published hostname and the files produced by your publishing source. Read /robots.txt at the domain root and compare public page responses. A private repository does not by itself make its GitHub Pages site private.

Rules for each bot

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot

GitHub Pages setup guide says: The docs do not say. The Pages setup guide gives no named-bot access policy. Check the hostname-root robots.txt in your publishing output. No AI firewall setting is documented here.

Find the site that is actually published

GitHub Pages setup guide says: Open the repository settings and select Pages. The setup guide says Visit site opens the published address. Use that address to identify whether you have a user or organization site or a project site below a repository path. Keep the hostname and path when you test a page. They tell you which published output you are looking at.

The guide warns that Pages sites are public even when the repository is private, where the plan or organization permits that setup. Repository visibility therefore cannot answer the bot-access question. Inspect the public site itself. If your concern is content being visible to any visitor, review what you published before relying on a robots.txt request.

Inspect the publishing output

GitHub Pages setup guide says: Pages publishes static files from your publishing source. A source can be a chosen folder or a workflow-produced site. The guide explains that a workflow artifact must include the entry file at its top level, while a source folder has its own entry file. Locate the output that reaches the public site before adding a new static robots.txt to it.

A robots.txt check uses the root of the hostname. For a project URL such as example.github.io/project/, inspect example.github.io/robots.txt as well as the project page you are testing. A file placed only inside /project/ is at a different address. This follows the root-file rule in the robots.txt reference read on 9 November 2026. It is a placement check, without a claim that GitHub supplies that file for every project. Google robots.txt introduction. Read 9 November 2026

Make a file change through the publishing source

GitHub Pages setup guide says: If you control the published root file and want to change crawler instructions, make that change in the files your build publishes. Preserve the existing instructions for other paths. The setup guide says static files are published in the structure of the source, so a file in a different folder may appear at a different public URL. Check the final address after the normal publication process.

GitHub Pages does not support server-side languages such as PHP, Ruby or Python. The saved setup guide does not describe a server rule you can edit to deny these bot headers. A static robots.txt can express a request, while the curl checks below show the responses you receive. The robots.txt reference describes the file as crawler instructions, separate from controlling access to content. Google robots.txt introduction. Read 9 November 2026

curl -sS -L https://example.com/robots.txt
curl -sS -L -D - -o /dev/null -A 'Mozilla/5.0' https://example.com/
curl -sS -L -D - -o /dev/null -A 'GPTBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'ClaudeBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'PerplexityBot' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Google-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'Applebot-Extended' https://example.com/
curl -sS -L -D - -o /dev/null -A 'CCBot' https://example.com/

Keep the file check tied to the hostname

Replace example.com below with your published hostname when fetching robots.txt. For the page requests, include the full project path if your site uses one. Compare the browser header with each named header on that same URL. If you use a custom domain, check that domain too. Save the final URL after redirects so a test on another hostname is not mistaken for a test of the intended site.