Can AI crawlers read your site?

Add a group to your robots.txt naming each crawler and disallowing the site — for example User-agent: GPTBot then Disallow: /. The rule generator in the results writes the whole section for you, split by what each crawler is for.

Think about which ones you block. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) only feed models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are how AI assistants find, quote and link to you — blocking them is like blocking Google.

They are three different OpenAI user agents. GPTBot collects pages that may be used to train models. OAI-SearchBot indexes pages for ChatGPT search, which is what puts your site in its answers with a link. ChatGPT-User visits a page when a person asks ChatGPT to open it. Each one reads its own group in robots.txt, so you can block training and still be found. Anthropic splits Claude the same way (ClaudeBot, Claude-SearchBot, Claude-User), as does Perplexity.

No. Google-Extended is not a crawler but a switch: it decides whether Google may use your pages for Gemini. Google Search, AI Overviews and AI Mode all come from Googlebot, which Google-Extended does not affect. The same goes for Applebot-Extended and Applebot.

Something in front of the site is turning them away before robots.txt matters: a firewall rule, a security plugin, or a CDN setting. On Cloudflare, look at Security → Bots for “Block AI bots” (and AI Crawl Control) — it blocks AI crawlers at the edge whatever robots.txt says.

One caveat: we ask from our own server while claiming to be the crawler. Some firewalls block requests that use a crawler’s name but do not come from its company’s addresses, and let the real one through. If you have no rule against AI crawlers, that is the likely explanation.

llms.txt is a proposed standard (llmstxt.org): a Markdown file at /llms.txt with your site’s name, a one-line summary and a list of the pages that matter, so an AI assistant can understand the site without crawling all of it.

It is optional. Developer tools and documentation sites use it most, and no major AI company has said its crawler reads it. It costs five minutes, though, and the checker writes you a starter from your own page.

The major ones say they do for crawling: OpenAI, Anthropic, Google, Apple, Perplexity’s indexer and Common Crawl all document their tokens and follow them. Fetches a person asks for are different — Perplexity and Meta say theirs may not follow robots.txt — and some crawlers, Bytespider among them, have been widely reported to ignore it. For a hard block, use a firewall rule, and check the result with the live test.