Do indie products let AI crawlers in?

51% of indie sites publish an llms.txt for AI assistants, and 11% block at least one AI crawler in robots.txt — almost always the ones that collect training data (11%), rarely the ones that power AI search (0.6%). 7.7% block OpenAI's GPTBot.

November 2026 edition7,272 indie sites checked this monthMeasured CSV
publish an llms.txt
51%
publish an llms.txt
n = 7,272
block an AI crawler
11%
block an AI crawler
n = 7,204
block a training crawler
11%
block a training crawler
block an AI search crawler
0.6%
block an AI search crawler

Key findings

  • 51% of indie sites serve an llms.txt — a plain-text guide to the site written for language models.
  • 11% of indie sites keep at least one AI crawler out of their home page in robots.txt; 89% let them all in.
  • Blocking is aimed at training: 11% block a training crawler, 0.6% an AI search crawler and 0.7% the fetchers that read a page when a user asks. 10% block training while leaving AI search open.
  • The most blocked crawler is Bytespider (ByteDance), shut out by 9% of sites.
  • 57% of the sites that block AI crawlers do it through Cloudflare's managed robots.txt — a switch in a dashboard rather than a line their maker wrote.
On this page
  1. llms.txt
  2. llms.txt by builder and framework
  3. Who gets blocked, by what they do
  4. The most blocked crawlers
  5. Blocking by default
  6. Newer sites
  7. How this was measured
  8. Edition by edition
  9. Questions
  10. Cite and reuse

llms.txt

llms.txt is a proposed convention: a Markdown file at the root of a site that tells language models what the site is and which pages matter, the way robots.txt speaks to crawlers.

Share of indie sites with an llms.txt
  • All indie sites51%
  • AI products55%
  • Everything else47%

A file counts only when /llms.txt answers with text, not with an HTML page.

Show the numbers
All indie sites51%
AI products55%
Everything else47%
Embed this chart
Paste into your page
<iframe src="https://awesomeindie.com/state-of-indie/embed/ai-crawlers/0" title="Share of indie sites with an llms.txt" width="100%" height="292" style="border:0" loading="lazy"></iframe><p>Source: <a href="https://awesomeindie.com/state-of-indie/ai-crawlers">State of Indie: AI crawlers</a>, Awesome Indie</p>

No AI company has committed to reading llms.txt, so publishing one is a bet rather than a standard. It is cheap to write, and several site builders and documentation tools now generate it on their own — which explains much of the adoption below.

llms.txt by builder and framework

Share of sites with an llms.txt, by builder or framework
Built withSitesllms.txt
Next.js2,90053%
WordPress50850%
Astro39661%
Nuxt19045%
jQuery16737%
Webflow15646%
Framer11433%
Alpine.js9243%
React (no meta-framework)6862%
WooCommerce6436%
Svelte4743%
Hotwire (Turbo)3743%
Vue (no Nuxt)2646%
Shopify2576%
Wix24100%
Remix2129%
Angular1765%
Gatsby1331%
Docusaurus1050%
Builders and frameworks with at least 10 sites; greyed rows have fewer than 30.

Who gets blocked, by what they do

AI crawlers come in three kinds: the ones that gather pages to train models, the ones that index pages for AI search and answers, and the ones that fetch a page because a user asked an assistant to read it.

Share of indie sites whose robots.txt blocks at least one crawler of each kind
  • Model training11%
  • AI search and answers0.6%
  • Visits on a user’s request0.7%
  • Search engines0.3%
  • Any AI crawler11%

Blocked means the home page is disallowed for that crawler. n = 7,204 sites whose robots.txt was read.

Show the numbers
Model training11%
AI search and answers0.6%
Visits on a user’s request0.7%
Search engines0.3%
Any AI crawler11%
Embed this chart
Paste into your page
<iframe src="https://awesomeindie.com/state-of-indie/embed/ai-crawlers/1" title="Share of indie sites whose robots.txt blocks at least one crawler of each kind" width="100%" height="360" style="border:0" loading="lazy"></iframe><p>Source: <a href="https://awesomeindie.com/state-of-indie/ai-crawlers">State of Indie: AI crawlers</a>, Awesome Indie</p>

Blocking a training crawler keeps a site's pages out of future models but does not stop AI search from linking to it; blocking an AI search crawler does. 10% of indie sites draw exactly that line — training out, search in.

The most blocked crawlers

Share of indie sites blocking each crawler
  • Bytespider (ByteDance)9%
  • CCBot (Common Crawl)8.5%
  • GPTBot (OpenAI)7.7%
  • Amazonbot (Amazon)7.5%
  • Meta-ExternalAgent (Meta)7.4%
  • ClaudeBot (Anthropic)7.1%
  • Google-Extended (Google)7.1%
  • Applebot-Extended (Apple)7%
  • ChatGPT-User (OpenAI)0.6%
  • PerplexityBot (Perplexity)0.5%
  • Meta-ExternalFetcher (Meta)0.5%
  • MistralAI-User (Mistral)0.5%
  • Perplexity-User (Perplexity)0.4%
  • DuckAssistBot (DuckDuckGo)0.4%

n = 7,204. Google-Extended and Applebot-Extended are switches that stop Google and Apple training on pages, not crawlers of their own.

Show the numbers
Bytespider (ByteDance)9%
CCBot (Common Crawl)8.5%
GPTBot (OpenAI)7.7%
Amazonbot (Amazon)7.5%
Meta-ExternalAgent (Meta)7.4%
ClaudeBot (Anthropic)7.1%
Google-Extended (Google)7.1%
Applebot-Extended (Apple)7%
ChatGPT-User (OpenAI)0.6%
PerplexityBot (Perplexity)0.5%
Meta-ExternalFetcher (Meta)0.5%
MistralAI-User (Mistral)0.5%
Perplexity-User (Perplexity)0.4%
DuckAssistBot (DuckDuckGo)0.4%
Embed this chart
Paste into your page
<iframe src="https://awesomeindie.com/state-of-indie/embed/ai-crawlers/2" title="Share of indie sites blocking each crawler" width="100%" height="666" style="border:0" loading="lazy"></iframe><p>Source: <a href="https://awesomeindie.com/state-of-indie/ai-crawlers">State of Indie: AI crawlers</a>, Awesome Indie</p>

Blocking by default

Cloudflare can write and maintain a site’s robots.txt for its owner, with the training crawlers already disallowed.

of sites use Cloudflare’s managed robots.txt
6.2%
of sites use Cloudflare’s managed robots.txt
n = 7,204
of AI-blocking sites block through it
57%
of AI-blocking sites block through it
n = 766

A large part of the blocking on indie sites is a setting rather than a decision: switching on Cloudflare’s managed robots.txt, or its "block AI bots" option, adds the training crawlers for every site on the account. Worth a look if AI search traffic matters to you.

Newer sites

Share of sites with an llms.txt, by when the product was submitted

Every site checked as it is today. Periods with fewer than 60 sites are left out.

Show the numbers
SeriesH1 2023H2 2023H1 2024H2 2024H1 2025H2 2025H1 2026H2 2026 (so far)
llms.txt33%36%30%38%42%46%55%64%
Embed this chart
Paste into your page
<iframe src="https://awesomeindie.com/state-of-indie/embed/ai-crawlers/3" title="Share of sites with an llms.txt, by when the product was submitted" width="100%" height="470" style="border:0" loading="lazy"></iframe><p>Source: <a href="https://awesomeindie.com/state-of-indie/ai-crawlers">State of Indie: AI crawlers</a>, Awesome Indie</p>
Share of sites blocking AI crawlers, by when the product was submitted
  • A training crawler
  • An AI search crawler

Sites whose robots.txt was read. Periods with fewer than 60 sites are left out.

Show the numbers
SeriesH1 2023H2 2023H1 2024H2 2024H1 2025H2 2025H1 2026H2 2026 (so far)
A training crawler5.4%11%7.7%7.4%7%10%13%13%
An AI search crawler0%1.3%0.7%0.5%1.3%1.2%0.6%0.2%
Embed this chart
Paste into your page
<iframe src="https://awesomeindie.com/state-of-indie/embed/ai-crawlers/4" title="Share of sites blocking AI crawlers, by when the product was submitted" width="100%" height="470" style="border:0" loading="lazy"></iframe><p>Source: <a href="https://awesomeindie.com/state-of-indie/ai-crawlers">State of Indie: AI crawlers</a>, Awesome Indie</p>

Every site is checked as it is today, grouped by the half-year its product was submitted to Awesome Indie.

How this was measured

Each month the site check requests `/llms.txt` and `/robots.txt` from the host each site's home page settles on. robots.txt is read the way Google and RFC 9309 read it, and a crawler counts as blocked when its own group — or the `*` group, if it has none — disallows the home page. A missing robots.txt blocks nothing; a robots.txt that could not be read at all is left out (n = 7,204 sites read this month).

The crawlers are the documented ones in the AI crawler checker, which runs the same reading on any site. Whether the same sites get indexed at all is in Website health.

Edition by edition

Every edition is frozen the month it is published and never recomputed, so a figure quoted from it stays checkable. Past editions are in the edition menu at the top.

This is the first edition of this chapter. Every month adds a point here, so the lines start with the next edition.

Questions

How many websites have an llms.txt file?

51% of indie product sites serve an llms.txt at their root (n = 7,272). Adoption is highest among AI products and on tools that generate the file automatically.

How many websites block GPTBot?

7.7% of indie product sites disallow OpenAI's GPTBot in robots.txt (n = 7,204); 11% block at least one AI crawler.

Does blocking GPTBot stop ChatGPT from showing my site?

No. GPTBot gathers training data; ChatGPT search uses OAI-SearchBot, and pages a user asks about are fetched as ChatGPT-User. Among indie sites, 10% block training crawlers while leaving AI search crawlers in.

What is llms.txt?

A proposed convention for a Markdown file at /llms.txt that describes a site for language models: what it is, and links to the pages that explain it best. It sits beside robots.txt and sitemap.xml rather than replacing them.

Should I block AI crawlers?

It depends on what you want from AI: blocking training crawlers keeps your pages out of future models, while blocking AI search crawlers also takes you out of AI answers and their links. The free AI crawler checker shows what your robots.txt currently allows.

Cite and reuse

The figures are free to quote, chart and republish under CC BY 4.0: credit State of Indie by Awesome Indie with a link to this page. Every chart above has its own embed code, and the whole table is a CSV.

Citation
Awesome Indie (2026). State of Indie: AI crawlers, November 2026 edition. https://awesomeindie.com/state-of-indie/ai-crawlers