Do indie products let AI crawlers in?
51% of indie sites publish an llms.txt for AI assistants, and 11% block at least one AI crawler in robots.txt — almost always the ones that collect training data (11%), rarely the ones that power AI search (0.6%). 7.7% block OpenAI's GPTBot.
- publish an llms.txt
- 51%
- publish an llms.txt
- n = 7,272
- block an AI crawler
- 11%
- block an AI crawler
- n = 7,204
- block a training crawler
- 11%
- block a training crawler
- block an AI search crawler
- 0.6%
- block an AI search crawler
Key findings
- 51% of indie sites serve an llms.txt — a plain-text guide to the site written for language models.
- 11% of indie sites keep at least one AI crawler out of their home page in robots.txt; 89% let them all in.
- Blocking is aimed at training: 11% block a training crawler, 0.6% an AI search crawler and 0.7% the fetchers that read a page when a user asks. 10% block training while leaving AI search open.
- The most blocked crawler is Bytespider (ByteDance), shut out by 9% of sites.
- 57% of the sites that block AI crawlers do it through Cloudflare's managed robots.txt — a switch in a dashboard rather than a line their maker wrote.