NowThe AI Readiness StudyPart 2 of 4

Robots.txt and AI crawlers: 96% of 1,124 brand websites let ChatGPT and Claude search in

96% of 1,124 brand websites let ChatGPT and Claude search in. What each AI crawler does, and three robots.txt blocks ready to paste.

Gabriele FajaFounder, remake

· 7 min read

It’s Thursday, and the web team has had enough of bots. Someone writes two lines that close the whole site to every visitor (User-agent: *, then Disallow: /). Nobody wants to vanish from Google, so they add a group that lets Googlebot back in by name. Coffee. Job done. ChatGPT search is now standing outside, along with every other AI search crawler, because nobody wrote their names down.

On 14 and 15 October we went looking for that file on brand websites, as part of The AI Readiness Study.

How do AI crawlers read your robots.txt?

By name. ChatGPT search sends a crawler called OAI-SearchBot, and Claude search sends Claude-SearchBot. Each looks for a group with its own name. Finding none, it follows the User-agent: * lines, the rules for everyone.

120 of the 1,124 brand websites name an AI crawler in their robots.txtThe rest leave AI crawlers to the rules written for every visitor, or have no file.
No robots.txt, or an empty one98 of 1,124
A file that names no AI crawler883 of 1,124
Names at least one AI crawler120 of 1,124
Answered with an error when asked for the file23 of 1,124

1,124 brand websites that answered when we asked for their robots.txt, 14–15 Oct 2026.Source: remake study of brand websites drawn from Wikidata.

That’s how brand sites let AI search in. By saying nothing at all. Fine, until those lines change (a relaunch, a tidy-up, a new agency). Naming GPTBot or ClaudeBot doesn’t count, by the way. Those are training crawlers, and search has its own.

51 of the 64 files with rules for GPTBot or ClaudeBot never name the same company's search crawlerWhere the search crawler isn't named, it follows the file's general rules. In all but one of those files, those rules leave it open today.
Leave the search crawler to the general rules51 of 64
Name the search crawler too13 of 64

Files that name GPTBot or ClaudeBot, out of the 120 brand robots.txt files that name at least one AI crawler.Source: remake study of brand websites drawn from Wikidata.

The Thursday file isn’t made up. Eleven sites with no AI lines close the whole site to everyone, and five of them let Googlebot back in by name. At those five, ChatGPT’s, Claude’s and Perplexity’s search crawlers stay outside. So does every other AI search crawler on our list.

The fix is a group of their own, like choice 2 below.

What does each AI crawler do?

Most AI companies send more than one crawler, each with its own job and a name only an engineer could love. Here they are, as each company listed them on 14 October 2026.

Job Crawlers What it does
AI search OAI-SearchBot (ChatGPT), Claude-SearchBot, PerplexityBot, Amzn-SearchBot, Meta-WebIndexer, MistralAI-Index, DuckAssistBot (DuckDuckGo), PetalBot (Huawei) Crawls pages so they can appear in AI answers. Google’s AI Overviews follow Googlebot’s rules, so they need no line of their own.
Assistants ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, MistralAI-User Fetch a page when a person asks about it. OpenAI, Perplexity, Amazon and Meta say robots.txt may not apply to theirs.
Training GPTBot, ClaudeBot, MistralAI-Training, CCBot Collects pages to train AI models. CCBot builds Common Crawl’s public archive of the web, one of the most widely used sources of AI training data.
Training and other products Amazonbot, Meta-ExternalAgent May train models and also improve their makers’ other products.
Permission only Google-Extended, Applebot-Extended Never visit. They tell Google and Apple whether their AI models may use your pages.

How to allow AI crawlers in robots.txt

Three ways, from most open to most choosy, each ready to paste. First, three things to tidy.

  • Delete old AI lines first, including names that aren’t in the table above. Two of them, anthropic-ai and Claude-Web, turn up in 32 of the 120 files that name an AI crawler. Anthropic’s own crawler page lists neither. A crawler reads every group with its name, so an old line can clash with a new one.
  • Copy your private areas into each named group. Checkout and account pages, for example. Palo Alto Networks does exactly this. A crawler with its own group skips your User-agent: * lines.
  • Leave Googlebot, Bingbot and Applebot out of these blocks. They crawl for Google’s, Bing’s and Apple’s main search (Google’s AI Overviews included). A block for AI shouldn’t change how ordinary search sees you.

1. Everything open

969 of the 1,124 sites already work like this, without a single AI line. Writing it down keeps every AI crawler in, whatever happens to the rest of the file. Vertiv even gives each crawler a name tag. “# OAI-SearchBot — real-time retrieval for ChatGPT search and citations”, reads one.

# Every AI crawler: welcome
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: PetalBot
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
# Your private areas, for example:
# Disallow: /checkout/

2. AI search and assistants open, training closed

Search and assistant crawlers get one group that lets them in. Training crawlers get one that keeps them out. Neither cares about your User-agent: * lines.

Decide on Google-Extended first. Google says it covers training Gemini models and the pages the Gemini app draws on for its answers. Google Search is unaffected (AI Overviews included). Want to stay in the Gemini app’s answers? Delete the last group.

# AI search and assistants: welcome
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: PetalBot
Allow: /
# Your private areas, for example:
# Disallow: /checkout/

# Training: closed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Gemini training and Gemini app answers: closed
User-agent: Google-Extended
Disallow: /

3. Close particular crawlers by name

Korn Ferry’s file keeps out exactly one: Bytespider, run by TikTok’s owner, ByteDance. Add a group like this to either choice, one User-agent line per crawler.

# Closed by name
User-agent: Bytespider
Disallow: /

PetalBot is the name brands close most often (in 34 of the 120 files, and for more than half of those it’s the only one). Huawei says PetalBot builds the index for Petal Search and for recommendations in Huawei Assistant and AI Search. Close it, and your site is out of both.

Does your bot protection let crawlers in?

Our checker gives its name at every door. It was turned away at 246 live sites. At 197 of those, it couldn’t even read the robots.txt on that visit. A crawler turned away at the door never gets to read the rules inside.

How often brand websites turned away an automated visitor that gave its name, by country or group of countriesA refusal, a stall or a bot check.
Germany7% ±3
France, Italy and Spain17% ±5
Canada and Australia22% ±6
US24% ±4
UK25% ±6
All eight countries19% ±2

1,287 live brand websites, 14–15 Oct 2026. The visitor gave its name; one visit per site, from one location.Source: remake study of brand websites drawn from Wikidata.

Bot protection is the likely cause, so start there. It can be set to recognise AI crawlers, because OpenAI, Anthropic and Perplexity publish the addresses their crawlers come from. Take three requests to whoever runs it.

  • Let in the crawlers your robots.txt welcomes. For OpenAI, Anthropic and Perplexity, match each one to the addresses they publish.
  • Keep robots.txt open to every visitor, so every crawler can read your rules.
  • Ask to see it working. Its own records should show those crawlers getting through.

Back to Thursday, then. Keep the two lines that close the site, and keep Googlebot’s group. Paste choice 2 underneath, with your private areas copied in. Same coffee, three more groups (two, if you keep the Gemini app), and ChatGPT search is back inside.

Check your live robots.txt after the change

How we measured

The websites. We drew brand websites at random from Wikidata, across 11 industries and eight countries (the US, the UK, Germany, France, Italy, Spain, Canada and Australia). We cleared out public bodies, non-brands and addresses now owned by someone else, leaving 1,349. This article is part of our study of 1,000 of those websites, read page by page. The robots.txt figures cover more of them, 1,124. A robots.txt sits at its own address, so we could still read it at some sites that turned our visit away.

What we checked. On 14 and 15 October 2026 we read each site’s robots.txt against each AI company’s published crawler names. A site counts as open to a crawler when its file, or the lack of one, doesn’t shut that crawler out. Among the 1,124, 87 had no file, 11 an empty one and 23 an error, which never counts as open. Sites that gave no answer for the file, or a bot check instead, aren’t among the 1,124. Each live site got one visit, from one place. Our figure of 246 counts bot checks as well, four of them, which is why it is higher than the 242 in the study’s other articles.

Limits. Robots.txt is a request that only well-behaved crawlers follow. Shares of the 1,124 and of the 1,287 live sites carry a margin of up to ±2 points. Shops and online stores (139 sites, ±8) and finance and insurance (110 sites, ±6) sit further apart than both margins. We drew only companies whose Wikidata entry links to Wikipedia or one of its sister projects. So the study leans toward established companies, with fewer young, online-only brands than the web as a whole.

Company list from Wikidata (CC0); chain brands from the Name Suggestion Index (BSD-3).

More from the study

The AI Readiness Study: all 4 articles

See what AI says about your brand

See how well AI can read your site, then what ChatGPT says about you.

Start free