# Robots.txt and AI crawlers: 96% of 1,124 brand websites let ChatGPT and Claude search in

> 96% of 1,124 brand websites let ChatGPT and Claude search in. What each AI crawler does, and three robots.txt blocks ready to paste.

Article · 15 Oct 2026 · Gabriele Faja, Founder · remake
Page: https://www.tryremake.com/now/ai-crawlers-brand-websites
Part 2 of 4 in The AI Readiness Study: https://www.tryremake.com/now/ai-readiness-study

It's Thursday, and the web team has had enough of bots. Someone writes two lines that close the whole site to every visitor (`User-agent: *`, then `Disallow: /`). Nobody wants to vanish from Google, so they add a group that lets Googlebot back in by name. Coffee. Job done. ChatGPT search is now standing outside, along with every other AI search crawler, because nobody wrote their names down.

On 14 and 15 October we went looking for that file on brand websites, as part of [The AI Readiness Study](/now/ai-readiness-study).

## Key findings

- 96% of brand websites let ChatGPT search and Claude search in. That's 1,083 of the 1,124 that answered when we asked for their robots.txt.
- Only 32 of the 1,124 (3%) name either search crawler. Everywhere else, the rules for every visitor decide. Close a folder to everyone, and AI search is shut out of it too.
- 51 of the 64 sites with rules for GPTBot or ClaudeBot (OpenAI's and Anthropic's training crawlers) never name the same company's search crawler. Search runs on its own crawler, so the training rule never reaches it.
- We sent a checker that announces itself to 1,287 live brand websites. At 246 of them (19%), it was refused, stalled or shown a bot check. Shops and online stores did this more often (32%) than finance and insurance (14%).

## How do AI crawlers read your robots.txt?

By name. ChatGPT search sends a crawler called OAI-SearchBot, and Claude search sends Claude-SearchBot. Each looks for a group with its own name. Finding none, it follows the `User-agent: *` lines, the rules for everyone.

**120 of the 1,124 brand websites name an AI crawler in their robots.txt**

The rest leave AI crawlers to the rules written for every visitor, or have no file.

| | |
|---|---|
| No robots.txt, or an empty one | 98 (9%) |
| A file that names no AI crawler | 883 (79%) |
| Names at least one AI crawler | 120 (11%) |
| Answered with an error when asked for the file | 23 (2%) |

1,124 brand websites that answered when we asked for their robots.txt, 14–15 Oct 2026. Source: remake study of brand websites drawn from Wikidata.

That's how brand sites let AI search in. By saying nothing at all. Fine, until those lines change (a relaunch, a tidy-up, a new agency). Naming GPTBot or ClaudeBot doesn't count, by the way. Those are training crawlers, and search has its own.

**51 of the 64 files with rules for GPTBot or ClaudeBot never name the same company's search crawler**

Where the search crawler isn't named, it follows the file's general rules. In all but one of those files, those rules leave it open today.

| | |
|---|---|
| Leave the search crawler to the general rules | 51 (80%) |
| Name the search crawler too | 13 (20%) |

Files that name GPTBot or ClaudeBot, out of the 120 brand robots.txt files that name at least one AI crawler. Source: remake study of brand websites drawn from Wikidata.

The Thursday file isn't made up. Eleven sites with no AI lines close the whole site to everyone, and five of them let Googlebot back in by name. At those five, ChatGPT's, Claude's and Perplexity's search crawlers stay outside. So does every other AI search crawler on our list.

The fix is a group of their own, like [choice 2 below](./now/ai-crawlers-brand-websites#2-ai-search-and-assistants-open-training-closed).

[Check which AI crawlers your robots.txt lets in](/tools/ai-crawler-check)

## What does each AI crawler do?

Most AI companies send more than one crawler, each with its own job and a name only an engineer could love. Here they are, as each company listed them on 14 October 2026.

| Job | Crawlers | What it does |
|---|---|---|
| AI search | OAI-SearchBot (ChatGPT), Claude-SearchBot, PerplexityBot, Amzn-SearchBot, Meta-WebIndexer, MistralAI-Index, DuckAssistBot (DuckDuckGo), PetalBot (Huawei) | Crawls pages so they can appear in AI answers. Google's AI Overviews follow Googlebot's rules, so they need no line of their own. |
| Assistants | ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, MistralAI-User | Fetch a page when a person asks about it. OpenAI, Perplexity, Amazon and Meta say robots.txt may not apply to theirs. |
| Training | GPTBot, ClaudeBot, MistralAI-Training, CCBot | Collects pages to train AI models. CCBot builds Common Crawl's public archive of the web, one of the most widely used sources of AI training data. |
| Training and other products | Amazonbot, Meta-ExternalAgent | May train models and also improve their makers' other products. |
| Permission only | Google-Extended, Applebot-Extended | Never visit. They tell Google and Apple whether their AI models may use your pages. |

## How to allow AI crawlers in robots.txt

Three ways, from most open to most choosy, each ready to paste. First, three things to tidy.

- **Delete old AI lines first**, including names that aren't in the table above. Two of them, anthropic-ai and Claude-Web, turn up in 32 of the 120 files that name an AI crawler. Anthropic's own crawler page lists neither. A crawler reads every group with its name, so an old line can clash with a new one.
- **Copy your private areas into each named group.** Checkout and account pages, for example. Palo Alto Networks does exactly this. A crawler with its own group skips your `User-agent: *` lines.
- **Leave Googlebot, Bingbot and Applebot out of these blocks.** They crawl for Google's, Bing's and Apple's main search (Google's AI Overviews included). A block for AI shouldn't change how ordinary search sees you.

### 1. Everything open

969 of the 1,124 sites already work like this, without a single AI line. Writing it down keeps every AI crawler in, whatever happens to the rest of the file. Vertiv even gives each crawler a name tag. "# OAI-SearchBot — real-time retrieval for ChatGPT search and citations", reads one.

    # Every AI crawler: welcome
    User-agent: OAI-SearchBot
    User-agent: ChatGPT-User
    User-agent: Claude-SearchBot
    User-agent: Claude-User
    User-agent: PerplexityBot
    User-agent: Perplexity-User
    User-agent: Amzn-SearchBot
    User-agent: Amzn-User
    User-agent: Meta-WebIndexer
    User-agent: Meta-ExternalFetcher
    User-agent: MistralAI-Index
    User-agent: MistralAI-User
    User-agent: DuckAssistBot
    User-agent: PetalBot
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Amazonbot
    User-agent: Meta-ExternalAgent
    User-agent: MistralAI-Training
    User-agent: CCBot
    User-agent: Google-Extended
    User-agent: Applebot-Extended
    Allow: /
    # Your private areas, for example:
    # Disallow: /checkout/

### 2. AI search and assistants open, training closed

Search and assistant crawlers get one group that lets them in. Training crawlers get one that keeps them out. Neither cares about your `User-agent: *` lines.

Decide on Google-Extended first. Google says it covers training Gemini models and the pages the Gemini app draws on for its answers. Google Search is unaffected (AI Overviews included). Want to stay in the Gemini app's answers? Delete the last group.

    # AI search and assistants: welcome
    User-agent: OAI-SearchBot
    User-agent: ChatGPT-User
    User-agent: Claude-SearchBot
    User-agent: Claude-User
    User-agent: PerplexityBot
    User-agent: Perplexity-User
    User-agent: Amzn-SearchBot
    User-agent: Amzn-User
    User-agent: Meta-WebIndexer
    User-agent: Meta-ExternalFetcher
    User-agent: MistralAI-Index
    User-agent: MistralAI-User
    User-agent: DuckAssistBot
    User-agent: PetalBot
    Allow: /
    # Your private areas, for example:
    # Disallow: /checkout/

    # Training: closed
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Amazonbot
    User-agent: Meta-ExternalAgent
    User-agent: MistralAI-Training
    User-agent: Applebot-Extended
    User-agent: CCBot
    Disallow: /

    # Gemini training and Gemini app answers: closed
    User-agent: Google-Extended
    Disallow: /

### 3. Close particular crawlers by name

Korn Ferry's file keeps out exactly one: Bytespider, run by TikTok's owner, ByteDance. Add a group like this to either choice, one `User-agent` line per crawler.

    # Closed by name
    User-agent: Bytespider
    Disallow: /

PetalBot is the name brands close most often (in 34 of the 120 files, and for more than half of those it's the only one). Huawei says PetalBot builds the index for Petal Search and for recommendations in Huawei Assistant and AI Search. Close it, and your site is out of both.

## Does your bot protection let crawlers in?

Our checker gives its name at every door. It was turned away at 246 live sites. At 197 of those, it couldn't even read the robots.txt on that visit. A crawler turned away at the door never gets to read the rules inside.

**How often brand websites turned away an automated visitor that gave its name, by country or group of countries**

A refusal, a stall or a bot check.

| | |
|---|---|
| Germany | 7% ±3 |
| France, Italy and Spain | 17% ±5 |
| Canada and Australia | 22% ±6 |
| US | 24% ±4 |
| UK | 25% ±6 |
| All eight countries | 19% ±2 |

1,287 live brand websites, 14–15 Oct 2026. The visitor gave its name; one visit per site, from one location. Source: remake study of brand websites drawn from Wikidata.

Bot protection is the likely cause, so start there. It can be set to recognise AI crawlers, because OpenAI, Anthropic and Perplexity publish the addresses their crawlers come from. Take three requests to whoever runs it.

- **Let in the crawlers your robots.txt welcomes.** For OpenAI, Anthropic and Perplexity, match each one to the addresses they publish.
- **Keep robots.txt open to every visitor**, so every crawler can read your rules.
- **Ask to see it working.** Its own records should show those crawlers getting through.

Back to Thursday, then. Keep the two lines that close the site, and keep Googlebot's group. Paste choice 2 underneath, with your private areas copied in. Same coffee, three more groups (two, if you keep the Gemini app), and ChatGPT search is back inside.

[Check your live robots.txt after the change](/tools/ai-crawler-check)

## How we measured

**The websites.** We drew brand websites at random from Wikidata, across 11 industries and eight countries (the US, the UK, Germany, France, Italy, Spain, Canada and Australia). We cleared out public bodies, non-brands and addresses now owned by someone else, leaving 1,349. This article is part of our study of 1,000 of those websites, read page by page. The robots.txt figures cover more of them, 1,124. A robots.txt sits at its own address, so we could still read it at some sites that turned our visit away.

**What we checked.** On 14 and 15 October 2026 we read each site's robots.txt against each AI company's published crawler names. A site counts as open to a crawler when its file, or the lack of one, doesn't shut that crawler out. Among the 1,124, 87 had no file, 11 an empty one and 23 an error, which never counts as open. Sites that gave no answer for the file, or a bot check instead, aren't among the 1,124. Each live site got one visit, from one place. Our figure of 246 counts bot checks as well, four of them, which is why it is higher than the 242 in the study's other articles.

**Limits.** Robots.txt is a request that only well-behaved crawlers follow. Shares of the 1,124 and of the 1,287 live sites carry a margin of up to ±2 points. Shops and online stores (139 sites, ±8) and finance and insurance (110 sites, ±6) sit further apart than both margins. We drew only companies whose Wikidata entry links to Wikipedia or one of its sister projects. So the study leans toward established companies, with fewer young, online-only brands than the web as a whole.

Company list from Wikidata (CC0); chain brands from the Name Suggestion Index (BSD-3).
