Free AI Visibility Tools

Which AI crawlers can read your site?

See which AI crawlers your robots.txt lets in.

Free · no sign-up

notion.com

176of 179 AI crawlers allowed

Read it, then fix it

Each crawler gets one mark. Two lines let one in.

One mark per crawler

Allowed or blocked, with the rule behind it.

  • ClaudeBotAnthropicAllowed
  • OAI‑SearchBotOpenAIAllowed
  • GPTBotOpenAI · Disallow: /Blocked
  • CCBotCommon CrawlHomepage blocked
  • PerplexityBotPerplexityAllowed

Let a crawler in

Two lines in your robots.txt.

robots.txt
User-agent: *
Disallow: /checkout
User-agent: OAI-SearchBot
Allow: /
Sitemap: /sitemap.xml

What the check covers

Every AI crawler we know of, grouped by the job it does.

Search

Build the index that AI search and AI answers draw on.

  • OAI-SearchBotOpenAI
  • Claude-SearchBotAnthropic
  • PerplexityBotPerplexity

Questions about AI crawlers

Something else?

  • A small file at the root of a site, like yourbrand.com/robots.txt, that tells crawlers which pages they may read.

  • The Search and Assistant groups: they read your pages for AI search, and when someone asks an assistant about you.

  • Yes, with no sign-up. Each visitor can run 20 checks an hour across the free tools, and 60 a day.

Hi. Ask me anything about AI crawlers and your robots.txt.
What is robots.txt?
A small file at the root of a site, like yourbrand.com/robots.txt, that tells crawlers which pages they may read.

See what AI says about you

Start free