One mark per crawler
Allowed or blocked, with the rule behind it.
- ClaudeBotAnthropicAllowed
- OAI‑SearchBotOpenAIAllowed
- GPTBotOpenAI · Disallow: /Blocked
- CCBotCommon CrawlHomepage blocked
- PerplexityBotPerplexityAllowed
See which AI crawlers your robots.txt lets in.
Free · no sign-up
notion.com
176of 179 AI crawlers allowed
Each crawler gets one mark. Two lines let one in.
Allowed or blocked, with the rule behind it.
Two lines in your robots.txt.
User-agent: * Disallow: /checkout User-agent: OAI-SearchBot Allow: / Sitemap: /sitemap.xml
Every AI crawler we know of, grouped by the job it does.
Build the index that AI search and AI answers draw on.
Something else?
A small file at the root of a site, like yourbrand.com/robots.txt, that tells crawlers which pages they may read.
The Search and Assistant groups: they read your pages for AI search, and when someone asks an assistant about you.
Yes, with no sign-up. Each visitor can run 20 checks an hour across the free tools, and 60 a day.