AI Crawler Checker for robots.txt

Enter your domain. We fetch your public robots.txt once and show which AI search and training crawlers it allows or blocks, using each company's published crawler names.

Loading your free checks…

How to read the results

  • AI search / answers crawlers (OAI-SearchBot, Claude-SearchBot, Googlebot, Applebot and others) affect whether you can be found and linked. If you want customers to find you through AI search, these should usually be allowed.
  • Model training controls (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) affect whether your content may be used for training. Blocking them is a legitimate choice. We don't recommend either way.
  • User-requested agents (ChatGPT-User, Claude-User and similar) visit when a person asks. Their operators say robots.txt may not apply to them.

Crawler names verified against official docs on 25 Oct 2026: OpenAI, Anthropic, Google, Apple, Common Crawl. Matching follows Google's robots.txt rules (RFC 9309).

What these checks are, and aren't
  • Basic, automated checks of public files on your site. They can miss things and are not a ranking score.
  • Passing every check doesn't mean any AI assistant or search engine will show or recommend you. Nobody outside those companies can promise that.
  • robots.txt is a request crawlers choose to honor. It isn't access control, and firewalls or CDN bot settings can block crawlers even when robots.txt allows them.
  • General information only, not legal or professional advice.