RankMesh
Free report

Free SEO checker

AI crawler checker for robots.txt

An AI crawler checker reads your robots.txt and tells you, bot by bot, whether OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl may use a page. Search crawlers decide whether you can be cited in AI answers; training crawlers decide whether your content trains models. They are separate choices.

What this tool checks

Why it matters

Many sites block AI bots by accident, through a security plugin, a CDN setting or a copied robots.txt. If the search crawlers are blocked, AI assistants cannot cite you, no matter how good the content is.

How to fix what it finds

  1. Remove Disallow rules for AI search crawlers you want to cite you.
  2. Block training crawlers separately if you do not want content used for training.
  3. Check your CDN or firewall too; some block AI bots before robots.txt is read.

Want this handled for you? RankMesh finds these issues across your whole site every week and ships the fixes after your approval. See how the platform works, pricing or start with the free AI visibility report.

AI crawler checker: common questions

Does allowing GPTBot get me into ChatGPT answers?

GPTBot collects training data. ChatGPT search uses OAI-SearchBot to find pages and ChatGPT-User to open them for a user. To be cited in ChatGPT search, allow those two.

My robots.txt allows everything. Why would a bot still be blocked?

A firewall, CDN bot protection or server rule can refuse AI crawlers before they read robots.txt. This tool checks robots.txt rules only.

Which rule wins if several match?

The most specific matching rule for that bot wins, and on a tie Allow wins. A bot with its own User-agent group ignores the * group. The tool shows the rule it applied.

Related

Fix it once. Keep it fixed.

RankMesh checks your site every week, writes and publishes content, and tracks how you show up in Google and AI answers. Start free, no card required.

Get a free plan for your site