Free SEO checker

Free robots.txt tester and checker

A robots.txt tester reads a site's robots.txt and tells you whether a given crawler may fetch a given page, and which rule decides it. Enter any URL and this tool tests that exact path for Googlebot, Bingbot and nine AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, using Google's longest-match rule.

What this tool checks

  • The robots.txt at the root of the domain you enter
  • Googlebot and Bingbot, which feed Google, Bing, Copilot and ChatGPT search
  • GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI
  • ClaudeBot and Claude-SearchBot from Anthropic, and PerplexityBot
  • Google-Extended, Applebot-Extended and CCBot
  • The exact rule that allows or blocks each crawler

Why it matters

One overly broad Disallow line can hide a page from Google or from every AI assistant. AI search tools only cite pages their crawlers may read, so a blocked page cannot be recommended no matter how good it is.

How to fix what it finds

  1. Make sure Googlebot and Bingbot are allowed on every page you want to rank.
  2. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to be cited in AI answers.
  3. Block training crawlers such as GPTBot or CCBot only if you mean to; it does not affect search.
  4. Prefer narrow Disallow paths over site-wide blocks.

Want this handled for you?

RankMesh finds these issues across your whole site every week and ships the fixes after your approval.

robots.txt tester: common questions

What is the difference between a robots.txt tester and a robots.txt checker?

They are the same tool under two names. Both read your robots.txt and tell you whether a crawler may fetch a URL. Google removed its own tester from Search Console in 2023, so a free tester like this one is the quickest way to check.

Which rule wins when several match?

Google and most crawlers use the longest matching rule. When an Allow and a Disallow are the same length, Allow wins. This tester follows the same logic.

Does blocking GPTBot remove me from ChatGPT?

GPTBot collects training data. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which are separate. You can block one and allow the others.

Is robots.txt enough to keep a page out of Google?

No. robots.txt stops crawling, not indexing. Use a noindex tag on a crawlable page to keep it out of search results.

Can I build a new robots.txt here?

Yes. Use the free robots.txt generator to choose which AI crawlers to allow and copy the finished file.

Fix it once. Keep it fixed.

RankMesh checks your site every week, writes and publishes content, and tracks how you show up in Google and AI answers. Start free, no card required.

Get a free plan for your site

Book a 20-minute demo

Tell us a little about your business, then pick a time that suits you.

Before you go: see how AI search sees your site

A free report on whether ChatGPT, Gemini and Google AI recommend your business. No sign-up, results in seconds.