Free SEO checker
Free robots.txt tester and checker
A robots.txt tester reads a site's robots.txt and tells you whether a given crawler may fetch a given page, and which rule decides it. Enter any URL and this tool tests that exact path for Googlebot, Bingbot and nine AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, using Google's longest-match rule.
What this tool checks
- The robots.txt at the root of the domain you enter
- Googlebot and Bingbot, which feed Google, Bing, Copilot and ChatGPT search
- GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI
- ClaudeBot and Claude-SearchBot from Anthropic, and PerplexityBot
- Google-Extended, Applebot-Extended and CCBot
- The exact rule that allows or blocks each crawler
Why it matters
One overly broad Disallow line can hide a page from Google or from every AI assistant. AI search tools only cite pages their crawlers may read, so a blocked page cannot be recommended no matter how good it is.
How to fix what it finds
- Make sure Googlebot and Bingbot are allowed on every page you want to rank.
- Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to be cited in AI answers.
- Block training crawlers such as GPTBot or CCBot only if you mean to; it does not affect search.
- Prefer narrow Disallow paths over site-wide blocks.
Want this handled for you?
RankMesh finds these issues across your whole site every week and ships the fixes after your approval.
robots.txt tester: common questions
They are the same tool under two names. Both read your robots.txt and tell you whether a crawler may fetch a URL. Google removed its own tester from Search Console in 2023, so a free tester like this one is the quickest way to check.
Google and most crawlers use the longest matching rule. When an Allow and a Disallow are the same length, Allow wins. This tester follows the same logic.
GPTBot collects training data. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which are separate. You can block one and allow the others.
No. robots.txt stops crawling, not indexing. Use a noindex tag on a crawlable page to keep it out of search results.
Yes. Use the free robots.txt generator to choose which AI crawlers to allow and copy the finished file.
Related
- robots.txt generator
- GPTBot and AI crawler checker
- AI crawler tracking
- Free report: AI search readiness
- Website SEO checkerStatus, redirects, HTTPS, canonical, robots.txt, sitemap, titles, H1, mobile and images in one scan.
- Why am I not ranking?Enter a page and a keyword to see what is stopping it from ranking, in the order to fix it.
- AEO checkerAnswer engine readiness: question headings, direct answers, FAQ markup, lists and summaries.
- GEO checkerAI search readiness: AI crawler access, llms.txt, Organization schema, server-rendered text and citable passages.
- All free SEO tools
Fix it once. Keep it fixed.
RankMesh checks your site every week, writes and publishes content, and tracks how you show up in Google and AI answers. Start free, no card required.
Get a free plan for your site