Free generator
Free robots.txt generator
A robots.txt generator writes the plain text file that tells crawlers which parts of your site they may read. This one lets you choose, crawler by crawler, which AI bots to allow or block, add private paths every crawler should skip and your sitemap address, then copy the finished robots.txt. It runs in your browser.
robots.txt
User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
9 of 9 AI crawlers allowed. Upload the file to the root of your domain, then check it with the robots.txt tester.
What this tool covers
- One User-agent: * group with your disallowed paths
- A separate block for each AI crawler you choose to block
- Nine AI crawlers, from GPTBot and ClaudeBot to PerplexityBot and CCBot
- A Sitemap: line with your sitemap address
Why it matters
robots.txt decides which search engines and AI assistants can read your pages. Blocking the wrong crawler can remove you from ChatGPT search or Perplexity answers; allowing everything may not suit every business.
How to fix what it finds
- Keep search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed if you want AI citations.
- Block training crawlers only if you do not want your content used for model training.
- Upload the file to the root of your domain as /robots.txt and test it with the robots.txt tester.
Want this handled for you?
RankMesh finds these issues across your whole site every week and ships the fixes after your approval.
robots.txt generator: common questions
At the root of your domain, for example https://example.com/robots.txt. Each subdomain needs its own file.
Block training crawlers if you object to model training. Allow search crawlers like OAI-SearchBot and PerplexityBot if you want AI assistants to cite and link to you.
It stops crawling, not indexing. To keep a page out of results, let it be crawled and add a noindex tag.
No. The file is built in your browser.
Related
- robots.txt tester
- GPTBot and AI crawler checker
- llms.txt generator
- Free report: AI search readiness
- llms.txt generatorDraft an llms.txt for your site from its sitemap and page titles, ready to copy.
- FAQ schema generatorTurn questions and answers into valid FAQPage JSON-LD, ready to paste.
- SERP simulatorPreview your title and meta description as a Google result, with length warnings.
- UTM builderBuild tagged links with utm_source, utm_medium and utm_campaign to track every channel.
- All free SEO tools
Fix it once. Keep it fixed.
RankMesh checks your site every week, writes and publishes content, and tracks how you show up in Google and AI answers. Start free, no card required.
Get a free plan for your site