Free generator

Free robots.txt generator

A robots.txt generator writes the plain text file that tells crawlers which parts of your site they may read. This one lets you choose, crawler by crawler, which AI bots to allow or block, add private paths every crawler should skip and your sitemap address, then copy the finished robots.txt. It runs in your browser.

AI crawlers (allowed unless you block them)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

9 of 9 AI crawlers allowed. Upload the file to the root of your domain, then check it with the robots.txt tester.

What this tool covers

  • One User-agent: * group with your disallowed paths
  • A separate block for each AI crawler you choose to block
  • Nine AI crawlers, from GPTBot and ClaudeBot to PerplexityBot and CCBot
  • A Sitemap: line with your sitemap address

Why it matters

robots.txt decides which search engines and AI assistants can read your pages. Blocking the wrong crawler can remove you from ChatGPT search or Perplexity answers; allowing everything may not suit every business.

How to fix what it finds

  1. Keep search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed if you want AI citations.
  2. Block training crawlers only if you do not want your content used for model training.
  3. Upload the file to the root of your domain as /robots.txt and test it with the robots.txt tester.

Want this handled for you?

RankMesh finds these issues across your whole site every week and ships the fixes after your approval.

robots.txt generator: common questions

Where does robots.txt go?

At the root of your domain, for example https://example.com/robots.txt. Each subdomain needs its own file.

Should I block AI crawlers?

Block training crawlers if you object to model training. Allow search crawlers like OAI-SearchBot and PerplexityBot if you want AI assistants to cite and link to you.

Does a Disallow line hide a page from Google?

It stops crawling, not indexing. To keep a page out of results, let it be crawled and add a noindex tag.

Is anything I type stored?

No. The file is built in your browser.

Fix it once. Keep it fixed.

RankMesh checks your site every week, writes and publishes content, and tracks how you show up in Google and AI answers. Start free, no card required.

Get a free plan for your site

Book a 20-minute demo

Tell us a little about your business, then pick a time that suits you.

Before you go: see how AI search sees your site

A free report on whether ChatGPT, Gemini and Google AI recommend your business. No sign-up, results in seconds.