What is the AI crawler robots.txt generator?
A free, browser-based tool that builds a robots.txt block for AI and answer-engine crawlers. Toggle allow or block for each bot — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and more — and copy a ready-to-paste robots.txt. Each crawler has a one-line note on what it does so you can decide deliberately. Everything runs locally in your browser, so nothing is uploaded.
Block AI training while staying in answer engines
Not all AI crawlers do the same thing. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data to train models, while answer-engine indexers (OAI-SearchBot, PerplexityBot) index pages so they can surface and cite your site. A common stance is to block training but allow indexing — keeping referral traffic from ChatGPT search and Perplexity. This tool defaults to exactly that, and you can adjust any bot.
How to use it
- For each AI crawler, choose Allow or Block. Read the one-line note if you're unsure.
- Use Allow all / Block all to start from a baseline, then fine-tune.
- Copy the generated robots.txt and add it to the file at the root of your site (https://example.com/robots.txt). Append the groups if you already have a robots.txt.
Common use cases
- Blocking GPTBot, ClaudeBot, and CCBot so your content isn't used for AI training.
- Allowing OAI-SearchBot and PerplexityBot to keep your site visible in answer engines.
- Reviewing the full list of named AI crawlers before deciding a site-wide policy.
