Guide
What is the robots.txt generator?
The robots.txt generator builds a ready-to-save crawler rules file from User-agent groups, path rules, sitemap URLs, and an optional crawl delay.
When it comes in handy
- Blocking crawler access to internal search, admin, or duplicate parameter paths.
- Giving a specific crawler a different set of rules from the wildcard group.
- Adding sitemap declarations while preparing a site launch.
- Reviewing a generated file for dangerous site-wide blocking before deployment.
How to use
- Add one or more sitemap URLs.
- Enter one User-agent per line in the first group.
- Add Allow and Disallow paths, each beginning with a slash.
- Add more groups when another crawler needs different rules.
- Review warnings, then copy or download robots.txt.
Example: User-agent "*" with Disallow path "/admin/" produces "User-agent: *" followed by "Disallow: /admin/". Entering "/" instead triggers a site-wide blocking warning.
Notes
- Disallow controls crawling, not authentication or guaranteed removal from search results.
- Disallow: / blocks the whole site for the matching group. Deploy it only when intentional.
- Crawl-delay support varies by crawler.
- Upload the final file at the site root so it is available at /robots.txt.
- For curated AI-bot choices, use the AI crawler robots.txt generator.
- After publishing, use the robots.txt checker to inspect the live file and test a path.
