Robots.txt Generator
Quick Presets
Crawler Rules
Global Options
Generated robots.txt
User-agent: * Disallow:
Tip: Upload this file as https://yourdomain.com/robots.txt
Rules are applied in order. An Allow rule overrides a Disallow for the same path.
Empty Disallow: means allow everything.
Privacy First: Your files are processed securely. Whenever possible, processing happens in your browser and files are never stored on our servers.
Create a perfectly formatted robots.txt file without writing a single line manually. Choose from five ready-made presets (Allow All, Block All, WordPress, E-Commerce, Block SEO Spiders) or build custom rules from scratch. For each User-agent block you can pick from 24 common bots (Googlebot, Bingbot, AhrefsBot, SemrushBot, and more), add Disallow paths using a quick-pick list of 16 common patterns, add Allow overrides, and set a Crawl-delay in seconds. Add a global Sitemap URL and an optional Host directive for Yandex. The generated robots.txt updates live in the preview panel as you build. Copy to clipboard with one click or download the file ready to upload to your domain root. No account required.
How to Use
- 1Choose a preset (Allow All, WordPress, E-Commerce, etc.) or start with a blank rule
- 2Click a rule block to expand it and set the User-agent - use '*' for all bots or pick a specific bot from the dropdown
- 3Add Disallow paths for pages you want to block - pick from common patterns or type custom paths
- 4Add Allow paths to override a Disallow for specific sub-paths (e.g. allow admin-ajax.php while blocking /wp-admin/)
- 5Optionally set a Crawl-delay in seconds to reduce server load
- 6Add your Sitemap URL in the Global Options section
- 7Click 'Add User-Agent' to add rules for additional bots (e.g. separate rules for Googlebot and AhrefsBot)
- 8Copy the output or click Download to save robots.txt, then upload to your domain root
Features
- 5 ready-made presets (WordPress, E-Commerce, etc.)
- 24 common bots in dropdown
- 16 common path shortcuts
- Disallow + Allow rules per bot
- Crawl-delay support
- Sitemap & Host directives
- Live preview as you build
- One-click copy or download
Frequently Asked Questions
/ related tools
Community Feedback
Like
Total Rating
No ratings yetReviews
No reviews yet. Be the first!
Rate & Review
Was this tool helpful?
Rate this tool:
Write a Review