Robots.txt Rule Tester
Note: Many sites block cross-origin fetches. Paste manually if fetch fails.
Privacy First: Your files are processed securely. Whenever possible, processing happens in your browser and files are never stored on our servers.
Validate and debug your robots.txt file right in the browser. Paste robots.txt content directly or enter a domain URL to fetch it automatically. Choose a user-agent: Googlebot, Bingbot, GPTBot, ClaudeBot, AhrefsBot, SemrushBot, wildcard (*), or custom. Test a single URL or batch-test up to 100 URLs. Parses groups, Allow/Disallow rules, Crawl-delay, and Sitemap directives per RFC 9309. Pattern matching supports * wildcards and $ end-of-path anchors. For each URL the result shows: Allowed or Blocked, matched rule line number and pattern, applied user-agent group. The parsed structure panel shows all groups and rules with line numbers and highlights syntax errors. All parsing is 100% client-side JavaScript.
How to Use
- 1Paste your robots.txt content into the left text area, or enter your domain URL and click Fetch
- 2Select the user-agent to test as - choose from common bots or enter a custom one
- 3Type the URL path to test in the 'URL to test' field (e.g. /admin/settings)
- 4Optionally add multiple URLs in the Batch Test area - one per line
- 5Click 'Test URL' to run the analysis
- 6Expand 'Parsed Structure' to see all groups, rules, line numbers, and any syntax warnings
Features
- Paste robots.txt manually or auto-fetch from any domain URL
- User-agent selection: Googlebot, Bingbot, GPTBot, ClaudeBot, AhrefsBot, SemrushBot, *, or custom
- Single URL test with Allowed / Blocked result badge
- Batch URL test (multiple URLs, one per line) - up to 100 URLs
- Shows matched rule type (Allow/Disallow), line number, and pattern
- Parsed structure view: all user-agent groups, rules, crawl-delays, sitemaps
- Syntax warning panel for malformed directives
- RFC 9309 compliant: Allow wins on equal-length ties; specific UA overrides wildcard
- Pattern matching: * wildcard and $ end-of-path anchor
- 100% client-side JavaScript - no server requests
Frequently Asked Questions
/ related tools
Community Feedback
Like
Total Rating
No ratings yetReviews
No reviews yet. Be the first!
Rate & Review
Was this tool helpful?
Rate this tool:
Write a Review