Robots.txt Tester
Paste your robots.txt and test whether a URL path is allowed for a given crawler.
How it works
- 1
Paste your robots.txt
The full file content.
- 2
Enter a path and user agent
For example /private/page and Googlebot.
- 3
Read the verdict
Allowed or blocked, with the matching rule.
Related tools
Other tools people use right after this one.
Built to get out of your way
487
Tools available
6
Languages
0
Watermarks added
2h
Until your files are erased
100%
Free to start
∞
Devices supported
Frequently asked questions
The longest matching path; on a tie, Allow wins.
Test robots.txt rules before you publish
A single wrong Disallow can hide your whole site from search engines. Testing a few important URLs against your rules catches mistakes before a crawler does.
The tester follows the standard rules: the most specific (longest) match wins, Allow beats Disallow on a tie, and * and $ are supported.