URL access under robots.txt rules

Site owners and indexing specialists can test a URL before publication whenever its treatment under the current robots.txt rules is uncertain.

Check whether a URL is allowed by your robots.txt rules, so an unexpected restriction can be spotted before the page is published.

How to use

Paste the content of your robots.txt file into the first part of the input area.
Add the separator shown by the tool, then type the path or address you want to test in the second part.
Run the test to see whether the path is allowed or blocked under the rules you provided.
If the result is surprising, change the rules or the path and test again to compare outcomes.

Example

The sample pairs a rule that blocks a folder with a path inside that folder. After testing, check whether the displayed decision matches the rule and whether the selected crawler is the one you meant to check.

Input

User-agent: Googlebot
Disallow: /admin/
--
/admin/page

Limits and formats

The test uses only the robots.txt content you paste, not the file actually published on a live domain.
Rules depend on order and prefix: a more specific rule can change the decision for the same path.
The tool does not verify whether search engine crawlers truly obey your directives.
A path allowed here may still stay out of search results because of a noindex tag, a password, or another signal.

Where processing happens

The check runs locally in your browser. The robots.txt content and the tested path stay on your device, with no request to an external site, and the decision appears directly on the page.

Frequently asked questions

Why does the test not match what Google does?

The test applies the rules you pasted. Google may use a different version of the file, rely on a cached copy, or apply other signals. Check the file that is actually reachable online before drawing conclusions.

Is the separator between rules and path required?

Yes, the tool needs a clear separation between the robots.txt content and the path being tested. Without it, the two parts cannot be told apart, and the result may be inconsistent or impossible to produce.

Can I test several addresses at once?

The tool usually handles one path at a time. To check multiple addresses, run the test again for each path and record the decisions. That helps reveal rules that are too broad or poorly ordered.