How to use Robots.txt Generator and Path Tester
- Paste your Allow and Disallow rule lines into Rules, one per line, or keep the example to see how a file is built.
- Set the User-agent the rules apply to, and add a Sitemap address if the file should point at one.
- Type the path you want to check and the crawler name to test as.
- Read the verdict, copy the file, and upload it to the site root before you check it live.
Example: Robots.txt Generator and Path Tester
Block a private folder, keep a public part of it reachable, and confirm the rule catches a real path.
Options
- User-agent groups
- The User-agent control writes one group for the crawler you name. Paste a file that already carries its own User-agent lines and those groups are kept instead, so a saved file can be checked as it stands.
- Patterns and matching
- A pattern matches from the start of the path, so /private/ also covers /private/report.pdf. The * wildcard stands for any characters and a trailing $ anchors the pattern to the end of the path, which is how /*?s=$ targets a search query without catching every later folder.
Supported inputs and limits
Where your input is processed
This tool processes your input in this browser. Your text and files are not uploaded to UseFreeTools. Check this tool's limits for anything it may save on your device.
Crawl access and indexing are two separate decisions
robots.txt tells a crawler which addresses it may request. Indexing is a later decision made by the search engine. An address that is disallowed is often still eligible to be indexed from external links, because the engine can learn about the page without fetching it. When a page must stay out of results, leave it reachable for the crawler and return a noindex directive, because a crawler cannot read a noindex tag on a page it is not allowed to request.
Questions about Robots.txt Generator and Path Tester
Does a Disallow rule remove a page from search results?
No. A robots.txt controls whether a crawler may fetch an address, and it is not a removal tool. An address that is disallowed can still be indexed from links elsewhere, so use a noindex meta tag or an X-Robots-Tag header when a page must stay out of results.
Why does the test say allowed when I expected a block?
Either no pattern matched, or an Allow rule matched with the same length as a Disallow rule. The longest matching pattern wins and Allow wins a tie, so write a longer Disallow pattern or shorten the Allow rule that overlaps it.
Which rule types can I paste?
User-agent groups, Allow and Disallow lines, Sitemap lines and # comments. A line outside that subset is listed as a note and left out of the verdict, which follows the way Google ignores directives it does not recognise.
Does every crawler follow robots.txt?
Only crawlers that choose to. Well-behaved search engines do, and scrapers often do not. Treat the file as a request that polite crawlers honour, not as access control.