How to use Robots.txt Generator and Path Tester
- Enter Allow/Disallow lines, use editable crawler groups or paste a full supported robots.txt. Existing User-agent groups take priority over the separate build-agent field.
- Optionally add a full HTTP(S) sitemap URL, or start from a preset such as Block AI training crawlers. Presets replace the Rules box, so review them as a starting point for your own crawl policy, not an access control.
- Enter a path or full URL and the crawler name to test. Use an explicit * to test wildcard-group behavior; blank uses the build-agent fallback described below.
- Read the updated local verdict and winning-rule evidence. Download/copy the file and verify an eventual real implementation separately; this page changes no server.
Example: Robots.txt Generator and Path Tester
Build five wildcard-group rules and confirm that /private/report.pdf is blocked for the supplied Googlebot test name.
Options
- User-agent groups
- The User-agent control writes one group for the crawler you name. Paste a file that already carries its own User-agent lines and those groups are kept instead, so a saved file can be checked as it stands.
- Patterns and matching
- Patterns start at the beginning of the tested path. /private/ also matches /private/report.pdf; * matches any character sequence and a final $ requires the path to end there. /search$ matches /search exactly, while /search can match longer paths. This is the supported local pattern model.
- Crawler the rules apply to (User-agent)
- Use * for every crawler, or a name such as Googlebot. Ignored when the rules box already has User-agent lines.
- Rules
- One Allow or Disallow rule per line. # starts a comment. Paste a full file to keep its own User-agent groups.
- Sitemap address
- Optional. Adds a Sitemap line using a complete HTTP or HTTPS address.
- Path to test
- A path such as /private/ or a full address. The site name and the # fragment are ignored.
- Test as crawler
- The declared crawler for the path test. Blank uses the build-agent field, or Googlebot when that field is *. Enter * explicitly to test the wildcard group.
Supported inputs and limits
- Up to 20,000 rule characters, 500 rule lines, ten sitemap URLs and 2,000 characters per tested path/sitemap address.
- Supports User-agent, Allow, Disallow, Sitemap and comments plus the stated wildcard/end-anchor subset.
- Equally specific named groups are combined; explicit subtype fallback is a local model.
- Unsupported directives are noted, not evaluated.
- No live robots.txt, sitemap, crawler access or indexing check; robots rules do not protect private content.
Where your input is processed
This tool does not upload your inputs. See the limits for anything saved on your device.
Crawl access and indexing are two separate decisions
robots.txt tells a crawler which addresses it may request. Indexing is a later decision made by the search engine. An address that is disallowed is often still eligible to be indexed from external links, because the engine can learn about the page without fetching it. When a page must stay out of results, leave it reachable for the crawler and return a noindex directive, because a crawler cannot read a noindex tag on a page it is not allowed to request.
Questions about Robots.txt Generator and Path Tester
Does a Disallow rule remove a page from search results?
No. A robots.txt controls whether a crawler may fetch an address, and it is not a removal tool. An address that is disallowed can still be indexed from links elsewhere, so use a noindex meta tag or an X-Robots-Tag header when a page must stay out of results.
Why does the test say allowed when I expected a block?
Review the selected crawler group and matching rules first. No matching rule permits the path in this model; a tied Allow also wins. Change a rule only if that matches your intended crawl policy, then test overlapping paths as well.
Which rule types can I paste?
User-agent groups, Allow and Disallow lines, Sitemap lines and # comments. A line outside that subset is listed as a note and left out of the verdict, which follows the way Google ignores directives it does not recognise.
Does every crawler follow robots.txt?
Only crawlers that choose to. Well-behaved search engines do, and scrapers often do not. Treat the file as a request that polite crawlers honour, not as access control.
How do I block AI crawlers?
Choose Block AI training crawlers to ask GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent to skip the site while search engines stay allowed. Block all AI crawlers also lists AI search and assistant agents such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. Crawlers choose whether to follow robots.txt.