Robots.txt Generator and Path Tester

Build a robots.txt with crawler rows or one-click presets, including AI crawler blocks, and test any path.

On-device No sign-up Free to use

Build rules controls

?

Use * for every crawler, or a name such as Googlebot. Ignored when the rules box already has User-agent lines.

?

One Allow or Disallow rule per line. # starts a comment. Paste a full file to keep its own User-agent groups.

?

Optional. Adds a Sitemap line using a complete HTTP or HTTPS address.

?

A path such as /private/ or a full address. The site name and the # fragment are ignored.

?

The declared crawler for the path test. Blank uses the build-agent field, or Googlebot when that field is *. Enter * explicitly to test the wildcard group.

Processed in your browser. Your inputs stay on this device.

Showing an example. Change the fields, then use build rules to update it.

Result

Showing an example.

Build five wildcard-group rules and confirm that /private/report.pdf is blocked for the supplied Googlebot test name.

Built a robots.txt with 5 rules in 1 group and 1 Sitemap line. /private/report.pdf is blocked for Googlebot.

Loading the tool. Your result will appear here.

https://usefreetools.net/robots-txt-generator · 20 November 2026

Was this helpful?

How to use Robots.txt Generator and Path Tester

  1. Enter Allow/Disallow lines, use editable crawler groups or paste a full supported robots.txt. Existing User-agent groups take priority over the separate build-agent field.
  2. Optionally add a full HTTP(S) sitemap URL, or start from a preset such as Block AI training crawlers. Presets replace the Rules box, so review them as a starting point for your own crawl policy, not an access control.
  3. Enter a path or full URL and the crawler name to test. Use an explicit * to test wildcard-group behavior; blank uses the build-agent fallback described below.
  4. Read the updated local verdict and winning-rule evidence. Download/copy the file and verify an eventual real implementation separately; this page changes no server.

Example: Robots.txt Generator and Path Tester

Build five wildcard-group rules and confirm that /private/report.pdf is blocked for the supplied Googlebot test name.

You add
Crawler the rules apply to (User-agent): *; Rules: # Request that crawlers skip private paths Disallow: /admin/ Allow: /admin/public/ Disallow: /private/ Disallow: /search Disallow: /*?s=; Sitemap address: https://example.com/sitemap.xml; Path to test: /private/report.pdf; Test as crawler: Googlebot
You get
Built a robots.txt with 5 rules in 1 group and 1 Sitemap line. /private/report.pdf is blocked for Googlebot.

Options

User-agent groups
The User-agent control writes one group for the crawler you name. Paste a file that already carries its own User-agent lines and those groups are kept instead, so a saved file can be checked as it stands.
Patterns and matching
Patterns start at the beginning of the tested path. /private/ also matches /private/report.pdf; * matches any character sequence and a final $ requires the path to end there. /search$ matches /search exactly, while /search can match longer paths. This is the supported local pattern model.
Crawler the rules apply to (User-agent)
Use * for every crawler, or a name such as Googlebot. Ignored when the rules box already has User-agent lines.
Rules
One Allow or Disallow rule per line. # starts a comment. Paste a full file to keep its own User-agent groups.
Sitemap address
Optional. Adds a Sitemap line using a complete HTTP or HTTPS address.
Path to test
A path such as /private/ or a full address. The site name and the # fragment are ignored.
Test as crawler
The declared crawler for the path test. Blank uses the build-agent field, or Googlebot when that field is *. Enter * explicitly to test the wildcard group.

Supported inputs and limits

  • Up to 20,000 rule characters, 500 rule lines, ten sitemap URLs and 2,000 characters per tested path/sitemap address.
  • Supports User-agent, Allow, Disallow, Sitemap and comments plus the stated wildcard/end-anchor subset.
  • Equally specific named groups are combined; explicit subtype fallback is a local model.
  • Unsupported directives are noted, not evaluated.
  • No live robots.txt, sitemap, crawler access or indexing check; robots rules do not protect private content.

Where your input is processed

This tool does not upload your inputs. See the limits for anything saved on your device.

Crawl access and indexing are two separate decisions

robots.txt tells a crawler which addresses it may request. Indexing is a later decision made by the search engine. An address that is disallowed is often still eligible to be indexed from external links, because the engine can learn about the page without fetching it. When a page must stay out of results, leave it reachable for the crawler and return a noindex directive, because a crawler cannot read a noindex tag on a page it is not allowed to request.

Questions about Robots.txt Generator and Path Tester

Does a Disallow rule remove a page from search results?

No. A robots.txt controls whether a crawler may fetch an address, and it is not a removal tool. An address that is disallowed can still be indexed from links elsewhere, so use a noindex meta tag or an X-Robots-Tag header when a page must stay out of results.

Why does the test say allowed when I expected a block?

Review the selected crawler group and matching rules first. No matching rule permits the path in this model; a tied Allow also wins. Change a rule only if that matches your intended crawl policy, then test overlapping paths as well.

Which rule types can I paste?

User-agent groups, Allow and Disallow lines, Sitemap lines and # comments. A line outside that subset is listed as a note and left out of the verdict, which follows the way Google ignores directives it does not recognise.

Does every crawler follow robots.txt?

Only crawlers that choose to. Well-behaved search engines do, and scrapers often do not. Treat the file as a request that polite crawlers honour, not as access control.

How do I block AI crawlers?

Choose Block AI training crawlers to ask GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent to skip the site while search engines stay allowed. Block all AI crawlers also lists AI search and assistant agents such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. Crawlers choose whether to follow robots.txt.

Page updated · Project managed by Tony Hines · Report a problem with Robots.txt Generator and Path Tester

Related reading

All guides