How to use Meta Robots Tag Generator
- Choose whether indexing or following links should be restricted. The defaults need no tag.
- Pick a text-snippet rule and set a maximum only when that rule is selected.
- Choose all search crawlers or Googlebot, then choose HTML meta tag or X-Robots-Tag header.
- Copy the code into the page head or server response and inspect the published response.
Example: Meta Robots Tag Generator
Request that a public draft page stay out of search results.
Supported inputs and limits
Where your input is processed
This tool processes your input in this browser. Your text and files are not uploaded to UseFreeTools. Check this tool's limits for anything it may save on your device.
Place the rule where a crawler can read it
A robots meta tag belongs in an HTML response, while an X-Robots-Tag belongs in an HTTP response header. Google documents both forms and notes that restrictive rules only work after its crawler can reach the URL. Search engines can differ, so check the published response and the crawler-specific documentation when a rule is important.
Google Search Central: Robots meta and X-Robots-Tag specifications
Questions about Meta Robots Tag Generator
Do I need an index, follow tag?
No. Those are the usual default behaviors; the tool says no tag is needed when no restriction is selected.
Will noindex work on a page blocked by robots.txt?
A crawler must fetch the page to read the directive. Allow crawling and put noindex in the page or response if indexing is what you need to control.
When should I use X-Robots-Tag?
Use the response header when you need a robots rule for a non-HTML resource, such as a PDF, or when your server manages rules centrally.
Does Googlebot-only affect all search engines?
No. It targets Googlebot. Choose the general robots name when you want to address search crawlers more broadly, then test your actual setup.