Meta robots tag generator

How to build a robots meta tag that does what you meant

Pick the restrictions, add per-crawler exceptions if you have them, and read the effective rule. Three of the checks exist because the tools ranking for this query get the semantics wrong, not the syntax.

1. Decide what to restrict. Only restrictions matter. There is no "index" rule in Google's list, and "all" has no effect when listed explicitly, so a page you want indexed needs no tag at all.
2. Say whether robots.txt blocks the URL. If it does, and you asked for noindex, the generator stops. A blocked page is never fetched, so the noindex is never read — the single most common way a page refuses to leave the index.
3. Add per-crawler lines as restrictions only. Rules combine as the sum of the negative rules. A bot-specific tag can tighten what applies to that crawler; it cannot loosen what you set for everyone.
4. Take the header version for anything that is not HTML. PDFs, images, CSVs and generated downloads have no head to put a tag in. X-Robots-Tag carries the identical rules as a response header, with Apache and nginx snippets provided.
5. Handle AI crawlers in robots.txt, not here. Every AI crawler opt-out published by Google, OpenAI, Anthropic, Perplexity, Apple, Meta and Common Crawl is a robots.txt user-agent token. The third output is that block.

noindex and Disallow cancel each other out

A meta robots tag generator builds the <meta name="robots"> tag that tells search engines whether to index a page, follow its links, and how much of it to show. The most expensive mistake it can help you make is not a typo. It is combining that tag with a Disallow line.

Google's own wording: for the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results — from links pointing at it, with no snippet and no title you chose.

This is the loop people get stuck in. A page will not leave the index, so they block it in robots.txt as well, which guarantees it never leaves. And the obvious shortcut is closed too: specifying noindex in robots.txt is not supported by Google. The fix is always the same order of operations:

1. Remove the Disallow line so the page can be crawled.
2. Serve noindex — as a meta tag on HTML, as an X-Robots-Tag header on anything else.
3. Wait for a recrawl. Only after the page is out of the index does blocking crawling make sense, and usually it no longer needs to.

The checkbox in the form is there so the generator can refuse. Neither of the two tools currently ranking first and sixth for this query mentions Disallow anywhere on the page.

Per-crawler rules add up — they do not override

You can address one crawler by name: <meta name="googlebot">, <meta name="googlebot-news">, <meta name="bingbot">. What most explanations get backwards is how that combines with the generic tag.

Google's documentation: for situations where multiple crawlers are specified along with different rules, the search engine will use the sum of the negative rules. The example given is a generic nofollow plus a Googlebot noindex, and the result for Googlebot is noindex, nofollow — both, not the more specific one. The same principle is stated for headers: in the case of conflicting rules, the more restrictive rule applies.

Which means this pair does not do what it looks like it does:

<meta name="robots" content="noindex">
      <meta name="googlebot" content="index">      <!-- does not re-admit the page -->

There is no positive rule that undoes a negative one. To be indexed by Google and nothing else, you restrict the others; you do not permit Google. The generator computes the effective rule for every crawler you name and warns whenever a line is trying to loosen rather than tighten.

One more consequence, cheap to state and easy to miss: nosnippet and max-snippet:50 on the same page produce no snippet, because the more restrictive rule wins.

Directives that no longer do anything

Google publishes a list of rules it does not use, and generators keep offering them without a note. Checked on 2026-09-04:

The first-ranking tool for this query offers noarchive and nositelinkssearchbox as working directives with no caveat. This generator will still emit noarchive if you ask — other engines may read it — but it tells you Google does not. And it flags an index, follow tag for what it is: a tag that changes nothing, on a page that was already indexable.

Which leaves a useful rule of thumb. Ship a robots tag only when you are restricting something. If you have nothing to restrict, the best tag is no tag.

AI crawlers: the switch is in robots.txt, not in a meta tag

Plenty of advice suggests <meta name="noai">, or putting Google-Extended in a meta tag. We went and read each vendor's own documentation on 2026-09-04. Neither works, and here is the actual shape of it.

• Google-Extended has no user-agent string at all. Google's crawler documentation says crawling is done with existing Google user agents and the token is used in a control capacity in robots.txt. There is nothing for a meta tag to address. It governs Gemini training and grounding, and does not affect Google Search or ranking.
• OpenAI, Anthropic and Perplexity all document robots.txt tokens — GPTBot, OAI-SearchBot, ChatGPT-User; ClaudeBot, Claude-SearchBot, Claude-User; PerplexityBot, Perplexity-User. Anthropic's help page says opting out requires modifying the robots.txt file. None of the three documents a meta tag.
• noai and noimageai appear in none of those documents. They are a publisher proposal, not something a major engine states it honours.
• User-triggered fetchers are a different category. Perplexity documents that Perplexity-User generally ignores robots.txt because a person asked for the fetch; Meta says the same about Meta-ExternalFetcher. A robots.txt line is not a wall for those.

Two things on the meta side genuinely reach AI answers, and both are ordinary Google directives: nosnippet and max-snippet explicitly apply to AI Overviews and AI Mode, and limit how much of the page may be used as direct input to them. Apple's documentation adds that nosnippet is also the way to opt out of broad world-knowledge answers in its products. That is the honest list.

The third output here writes the robots.txt block for whichever policy you pick, grouped by vendor. Blocking the answer-side crawlers is a real choice with a real cost — it removes you from the results those engines cite. If being cited is the goal instead, the material is on GEO tools, grouped by what they actually do and in the llms.txt validator.