Meta robots and X-Robots-Tag: every rule, the syntax and how conflicts resolve

What the robots meta tag and X-Robots-Tag are

Both are page-level instructions. They do not control crawling of the whole site, as robots.txt does. They tell Google what to do with one specific URL once it has been fetched: index it or not, follow its links or not, and how much of it to show as a snippet.

The robots meta tag sits in the <head> of an HTML page. You can edit it in your CMS or template, and it applies only to the page it is on.

The X-Robots-Tag is an HTTP response header. Your server sends it together with the file, so it works for anything the server returns: HTML, PDFs, images, feeds. Files without an HTML <head> cannot carry a meta tag, so for them the header is the only way to set a noindex.

<head>
  <meta name="robots"
        content="noindex">
</head>

HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex

Without any robots rule, a page is treated as index, follow. That is the default, so you never need to write it.

What Google officially says

One more point from the same documentation matters in practice: if robots.txt disallows crawling of a page, Google cannot see its robots meta tag or X-Robots-Tag, so those rules are ignored.

Every rule Google supports

The middle column is Google's definition, quoted word for word. The right column is when you would actually use the rule.

Syntax and examples

The basic meta tag

Put the tag in the <head> of the page. The name attribute says which crawlers the rule is for; robots means all of them.

<meta name="robots" content="noindex">

Rules only for Google

Replace robots with one of the two user agent tokens Google supports. googlebot covers all text results; googlebot-news applies to Google News.

<meta name="googlebot" content="noindex">
<meta name="googlebot-news" content="nosnippet">

Combining rules

Separate several rules with commas in one tag, or use several tags. Both versions below mean the same thing.

<!-- one tag -->
<meta name="robots" content="max-snippet:120, max-image-preview:large">

<!-- two tags -->
<meta name="robots" content="max-snippet:120">
<meta name="robots" content="max-image-preview:large">

X-Robots-Tag in nginx

To keep every PDF on the site out of search, add the header in a location block that matches PDF files:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex";
}

Keep in mind that in nginx an add_header inside a location replaces the add_header lines inherited from the server level, so repeat any other headers you need there.

X-Robots-Tag in Apache

In the virtual host config or .htaccess, use a FilesMatch block. The Header directive needs mod_headers enabled.

<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

PDFs, images and other files

Change the pattern to cover other file types, for example \.(pdf|docx)$. For images, the header on the image file itself keeps that file out of search. If you only want to stop Google indexing images from one HTML page, use noimageindex on that page instead.

Conflicts and precedence

Google reads every robots rule it finds for a URL: the meta tags, the googlebot tag and the X-Robots-Tag header. If they disagree, the more restrictive rule applies. A page that says index in one place and noindex in another ends up with noindex.

The meta tag and the header are not ranked against each other. Neither one overrides the other; Google simply takes the stricter of the two.

robots.txt blocking

robots.txt controls crawling, not indexing. If a URL is disallowed, Google never fetches it, so it never reads the robots meta tag or the X-Robots-Tag header. If other pages link to that URL, it can still appear in search, usually without a description.

So the rule is simple: if you want a page out of the index, let Google crawl it and show it the noindex. Never combine noindex with a robots.txt Disallow on the same URL.

noindex, nofollow and links

Page-level nofollow vs rel="nofollow"

A nofollow in the robots meta tag applies to every link on the page. A rel attribute applies to one link only. Use the attribute on single links: rel="sponsored" for paid links, rel="ugc" for links in comments and forum posts, rel="nofollow" for other links you do not want to vouch for.

<a href="https://example.com/partner/" rel="sponsored">Our partner</a>

The link-level values nofollow, sponsored and ugc are hints, not strict commands.

What happens to links on noindex pages

A noindex page can still have its links followed for a while. Over time, though, pages that stay noindex are crawled less often, and a long-term noindex, follow tends to end up treated like noindex, nofollow. Do not rely on noindex pages to pass links. If a page is an important path to other pages, keep it indexable or make sure those pages are linked from elsewhere too.

When to use noindex and when not to

For staging sites, a noindex works, but HTTP authentication is better: it keeps out both crawlers and curious visitors, and there is no tag to forget when the site goes live.

noindex, canonical, 301 or robots.txt

• noindex keeps a page out of search results. Google still has to crawl the page to see the rule.
• rel="canonical" tells Google which of several duplicate URLs you prefer. Signals are consolidated on that URL, which is normally the only one shown. It is a hint, not a command.
• 301 redirect sends users and Googlebot from the old URL to the new one. Use it when the old URL is no longer needed.
• robots.txt Disallow stops crawling. It does not stop indexing, and it hides any noindex on the blocked URL.

Snippet controls

Some rules do not affect indexing at all. They only limit how much of the page Google may show.

• nosnippet removes the text snippet and video preview. The page can still rank, but results with no snippet give searchers less reason to click.
• max-snippet:[number] caps the snippet at that many characters.
• max-image-preview:[setting] sets the largest image preview: none, standard or large. It controls the preview size, not whether the image is indexed.
• max-video-preview:[number] caps the video preview at that many seconds.

Hiding part of a page: data-nosnippet

Google says: “You can designate textual parts of an HTML page not to be used as a snippet.” Add the data-nosnippet attribute to a span, div or section element. The text stays on the page and is still indexed; it just is not quoted in the snippet.

<p>Our standard delivery takes 3–5 days.
  <span data-nosnippet>Prices shown are valid today only.</span>
</p>

Snippet controls and AI features

Snippet controls such as nosnippet and max-snippet also limit how your content is shown in AI features in Google Search. That cuts both ways: it keeps your text from being quoted, but nosnippet can also reduce how visible the page is. Use it on specific pages for a clear reason, not site-wide.

Removing a page from Google fast

1. Add noindex to the page, or make it return 404 or 410 if it should be gone for users too.
2. Keep it crawlable. Do not block the URL in robots.txt, or Google will never see the change.
3. Request indexing in Search Console URL Inspection so Google recrawls the page sooner.
4. Use the Removals tool if it is urgent. It hides the URL from results only temporarily, so it never replaces step 1.

How to check what Google sees

URL Inspection

In Search Console, inspect the URL and look at Indexing allowed? in the page indexing details. If it says no, Google found a noindex. Run a live test to check the current version of the page rather than the last crawl.

Response headers with curl

The X-Robots-Tag never appears in the page source, so check the response headers directly:

$ curl -I https://example.com/files/price-list.pdf
HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex

Browser developer tools show the same information in the Network tab, under the response headers.

Raw and rendered HTML

Compare the HTML your server sends (view source, or curl without -I) with the rendered page in URL Inspection. The robots meta tag should be the same in both.

noindex and JavaScript

If the server HTML contains noindex, Google may skip rendering the page, so removing the noindex with JavaScript may not work. Adding noindex with JavaScript is also unreliable. Keep robots rules in the server response.

Common mistakes

• ✕noindex left over from stagingThe site launches with the staging noindex still in the template or the CMS setting.
• ✕noindex on paginated pagesPage 2, 3 and so on often hold the only links to older products or posts.
• ✕noindex plus robots.txt DisallowThe block stops Google from ever reading the noindex.
• ✕noindex plus a canonical to another URLOne signal asks Google to drop the page, the other says it duplicates another URL. Pick one.
• ✕Two sources printing robots tagsAn SEO plugin and the theme both output a robots meta tag, and the stricter one wins.
• ✕Typos in the valueno-index or no index is not a rule Google knows, so nothing happens.
• ✕noindex URLs in the sitemapThe sitemap asks for indexing while the page refuses it. List only indexable URLs.

FAQ

Sources

• Google Search Central — Robots meta tag, data-nosnippet and X-Robots-Tag specifications
• Google Search Central — Block search indexing with noindex
• Google Search Central — How to specify a canonical URL

Quotes checked against the official pages on 29 October 2026.