What Is the X-Robots-Tag Header?

The X-Robots-Tag is an HTTP response header that carries the same indexing instructions as the robots meta tag — noindex, nofollow, noarchive and the rest — but delivers them in the server's response instead of the page's HTML. That makes it the only way to control indexing of files that have no <head>: PDFs, images, videos, CSVs and other non-HTML responses.

It's also the directive people forget to check. A meta tag is visible to anyone who views source; a header is invisible unless you go looking for it. That asymmetry is why a page can look perfectly indexable and still be excluded, and why a link placement can quietly be worth nothing.

X-Robots-Tag vs the robots meta tag

They express the same directives and Google treats them as equivalent. The difference is delivery, and therefore reach.

	`<meta name="robots">`	`X-Robots-Tag`
Where it lives	In the HTML `<head>`	In the HTTP response headers
Works on non-HTML files	No	Yes — PDFs, images, video, JSON, CSV
Visible in view-source	Yes	No
Applied per URL or in bulk	Per page, in the template	Per URL or by pattern in server config
Who usually sets it	Developer or CMS plugin	Server, CDN or edge config

One thing that surprises people: if a page carries both and they conflict, the more restrictive instruction wins. A noindex in the header beats an index in the meta tag. So "I removed the noindex from the template" doesn't mean the page is indexable — the header may still be saying otherwise.

The syntax

The header value is a comma-separated list of directives, optionally prefixed by a crawler name:

X-Robots-Tag: noindex
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: noindex
X-Robots-Tag: unavailable_after: 2027-01-31T00:00:00+00:00

Without a crawler prefix, the rule applies to every crawler that reads the header. With a prefix, it applies only to that agent — and, exactly as with robots.txt, a crawler that has its own named rule may ignore the unnamed one, so mixing the two forms on the same URL is a reliable way to confuse yourself six months later.

The directives worth knowing:

noindex — keep this URL out of search results.
nofollow — don't pass ranking signals through links in this document.
none — shorthand for noindex, nofollow.
noarchive — don't show a cached copy.
nosnippet / max-snippet: [n] — suppress or cap the text snippet shown. max-snippet: 0 is effectively nosnippet.
noimageindex — don't index images found on this page.
unavailable_after: [ISO 8601 date] — drop the URL from results after a date. Genuinely useful for time-boxed content like an event or a limited registration page.

Google's robots meta tag and X-Robots-Tag documentation is the authoritative list; treat anything not on it as unsupported.

When you actually need it

Four situations where the header is the right tool rather than a preference:

Non-HTML files. A PDF price list, a CSV export, a generated invoice — none of them can carry a meta tag. If you don't want them competing in search results (or turning up for your brand name), the header is the only mechanism.
Bulk rules by pattern. Excluding every URL under /downloads/, or every file ending .pdf, is one server rule. Doing the same through a CMS is dozens of manual settings.
Responses you don't template. API endpoints, feeds, search-result pages generated by a framework, anything served before your page template runs.
Staging and preview environments. A header applied at the host level can't be forgotten in a single template, which is one fewer way to accidentally get a staging site indexed. Note that HTTP authentication is stronger still — a noindex only works if the crawler is allowed to fetch the URL and read the header.

What it is not for: making something private. The header controls search-result inclusion, nothing else. Anyone with the URL can still open the file.

How to set it

The mechanics depend on where your response is produced.

Apache (.htaccess or vhost config):

<FilesMatch "\.(pdf|csv)$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

Nginx:

location ~* \.(pdf|csv)$ {
  add_header X-Robots-Tag "noindex";
}

A Next.js or Node app sets it on the response like any other header — in next.config.js under headers(), in middleware, or per-route. A CDN or edge worker (Cloudflare, Fastly, Vercel) can add it without touching the origin at all, which is convenient and also the reason a header sometimes appears that nobody on the team remembers configuring.

One constraint applies everywhere: the URL must be crawlable for the header to be read. Blocking a URL in robots.txt and setting X-Robots-Tag: noindex on it is self-cancelling — the crawler never makes the request, so it never receives the instruction. What is a robots.txt file covers why that pairing keeps URLs in the index rather than removing them.

How to check a URL for one

Because it's invisible in view-source, you have to ask the server directly:

curl -sI "https://example.com/page" | grep -i x-robots

Three other routes to the same answer:

Search Console → URL Inspection. Under "Indexing allowed?" it names the directive and where it came from. This is the version that tells you what Google actually saw.
Browser DevTools → Network tab. Click the document request, read the Response Headers.
A crawler like Screaming Frog or Sitebulb, which surfaces the header as a column across a whole site — the practical option when you're auditing rather than spot-checking.

Check the header and the meta tag. Checking only the HTML is the single most common reason an indexing problem survives a diagnosis.

Where this quietly breaks link building

A noindex header on the page hosting your backlink has the same effect as a noindex meta tag: the link is crawled at first, then progressively discounted as Google stops recrawling a page it will never show. Do links on noindexed pages pass SEO value walks through the decay curve and what to do when you find one.

The reason it matters more in header form is discoverability. A publisher who noindexes their guest-post section via a meta tag can be caught in ten seconds. One who does it at the CDN cannot be caught at all unless you check headers — and a paid or traded placement on such a page looks identical to a good one in every reporting tool you'd use. How to check if a backlink is indexed covers the full set of reasons a live link might not be counting.

This is the practical argument for verification that runs continuously rather than once. Backlinkster confirms each side of a swap by code and keeps rechecking that the placement is still live on a page that can actually be indexed, so a header added months after the deal shows up as a broken swap instead of sitting in a spreadsheet as a win. Free accounts include five verified swaps a month; see the plans.

Frequently asked questions

What is the X-Robots-Tag header used for? It delivers indexing directives — noindex, nofollow, noarchive, nosnippet and others — in the HTTP response rather than in the HTML. Its main use is controlling files that can't carry a meta tag, such as PDFs and images, and applying rules in bulk by URL pattern at the server or CDN.

How do I noindex a PDF? Set X-Robots-Tag: noindex on the response for that file, usually with a server rule matching .pdf. A PDF has no <head>, so there is no meta-tag alternative. Make sure the file isn't also blocked in robots.txt, or Google will never fetch it and never read the directive.

Is X-Robots-Tag better than the robots meta tag? Neither is better — Google honours both identically for HTML pages. Use the header when the response isn't HTML, when you want a pattern-based rule across many URLs, or when you can't reliably edit the template. Use the meta tag when a single page needs a one-off directive.

Why can't I see the X-Robots-Tag in the page source? Because it isn't in the source. It's part of the HTTP response headers, which are sent before the body and never rendered. Inspect it with curl -sI, your browser's Network tab, or Search Console's URL Inspection tool.

Does X-Robots-Tag stop Google crawling the page? No. Crawling still happens — that's how the directive is received. It controls whether the URL appears in search results. To reduce crawling you'd use robots.txt, which is a separate control with a separate effect on crawl budget.

What happens if the header and the meta tag disagree? The more restrictive directive applies. A noindex in either location keeps the page out of results, regardless of what the other says.

The bottom line

Treat the X-Robots-Tag as the invisible half of your indexing configuration. It's the correct tool for non-HTML files and bulk rules, it overrides a permissive meta tag, and it can't be spotted by looking at a page. Add curl -sI | grep -i x-robots to your routine for any URL you're relying on — your own pages that should rank, and any page hosting a link you paid for or traded for.