The short answer
A noindex rule tells Google it may crawl a page but must keep it out of search results. You add it either as a robots meta tag in the page's head, or as an X-Robots-Tag HTTP header for files that are not HTML. Use it on pages that need to exist for users but have no business ranking.
The one rule that trips everyone up: noindex only works if Google can crawl the page. If you also block the URL in robots.txt, Google never sees the noindex, and the page can still show up in results through links. noindex and robots.txt are different tools for different jobs.
- robots.txt controls crawling. noindex controls indexing. They are not interchangeable.
- For noindex to work, the page must stay crawlable: never block it in robots.txt too.
- Use the meta tag for HTML, and the X-Robots-Tag header for PDFs, images and other files.
If you block a URL in robots.txt, noindex will not work
Googlebot has to crawl a page to read its noindex rule. If you Disallow the URL in robots.txt and also add noindex, Google cannot read the noindex, and the page may still appear in results through links pointing at it. Always leave crawling open when you need noindex to take effect.
noindex vs robots.txt: Crawling vs Indexing
Almost every noindex problem comes from one confusion, so it is worth being precise. Crawling and indexing are two separate stages. Crawling is Google fetching the page. Indexing is Google deciding to store it and show it in results. The two controls map onto those two stages:
- robots.txt Disallow stops the crawl. Google never fetches the page. But Google is explicit that this is not a way to keep a page out of search: if other pages link to the URL, it can still be indexed, shown as a bare URL and sometimes with the anchor text of those links.
- noindex lets the crawl happen but blocks the indexing. Google fetches the page, sees the rule, and keeps it out of results. This is the control that actually removes a page from search.
Put them together and they cancel out. Block a page in robots.txt and add noindex, and Google cannot fetch the page, so it cannot see the noindex, so the page can linger in results. If you want a page gone, let Google crawl it and let it read the noindex. If you only want to save crawl budget on pages that are already out of the index, that is where robots.txt belongs. Our guide to optimising your robots.txt covers that side, and the robots.txt generator helps you write the rules without mistakes.
The Syntax
Two ways to deliver the same instruction. The meta tag goes in the head of an HTML page. The HTTP header is for everything that is not HTML.
<!-- HTML pages: robots meta tag in the <head> -->
<meta name="robots" content="noindex" />
<!-- Keep crawlable but prevent indexing, still follow links -->
<meta name="robots" content="noindex, follow" />
<!-- Non-HTML files (PDFs, images): HTTP response header -->
X-Robots-Tag: noindex
<!-- Do NOT also block the URL in robots.txt, or the above
is never seen: Disallow: /secure-page/ <- wrong -->Noindex these
- •Internal search result pages (/search?q=...)
- •Checkout and order confirmation pages
- •Account, dashboard and login pages
- •Admin and internal utility pages
- •Staging or preview pages exposed to the web
- •Thin pages that exist for users but add nothing to search
Never noindex these
- •Your homepage
- •Core product, service and landing pages
- •Blog posts and articles you want to rank
- •Category pages with real, useful content
- •Any page earning backlinks or search traffic
- •Duplicates better handled with a canonical tag
How to Apply noindex, Step by Step
- 1
Confirm the page is crawlable
Check that the URL is not blocked in robots.txt. If it is, remove the block first. A noindex rule on a page Google cannot crawl is a rule Google never reads.
- 2
Add the right directive
For HTML, put <meta name="robots" content="noindex" /> in the page's head. For PDFs, images and other non-HTML files, send the X-Robots-Tag: noindex HTTP header instead.
- 3
Verify with URL Inspection
Use the URL Inspection tool in Search Console and run a live test. A correctly set page reports that it is not indexed because of a noindex rule. If it still says indexable, the tag is not being served where Google can see it.
- 4
Wait for the recrawl, do not re-block
Leave the page crawlable and the tag in place. Google drops it on the next crawl. Resist the urge to also disallow it in robots.txt afterwards, which would stop Google re-confirming the noindex.
A Worked Example
Illustrative scenario, not client data
A team ships a redesign on a Monday. By Thursday, several important landing pages have fallen out of Google and organic traffic to them is near zero. Nothing looks broken: the pages load fine for visitors. Here is the diagnosis.
- Inspect a lost page: URL Inspection in Search Console reports it is excluded by a noindex rule. That is the whole story in one line.
- Trace the tag: the staging environment carried a site-wide
noindexto keep it out of search while in development. The deploy pushed that template to production, so every page inherited it. - Fix it: remove the noindex from production templates, leaving it only where it belongs, and keep the pages crawlable.
- Confirm: run a live URL Inspection again. Once it reports the page as indexable, request indexing to speed up the recrawl. Rankings return as Google reprocesses the pages.
An accidental noindex is one of the fastest, most damaging technical mistakes there is, and one of the easiest to miss because the pages look perfect to humans. It is a classic cause of a sudden ranking drop, covered alongside the others in our guide to why Google rankings drop.
Catch an accidental noindex before it costs you.
Murkuz connects Google Search Console to Claude or ChatGPT, read-only. After a deploy, ask Inspect_URL about a key page: is it indexed, and if not, why? A page excluded by a noindex rule shows up right there, so you find the one that slipped through instead of waiting for the traffic to fall.
A Quick Self-Check
No page you intend to rank carries a noindex tag, especially after a staging deploy.
Every noindexed page is crawlable: none are also blocked in robots.txt.
Non-HTML files you want hidden (PDFs, images) use the X-Robots-Tag header.
Duplicates are handled with canonical tags, not blanket noindex rules.
Removals-tool requests are backed by a permanent noindex, not relied on alone.
Further reading
The clearest material on controlling indexation, starting with Google's own documentation. Each one was read in full before it was added here.
Official docsGoogle Search Central
Block Search indexing with noindexThe primary source. Covers both ways to apply noindex and states the rule this page is built around: for noindex to work, the page must not be blocked by robots.txt.
Official docsGoogle Search Central
Robots meta tag, data-nosnippet, and X-Robots-Tag specificationsThe full list of valid directives (noindex, nofollow, none, nosnippet and more) and when to reach for the X-Robots-Tag HTTP header instead of a meta tag, such as for PDFs and images.
Official docsGoogle Search Central
Introduction to robots.txtGoogle's own statement that robots.txt manages crawling, not indexing, and is not a way to keep a page out of Google. The clearest explanation of why disallow is not a substitute for noindex.
Official docsGoogle Search Console Help
Removals and SafeSearch reports ToolExplains that a removal request hides a URL for only about six months. Useful for understanding why the Removals tool speeds things up but noindex is the permanent fix.
GuideYoast
The meta robots tag explainedA practical reference with a compatibility table across search engines and a clear restatement of why meta robots and robots.txt are different tools that cannot see each other.
Frequently Asked Questions
What is the difference between noindex and robots.txt disallow?
They control different things. A robots.txt Disallow rule controls crawling: it tells Google not to fetch a URL. A noindex rule controls indexing: it tells Google it may fetch the page but must not include it in search results. The two are often confused because both sound like 'hide this page', but they work at different stages. The critical consequence is that for a noindex rule to be seen at all, Google has to be able to crawl the page, which means the page must not also be blocked in robots.txt.
Why does a page I blocked in robots.txt still appear in Google?
Because robots.txt is not a mechanism for keeping a page out of Google. Google says so directly: a URL disallowed in robots.txt can still be indexed if other pages link to it, and Google may show the URL, and sometimes the anchor text of those links, without ever crawling the content. If you need a page kept out of search, allow Google to crawl it and add a noindex rule, or protect it with a login. Blocking it in robots.txt does the opposite of what people expect.
What pages should I noindex?
Noindex pages that have a reason to exist for users but no reason to appear in search: internal search result pages, thank-you and order confirmation pages, account and admin screens, and staging or preview pages that leaked to the web. Be careful with thin or duplicate pages: often a canonical tag or better content is the right call rather than hiding the page. Never noindex pages you want to rank, which includes your homepage, core product and service pages, and articles.
Should I use the meta robots tag or the X-Robots-Tag header?
For normal HTML pages, use the meta robots tag in the page's head: it is the simplest page-level control. Use the X-Robots-Tag HTTP response header when you cannot put a meta tag in the file, which is the case for non-HTML resources such as PDFs, images and other documents. Both deliver the same noindex instruction to Google, so the choice is really about what kind of file you are controlling.
How long does it take for a noindex page to drop out of Google?
It drops out the next time Google crawls and reprocesses the page and sees the noindex rule, which can take days to weeks depending on how often Google crawls that URL. The one thing you must do is keep the page crawlable, so leave it out of robots.txt. If you need it gone faster, the Search Console Removals tool hides a URL quickly, but only for about six months, so it is a temporary measure. The noindex rule is what keeps it out for good.
Should I use noindex or a canonical tag for duplicate content?
Usually a canonical tag. If two URLs show the same or very similar content and you want the content to rank under one address, a canonical tag points Google to the preferred version while still letting it consolidate the signals from both. A noindex rule instead removes a page from search entirely, which is the right choice only when you genuinely do not want that page found at all. Reach for noindex to hide a page, and for a canonical to merge duplicates.