Robots.txt vs. noindex: keep a page out of Google Search

Choose the outcome before changing a setting

Use noindex when a page should stay available to visitors but should not appear in Google Search. Use robots.txt to control crawler access. If the information is private, protect access to it instead.

Crawling is fetching a resource. Indexing concerns whether it is included in the search index. A page can be reachable by a crawler while carrying an instruction not to index it.

These choices can look similar in a CMS, but they solve different problems. A setting called “hide from search” needs a check of the published result, not just a saved checkbox.

Google's guide to controlling shared content separates search visibility from password protection. A page being absent from search is not proof that access is restricted.

Why a robots.txt block can leave a URL in search

A crawler must retrieve a page to read its page-level instructions. If robots.txt prevents that retrieval, it can also prevent Google from seeing noindex.

Google's robots.txt introduction explains the consequence: links from other pages can make a blocked URL known to Google. The URL may still appear in results even though Google did not crawl its content. A disallow rule is therefore a poor choice for removing a public web page from search.

Adding noindex inside robots.txt does not fix this. Google does not support that instruction in the file. Put an indexing rule on the actual page or response, and keep that resource accessible to Google when this is your chosen removal method.

The practical question is not “Did I block something?” It is “Can Google retrieve the instruction that expresses my intended outcome?”

Add noindex to the resource you mean to exclude

For an HTML page, the standard page-level form belongs in the document's head:

<meta name="robots" content="noindex">

This is illustrative markup, not a change to your website. If your CMS supplies a per-page indexing setting, use its documented mechanism and inspect the output. Avoid applying a site-wide option when only one page should change.

For a PDF or another non-HTML resource, an HTTP response header can supply the rule:

X-Robots-Tag: noindex

A noindex tag on a web page that links to a PDF does not put that header on the PDF. Check the file's own response. Google's robots metadata reference documents the header and explains that, when robots rules conflict, the more restrictive rule applies.

Keep the implementation narrow. Do not add nofollow, change canonicals or edit unrelated crawler rules merely because they sit beside the indexing option. Each additional change needs its own reason.

A worked example: a public delivery checklist

Consider a fictional furniture shop with three resources:

• /delivery-checklist is a public page linked from order emails. Customers need it, but the shop does not want it in search.
• /delivery-guide.pdf is a separate downloadable copy with the same visibility goal.
• /customer-invoices/ contains private documents.

The owner initially proposes blocking all three paths in robots.txt. That treats public utility content and private customer information as the same problem.

A better brief for the person maintaining this fictional site is:

Keep the checklist available to customers and crawlable by Google, but return a page-level noindex rule. Put an equivalent response header on the downloadable PDF. Protect invoices with enforced access checks. Confirm those three outcomes independently.

For the checklist, first deploy and verify noindex, then remove any robots.txt restriction preventing Google from fetching that specific URL. Do not loosen invoice protection to make an indexing test pass. The checklist example assumes its contents are safe to publish openly.

Record the exact affected addresses before the change. A folder-wide edit could accidentally hide the main delivery service page or expose unrelated resources. A one-page request should come back with one-page evidence.

Verify the published rule and Google's last crawl separately

Start with what your server is serving now. Open the exact page, inspect its HTML and response headers, and check the relevant robots.txt rules. For the PDF, examine the PDF request rather than the page containing its download link. A CMS preview is insufficient evidence of the public response. Also check a neighboring page that should remain indexable, so a mistakenly global setting is caught.

Then use URL Inspection in Search Console for a property you control:

1. Inspect the complete affected URL.
2. Check Crawl allowed?, Page fetch and Indexing allowed? as separate facts.
3. Compare the last crawl with the time you deployed the change.
4. Use the live test to examine the current response; distinguish it from Google's stored indexed data.
5. Recheck the indexed result after Google has revisited the page.

An intentional noindex page should not be treated as a failure simply because an inspection reports that indexing is disallowed. That is the intended outcome. Conversely, a live test seeing the new rule does not prove Google's older search result has already changed.

Google's noindex troubleshooting guidance notes that a revisit can take a long time. Do not promise a fixed removal deadline or repeatedly switch methods because a result has not vanished immediately. Keep the intended noindex response in place; a one-time disappearance is not a reason to remove the instruction.

Handle urgent removal and private content differently

For an urgent result on a property you own, Google's Removals tool can temporarily suppress it. Google describes the block as lasting about six months. It still requires a lasting solution, such as an accessible noindex rule for public content or access protection for private content.

Check the request scope before submitting it. Removing a whole prefix is different from targeting one resource. A temporary search removal does not make a confidential file private or remove it from the internet.

Finish with a small change record: intended audience, affected URL, chosen control, public response evidence, deployment time and last observed Google crawl. That gives the next person a reason to preserve the setting rather than “fix” the intended exclusion.

Keep your discovery inventory consistent too: the XML sitemap guide explains which public indexable URLs belong in that file. For pages you do want discovered, the Google AI search checklist covers eligibility checks. This guide solves the opposite task: intentionally excluding a resource from Google Search while choosing the right access policy.