Excluded by ‘noindex’ tag: where it came from and how to undo it

In Search Console, open Indexing, then Pages, and scroll to the table called “Why pages aren't indexed”. Click the row “Excluded by ‘noindex’ tag”. You get a chart of how many URLs have had this status over time and a list of example URLs.

Read the URLs before the count. A noindex on the wrong page is a problem; a noindex on a page nobody should land on from a search is the setting doing its job.

Then read the chart. A count that has been flat for months usually reflects a setting somebody chose long ago. A count that climbs from one particular date points to a change made around then: a deploy, a plugin or theme update, a migration, a new CDN rule. Write that date down. It is the best clue you will get, and you will use it in the section on finding the source.

A related status: “Submitted URL marked ‘noindex’”

Older reports and many forum threads use this wording for a page that is noindexed and also listed in your sitemap. The sitemap asks Google to index the page and the page asks Google not to, so one of the two is wrong. Either remove the noindex or take the URL out of the sitemap.

With URL Inspection

Paste one of the listed URLs into the inspection bar at the top of Search Console and choose Test live URL. The indexed view is Google's copy from its last crawl; the live test fetches the page now. In the result, look at Indexing allowed?. If it says No, the detail tells you whether the noindex came from the meta tag or the header.

From a terminal

Quicker when you have more than a handful of URLs. First the header. This makes a normal GET request and prints only the response headers, because some servers answer a HEAD request (curl -I) differently from a GET:

Then the meta tags. The pattern below catches robots and googlebot, single or double quotes, and the attributes in either order:

One case curl can't see: a page whose robots tag is written by JavaScript in the browser. curl prints the HTML the server sent, before any script runs. For a page built in the browser, use the URL Inspection live test and look at the HTML Google rendered.

A few rules decide what happens when a page carries more than one of these.

• The more restrictive rule wins. Google's documentation says it uses “the sum of the negative rules”. A page with index in one place and noindex in another is noindexed.
• Case doesn't matter. NOINDEX, NoIndex and noindex are the same rule, in the tag, the header name and the value.
• The tag doesn't have to be in the head. Google says it will respect a robots meta tag in the body, so a noindex rendered by a component halfway down the page still counts.
• Google reads two names besides robots: googlebot and googlebot-news. The second one is for Google News only. Google ignores other names; other search engines read their own (Bing, for example, has bingbot).
• A user agent in the header limits it. X-Robots-Tag: googlebot: noindex applies to Google and nobody else. Without a user agent the header applies to every crawler.

1. 01

   Find a page that is fine

   Pick an indexed page of the same type as a noindexed one, and one of a different type. If only one section is affected, the cause is that section's template or a CMS setting for that content type. If every page is affected, look at site-wide settings, the server and the CDN.

2. 02

   Header or tag?

   A header comes from the web server, the hosting platform or the CDN, never from the page's template. A meta tag comes from the CMS, an SEO plugin, the theme or the framework. The two tests in the previous section tell you which.

3. 03

   Check the date

   Go back to the date the chart started rising. Look at what was deployed, updated or reconfigured in the days before it: a release, a plugin or theme update, a CDN rule, a change of host.

4. 04

   Search the configuration too

   A header rule usually lives in a dashboard or a config file that is not reviewed with the code: the CDN's rules, the host's settings, an nginx or Apache config, an edge function. Search all of them for X-Robots-Tag.

Changes

Where to look, platform by platform

How a staging noindex reaches production

Staging sites are noindexed on purpose, and that should stay. The failure is in how the code decides which environment it is in. These are the three routes that come up again and again in practice.

• A variable that isn't set. The template adds noindex unless APP_ENV is production. The new production server doesn't set APP_ENV at all, the check falls back to its default, and every page ships with a noindex. Nothing errors, because a missing variable is not an error.
• Config frozen at build time. Some frameworks bake configuration into the build. If the build ran with staging's settings, production serves staging's answer whatever its own environment says.
• A header rule copied across. Somebody adds X-Robots-Tag: noindex at the CDN for the staging hostname. Later the rule is copied to production, or written for a path instead of a hostname. It never appears in the repository or in view-source.

1. 01

   Fix the setting that adds it

   Change the setting, template, config or rule that adds the noindex. A page edited by hand gets the directive back from its template on the next deploy.

2. 02

   Purge the caches

   A CDN or page cache can keep serving the old headers and HTML after the fix. Purge it, then test again.

3. 03

   Test the live page again

   Run the two curl commands, or the URL Inspection live test, on a few pages from different sections. Both the header and the tags have to be clean.

4. 04

   Ask Google to recrawl the pages that matter most

   In URL Inspection, choose Request indexing for your most important URLs. There is a daily limit per property, so save it for the pages that earn traffic.

5. 05

   Resubmit the sitemap and validate the fix

   For many pages, resubmit the sitemap in the Sitemaps report. Then go back to the noindex row in the Page indexing report and choose Validate fix, which makes Google check the affected URLs.

Pages already in the index don't disappear the moment the directive appears. Each one goes when Google next crawls it. Pages Google visits often, such as the home page and the main categories, go within days. Deep pages can stay for weeks or months. So the traffic chart doesn't drop on the day of the deploy. It leans, a little more each week, and by the time the shape is clear the deploy that caused it is many releases back.

The fix is to check for the directive at the moment it can arrive, which is after a deploy or a configuration change, rather than after the traffic has gone. A small script in the deploy pipeline does it for a list of important URLs:

#!/usr/bin/env bash
# Fails when any URL in urls.txt carries a noindex in its header or its meta tags.
status=0
while read -r url; do
  [ -z "$url" ] && continue
  response=$(curl -s -i "$url")
  if printf '%s' "$response" | grep -qiE '^x-robots-tag:.*(noindex|none)'; then
    echo "noindex header:   $url"; status=1
  fi
  if printf '%s' "$response" | grep -Eio "<meta[^>]*name=[\"']?(robots|googlebot)[\"']?[^>]*>" | grep -qiE 'noindex|none'; then
    echo "noindex meta tag: $url"; status=1
  fi
done < urls.txt
exit $status

The script only knows the URLs you give it and runs only when the pipeline does. It also flags a tag inside an HTML comment, which no crawler reads; that is the safe mistake to make. A header rule changed in a CDN dashboard, with no deploy at all, gets past it. That is the case a scheduled check covers: the noindex checker reads the header and the tags on every page of the site on every crawl, and sends a critical alert when a page turns noindex.

Is “Excluded by ‘noindex’ tag” an error?

No. It is Search Console telling you what it found. It only needs fixing when the listed pages are ones you want people to find in Google.

How long until a page comes back after I remove the noindex?

Until Google crawls it again and indexes it. Request indexing in URL Inspection speeds this up for single pages. For pages Google rarely visits, its documentation says a revisit can take months, which is why a resubmitted sitemap and Validate fix are worth doing too.

URL Inspection says indexing is allowed, but the report still lists the page. Why?

The report shows what Google saw on its last crawl, and the live test shows the page now, so a fixed page stays in the report until it is recrawled. The other cause is robots.txt: when it blocks the URL, “Indexing allowed?” always says Yes, because Google can't see the page. Check “Crawl allowed?”.

Can I put noindex in robots.txt instead?

No. Google does not support it. A Disallow rule also stops Google from seeing a noindex on the page, so the URL can stay indexed from links. Use the meta tag or the header. See the robots.txt guide.

Does Google read a robots tag that JavaScript adds or removes?

Google renders JavaScript, but its robots meta tag documentation makes no promise about directives changed by scripts. Put the directive you mean in the HTML the server sends, and don't rely on a script to remove one.

Does a noindex also keep the page out of Bing?

The robots meta tag and an X-Robots-Tag header without a user agent apply to every crawler that reads them, Bing included. <meta name="googlebot"> and X-Robots-Tag: googlebot: apply to Google only, which is how a page ends up in Bing and missing from Google.

What is the difference between noindex and nofollow?

noindex keeps the page out of search results. nofollow tells the crawler not to follow the links on the page. none is both at once.