Blocked by robots.txt, and indexed though blocked

In Search Console, open Indexing, then Pages, and scroll to the table called Why pages aren't indexed. A row reading Blocked by robots.txt means Googlebot found the URL, checked your robots.txt before fetching it, and found a rule that disallowed it. Google never read the page. Google's help page for the report names the same reason URL blocked by robots.txt, so you will see both wordings.

Indexed, though blocked by robots.txt sits in a different table, Improve page experience, because the page is in the index. Google was not allowed to fetch it and indexed the URL anyway. Google's help page says it plainly: being blocked by robots.txt "does not guarantee that the page won't be indexed through some other means." Links from other pages, or from your own sitemap, are usually how it learns the URL exists.

Neither status is an error by itself. Many sites block their cart, account pages and internal search results on purpose, and those URLs show up in the Blocked by robots.txt row for as long as the rule stays. The report is worth reading for one thing: a URL in that list that you want people to find.

A third message comes from the same file. If your sitemap lists URLs that robots.txt blocks, the Sitemaps report can show Sitemap contains urls which are blocked by robots.txt. A sitemap is a list of pages you want crawled, so a blocked URL in it is a contradiction. Either the rule or the sitemap is wrong. The sitemap guide covers that report.

1. 01

   Take one URL from the report

   Click the Blocked by robots.txt row and copy an example URL that should be in search. In URL Inspection, the Crawl allowed? line tells you whether robots.txt still blocks it. Use Test live URL: the default view is Google's last crawl, which can be weeks old.

2. 02

   Fetch the file the way a crawler does

   Request robots.txt from the same host and protocol as the blocked URL. A robots.txt only applies to its own host, so https://www.example.com/robots.txt says nothing about https://example.com/ or https://shop.example.com/.

3. 03

   Check what status code it answers with

   A 200 means the rules in it apply. A 404 means there are no rules. A 5xx means something else entirely, covered further down.

4. 04

   Find the group Googlebot reads

   Look for a User-agent: Googlebot line. If there is one, that group is the only one Googlebot follows. If there is not, it follows the User-agent: * group.

5. 05

   Find the longest rule in that group that matches the path

   Rules match from the start of the path, so Disallow: /p matches /products/blue-shoes and /pricing. The longest matching rule decides. The tester below does this for you and names the line.

If the file you fetch has no rule that matches the URL, and URL Inspection's live test says crawling is allowed, the block is already gone. The report shows what Google saw on its last crawl, and it catches up when Google crawls the page again.

First, the group

A robots.txt file is split into groups. Each group starts with one or more User-agent lines and holds the Allow and Disallow rules under them. Google's crawlers read the group with the most specific user agent that matches their own name and ignore every other group. In Google's words: "User agent specific groups and global groups (*) are not combined."

This catches people out more than anything else in the file. A User-agent: Googlebot group added to block one folder from Google also stops Googlebot reading every rule in the * group. The drawing at the top of this page shows the result: /cart is blocked for every crawler except the one you care about most. If you add a Googlebot group, copy into it every * rule you still want Google to follow.

When the same user agent has more than one group in the file, Google combines those groups into one. Consecutive User-agent lines share the rules under them, so one group can serve Googlebot and Bingbot at once.

Then, the rule

Inside the group, Google compares the rules whose path matches the start of the URL path, query string included, and picks the one with the longest path. When an Allow and a Disallow match with paths of the same length, Google uses the less restrictive one, which is the Allow.

Which file applies to which URL

Google's rules apply "only to the host, protocol, and port number where the robots.txt file is hosted." Each subdomain needs its own file, and so does each protocol. Crawlers only look for the file at the root of the host, never in a folder.

Google reads up to 500 kibibytes of the file and ignores anything after that. Field names are case-insensitive, and Google recognises four of them: user-agent, allow, disallow and sitemap. A Sitemap line has to be a full URL, with the protocol and host, and it applies to the whole file whatever group it sits in.

Google keeps a copy of robots.txt for up to 24 hours, and longer if it cannot fetch a fresh one. A Cache-Control: max-age header on the response can change how long. So a fix you deploy now can take up to a day to reach Google, and a mistake can last as long.

1. 01

   Change the rule where it is made

   Edit the file, the template, the plugin setting or the code that writes robots.txt. On many sites robots.txt is generated, and a hand edit on the server is overwritten by the next deploy. Narrow the rule rather than deleting the file: Disallow: /p/ instead of Disallow: /p.

2. 02

   Fetch the live file again

   Run the curl command from above after the deploy. A CDN or page cache can keep serving the old copy, so purge it if the change is not there.

3. 03

   Test the URLs that were blocked

   Paste the live file into the tester and try a few of the URLs from the report as Googlebot. Try a URL that should stay blocked too, to make sure the narrower rule still does its job.

4. 04

   Let Google know the file changed

   Search Console's robots.txt report, under Settings, shows the copy Google last fetched and when, and lets you ask for the file to be fetched again. Otherwise Google picks up the change within about a day.

5. 05

   Ask for the pages to be crawled

   For a handful of important pages, inspect each URL and click Request indexing. There is a daily limit per property. For the rest, open the Blocked by robots.txt row and click Validate fix. Google says validation usually takes up to about two weeks, and sometimes longer.

Pages that were blocked for a long time are not ranked again the moment they are unblocked. Google has to crawl each one, and it reaches the less important pages later. Expect the Blocked by robots.txt count to fall over the following weeks as Google gets to each page.

SEOtto fetches robots.txt at the start of every crawl, keeps every version, and sends a critical alert when a change blocks the whole site or blocks Googlebot. That covers the next time this happens; the steps above cover this one.

Google is explicit about the overlap. Its Page indexing help says a robots.txt rule "is not recommended for blocking a page, and will actually prevent noindex from being seen by Google." The URL Inspection help adds that for a blocked page, Indexing allowed? always reads Yes, because Google cannot see a noindex it was not allowed to fetch. The noindex guide covers setting one and checking it took.

Removing a page that is indexed, though blocked

1. 01

   Allow crawling of the URL

   Remove or narrow the Disallow rule that matches it, or add a longer Allow rule for that path.

2. 02

   Add a noindex to the page

   <meta name="robots" content="noindex"> in the HTML, or an X-Robots-Tag: noindex header for files such as PDFs.

3. 03

   Wait for the recrawl, or ask for it

   Request indexing in URL Inspection for the URLs that matter. Each one drops out once Google has crawled it and seen the noindex.

4. 04

   Leave it crawlable

   If you block it again, Google stops seeing the noindex, and links to the URL can put it back in the index.

When it is fine to leave

If the URLs listed under Indexed, though blocked by robots.txt are parameter variants, cart links or other pages nobody searches for, leaving them is a reasonable choice. Read the list for a page that should be in search and is showing there without its content. That is the case worth fixing.

Lines Google ignores

How a whole site ends up blocked

Disallow: / under User-agent: * blocks every crawler without a group of its own from every page. It rarely arrives on purpose. The usual routes are ordinary deploy mistakes:

1. 01

   A staging file deployed to production

   Staging sites often block everything, correctly. When robots.txt is a file in the repository, the staging version can be the one that ships.

2. 02

   Rules chosen by environment

   Code that writes Disallow: / unless it knows it is running in production falls back to blocking when the setting it checks is missing on a new server.

3. 03

   A layer in front of the site

   A CDN, host or security tool that serves its own robots.txt when a rule changes, or when the origin is unreachable.

Changes

Does Disallow in robots.txt remove a page from Google?

No. It stops Google crawling the page. A page already in the index can stay there, and a new URL can be indexed from links to it. To remove a page, allow crawling and add a noindex. See the noindex guide.

How long does Google take to notice a robots.txt change?

Google caches robots.txt for up to 24 hours, so the new rules usually apply within a day. Pages that were blocked are then crawled again over the following days and weeks, the important ones first.

Why does Search Console still show pages as blocked after I fixed robots.txt?

The report shows what Google saw on its last crawl of each URL. It updates as the pages are crawled again. Click Validate fix on the row to have Google check the listed URLs; Google says that usually takes up to about two weeks.

Should I fix Indexed, though blocked by robots.txt?

Only if the listed URLs matter. If one should be in search, allow crawling so Google can read it. If one should be out of search, allow crawling and add a noindex. If they are cart links or filter variants nobody searches for, you can leave them.

Is it a problem if my site has no robots.txt?

No. When robots.txt answers 404, Google treats every URL as allowed. A robots.txt that answers 5xx or times out is the one to worry about, because Google stops crawling the site until it gets an answer.

Does Google support Crawl-delay?

No. Google's robots.txt documentation lists four supported fields: user-agent, allow, disallow and sitemap. Other fields, crawl-delay and noindex included, are ignored by Google.

Does the robots.txt on www.example.com apply to example.com?

No. A robots.txt applies only to the host, protocol and port it is served from. https://example.com/robots.txt, https://www.example.com/robots.txt and each subdomain's file are separate, and each needs the rules you intend.