“Blocked by robots.txt”: what it means and how to fix it
Your robots.txt doesn't allow Googlebot to crawl the URL. Here's how to find the rule and when blocking is the wrong tool.
Updated
“Blocked by robots.txt” means a Disallow rule in your robots.txt file stops Googlebot from crawling the URL. A related status, “Indexed, though blocked by robots.txt”, means Google indexed the bare URL anyway (because other pages link to it) without seeing its content.
How Google reads robots.txt
- Googlebot follows the group for
User-agent: Googlebotif one exists — otherwise theUser-agent: *group. - The most specific (longest) matching rule wins; if an Allow and a Disallow are equally long, Allow wins.
*matches any characters and$marks the end of the URL, e.g.Disallow: /*.pdf$.- If robots.txt returns a server error (5xx or 429), Google stops crawling the site for the first 12 hours and then falls back to its last cached copy. A 404 means “no restrictions”.
How to fix it
- Find the matching rule — the indexing checker shows which line blocks the URL.
- Remove or narrow the rule. A common mistake is
Disallow: /left over from a staging site, or a broad rule likeDisallow: /productsthat also hits/products-new/. - Check the change with the robots.txt report in Search Console, then request indexing.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /
Sitemap: https://www.example.com/sitemap.xmlDon't use robots.txt to remove pages
robots.txt controls crawling, not indexing. To keep a page out of Google, allow crawling and use a noindex tag instead — see Excluded by ‘noindex’ tag.
Check your page in seconds
Free, no signup: status code, noindex, robots.txt, canonical and sitemap in one check.
Check your page with the free indexing checker →