Skip to content
SiteMedic

“Blocked by robots.txt”: what it means and how to fix it

Your robots.txt doesn't allow Googlebot to crawl the URL. Here's how to find the rule and when blocking is the wrong tool.

Updated

“Blocked by robots.txt” means a Disallow rule in your robots.txt file stops Googlebot from crawling the URL. A related status, “Indexed, though blocked by robots.txt”, means Google indexed the bare URL anyway (because other pages link to it) without seeing its content.

How Google reads robots.txt

  • Googlebot follows the group for User-agent: Googlebot if one exists — otherwise the User-agent: * group.
  • The most specific (longest) matching rule wins; if an Allow and a Disallow are equally long, Allow wins.
  • * matches any characters and $ marks the end of the URL, e.g. Disallow: /*.pdf$.
  • If robots.txt returns a server error (5xx or 429), Google stops crawling the site for the first 12 hours and then falls back to its last cached copy. A 404 means “no restrictions”.

How to fix it

  1. Find the matching rule — the indexing checker shows which line blocks the URL.
  2. Remove or narrow the rule. A common mistake is Disallow: / left over from a staging site, or a broad rule like Disallow: /products that also hits /products-new/.
  3. Check the change with the robots.txt report in Search Console, then request indexing.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Don't use robots.txt to remove pages

robots.txt controls crawling, not indexing. To keep a page out of Google, allow crawling and use a noindex tag instead — see Excluded by ‘noindex’ tag.

Check your page in seconds

Free, no signup: status code, noindex, robots.txt, canonical and sitemap in one check.

Check your page with the free indexing checker →