“Blocked by robots.txt”: what it means and how to fix it
Your robots.txt file tells Google not to crawl this URL, so Google can't read the page. That's fine for pages you don't want crawled. But robots.txt doesn't keep a page out of search: Google can still index the URL without its content if other pages link to it. To keep a page out of Google, use a noindex tag and let Google crawl it.
- robots.txt controls crawling, not indexing.
- Expected for admin, cart and internal search pages.
- If a real post is blocked, find the Disallow rule that matches it and narrow or remove it.
What does Google say?
This page was blocked by your site's robots.txt file, but it could still be indexed if Google finds information about it elsewhere. To keep it out of search, remove the robots.txt block and use noindex.
Paraphrased from Google's Page indexing report help.
Why does it happen?
- 01A Disallow rule broader than intended, like Disallow: /blog.
- 02A staging robots.txt with Disallow: / shipped to production.
- 03A 'hide from search engines' setting in your CMS or site builder.
- 04Intentional rules for parameter URLs, search results or private areas.
Is “Blocked by robots.txt” a problem?
It depends on which URLs Search Console lists. Check them against these two lists.
Ignore it if
- The URL is an admin page, cart, account area, internal search result or filter URL.
Fix it if
- A post or landing page you want in search is listed.
- You're blocking pages to keep them out of search. Use noindex instead.
How do I fix “Blocked by robots.txt”?
Check any single URL with URL Inspection. After fixing, open the issue in the Page indexing report and click Validate fix; Google says it typically takes up to about two weeks.
# after a real changeInspect URL → Request indexing # in the Page indexing reportOpen the status → Validate fix
- 1
Find the rule
Open yoursite.com/robots.txt and find the Disallow line that matches the URL. Rules match from the start of the path, so Disallow: /blog also blocks /blog-post-ideas.
- 2
Check it in Search Console
Settings → robots.txt shows the file Google last fetched, when, and any errors. URL Inspection tells you whether a specific URL is blocked.
- 3
Narrow or remove the rule
Edit robots.txt so it only blocks what you mean to. If the whole site is blocked after a launch, remove Disallow: /.
- 4
For pages you want out of search, use noindex
Allow crawling and add <meta name="robots" content="noindex">. Google has to crawl the page to see the noindex.
- 5
Validate the fix
In the Page indexing report, open the issue and click Validate fix. Google says validation typically takes up to about two weeks.
How does Blogizi handle “Blocked by robots.txt”?
Blogizi writes your blog's robots.txt for you. It allows crawling and lists your sitemap. If you turn on 'Hide blog from search engines' in project settings, Blogizi adds a noindex tag to every page and keeps crawling allowed, so Google can actually see it.
Index checks for every post are part of Pro, $19/month with a 7-day trial for $0. Your AI agent can read the same statuses over MCP.
Search › IndexingChecked dailyFrequently asked questions
No. It stops crawling. Google can still index a blocked URL, without its content, if other pages link to it. Use noindex to keep pages out of search.
Last reviewed October 23, 2026 against Google's Page indexing report help.
Related Search Console statuses
Excluded by ‘noindex’ tag
The page tells Google not to index it, and Google listened.
Read the guide →Check itDiscovered - currently not indexed
Google knows the URL exists but hasn't crawled it yet.
Read the guide →Usually a problemSoft 404
The page looks like an error or empty page to Google but returns a success code.
Read the guide →Get every post indexed
Publish on your own domain with sitemaps and canonicals handled, and see which posts Google hasn't indexed without opening Search Console.
Ready to get started?
Your agents write. Blogizi publishes. Google sends the traffic to your domain.
