.Google's John Mueller answered an inquiry regarding why Google.com marks web pages that are actually refused coming from creeping through robots.txt and also why the it is actually risk-free to neglect the relevant Browse Console reports about those creeps.Crawler Website Traffic To Concern Specification URLs.The person talking to the concern documented that robots were actually developing web links to non-existent query parameter URLs (? q= xyz) to pages with noindex meta tags that are likewise blocked in robots.txt. What cued the inquiry is actually that Google is creeping the links to those web pages, receiving shut out through robots.txt (without envisioning a noindex robots meta tag) after that getting reported in Google.com Look Console as "Indexed, though blocked out by robots.txt.".The individual inquired the observing concern:." But here's the huge inquiry: why would Google.com index web pages when they can not also find the information? What is actually the conveniences because?".Google's John Mueller affirmed that if they can not creep the webpage they can't find the noindex meta tag. He likewise helps make an interesting reference of the website: hunt driver, recommending to dismiss the outcomes considering that the "ordinary" consumers won't observe those outcomes.He composed:." Yes, you're correct: if we can't crawl the webpage, our company can not observe the noindex. That stated, if our experts can not creep the pages, after that there is actually certainly not a great deal for us to index. Therefore while you might see a few of those pages with a targeted site:- question, the common customer will not view them, so I definitely would not bother it. Noindex is also fine (without robots.txt disallow), it only indicates the Links will definitely find yourself being crept (as well as end up in the Search Console report for crawled/not listed-- neither of these standings induce issues to the remainder of the web site). The important part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the restrictions being used the Site: search advanced search driver for analysis reasons. Among those factors is since it is actually not attached to the routine search index, it is actually a separate trait entirely.Google.com's John Mueller talked about the website hunt operator in 2021:." The brief response is actually that a web site: concern is actually certainly not suggested to become total, nor utilized for diagnostics purposes.A website inquiry is a specific sort of search that limits the end results to a particular internet site. It's essentially merely the word website, a bowel, and afterwards the internet site's domain.This question limits the end results to a particular site. It's certainly not implied to become a comprehensive compilation of all the web pages coming from that site.".2. Noindex tag without using a robots.txt is actually fine for these kinds of scenarios where a robot is actually linking to non-existent pages that are obtaining found by Googlebot.3. URLs along with the noindex tag will produce a "crawled/not listed" entry in Browse Console which those won't have a damaging impact on the remainder of the web site.Check out the question as well as answer on LinkedIn:.Why would Google mark pages when they can't also view the content?Featured Image through Shutterstock/Krakenimages. com.