.Google.com's John Mueller answered an inquiry concerning why Google.com indexes webpages that are actually disallowed from crawling through robots.txt and why the it is actually safe to overlook the similar Search Console reports about those crawls.Bot Web Traffic To Inquiry Criterion URLs.The individual talking to the question recorded that robots were actually developing links to non-existent inquiry guideline URLs (? q= xyz) to web pages along with noindex meta tags that are also blocked in robots.txt. What motivated the question is actually that Google.com is actually creeping the links to those web pages, obtaining obstructed through robots.txt (without envisioning a noindex robotics meta tag) at that point obtaining shown up in Google Explore Console as "Indexed, though shut out through robots.txt.".The person asked the complying with question:." However below is actually the large concern: why will Google mark web pages when they can not even observe the material? What's the perk during that?".Google's John Mueller validated that if they can not creep the page they can't view the noindex meta tag. He also creates a fascinating acknowledgment of the website: search operator, encouraging to dismiss the end results since the "common" users will not observe those outcomes.He wrote:." Yes, you are actually right: if our company can not crawl the page, we can not see the noindex. That mentioned, if we can not crawl the pages, at that point there is actually certainly not a lot for our company to index. Thus while you might see a few of those webpages with a targeted internet site:- concern, the typical user won't observe all of them, so I would not bother it. Noindex is likewise great (without robots.txt disallow), it simply indicates the Links will end up being crept (and also wind up in the Browse Console record for crawled/not catalogued-- neither of these statuses induce problems to the rest of the site). The vital part is that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the limits in operation the Site: search progressed search operator for diagnostic reasons. One of those factors is actually given that it's certainly not attached to the normal hunt index, it's a separate point completely.Google.com's John Mueller talked about the website search operator in 2021:." The quick solution is that a site: query is actually certainly not meant to become comprehensive, nor made use of for diagnostics reasons.A web site inquiry is actually a details kind of hunt that restricts the results to a certain site. It is actually essentially only words website, a bowel, and then the website's domain name.This query confines the end results to a details site. It is actually not meant to be a detailed assortment of all the web pages from that site.".2. Noindex tag without making use of a robots.txt is alright for these sort of circumstances where a bot is connecting to non-existent web pages that are obtaining found by Googlebot.3. URLs along with the noindex tag will generate a "crawled/not catalogued" entry in Search Console which those will not possess a negative result on the remainder of the web site.Read through the inquiry and respond to on LinkedIn:.Why will Google mark webpages when they can't even observe the material?Included Picture by Shutterstock/Krakenimages. com.