Today we will be talking about one of the biggest SEO mistakes a website owner (or web developer) can make: noindex. Just the mention of it can send shivers down a developer’s spine.
29/08/2018 Update: Please see the update notes from the end of the post.
What is the <meta name="robots" content="" /> Tag?
To put it in simple terms, this meta tag tells search engines what actions they can take (or not take) on a certain page. Major search engines will abide by the commands included in this tag.
This meta tag can be included anywhere between the <head> and </head> tags in a page’s header, like seen below:
IMPORTANT: This tag does not have a site-wide effect. It can contain different values on different pages of the same website.
Available Values for the META ROBOTS tag
Here’s a list of valid values for the META ROBOTS tag.
- Index (default value)
- Noindex
- None
- Follow
- Nofollow
- Noarchive
- Nosnippet
- Noodp (no longer relevant)
- Noydir (no longer relevant)
These values can be combined, so for example the variations below are all perfectly valid meta robots tags:
- <meta name="robots" content="index,follow" />
- <meta name="robots" content="index,follow,noarchive" />
- <meta name="robots" content="noindex" />
The Effect of NOINDEX,NOFOLLOW
The NOINDEX value tells search engines NOT to index this page, so basically this page should not show up in search results.
The NOFOLLOW value tells search engines NOT to follow (discover) the pages that are LINKED TO on this page.
Sometimes developers will add the NOINDEX,NOFOLLOW meta robots tag on development websites, so that search engines don’t accidentally start sending traffic to a website that is still under construction.
Or you might have your current (live) website on www.example.com, but you also keep a development copy on www.dev.example.com/. In this case it is advisable to noindex,nofollow the Dev version, so as to avoid many potential issues.
What often happens is that people will accidentally add this tag to live websites, forget to add it to development copies, or worse of all: forget to remove it from live websites after going live.
Yes, the same results and issues can arise from a poor robots.txt file in the root of a website, but that’s beyond the topic of this post.
~3% Of Hotel Websites Are Affected
According to an independent analysis by HermesThemes.com of 50,000+ hotel websites from around the world, it was determined that ~3% of websites block search engines from indexing their website.
This was a shocking discovery, which prompted us to address the issue with this article.
How To Verify If My Website Contains This Error?
Fortunately there is a very simple way to check any website/page for this error.
Simply open up the page in your browser, right click somewhere on the page (but not on links or images) and choose “View Page Source”. In most browsers on Windows you can simply press CTRL+U on your keyboard.
Doing this will open a new tab with the complete HTML code (as the browser sees it) for the current page. As mentioned earlier, the meta tags are usually at the top of the website, like in this example:
If you see a META ROBOTS line on this page with the NOINDEX or NONE value then you need to take action immediately!
How Do Affected Websites Look Like in Search Results?
I’m glad you (hopefully) asked.
There’s a very handy way to search Google for indexed pages from a certain domain name: [site:example.com] (without the square brackets).
So we go to Google and search the domain that is using the NOINDEX meta robots on their website, and here’s what we get:
I hope that you understand how much damage can be caused by having your website completely removed from Google and other search engines. Your organic search traffic will drop to zero in a matter of days.
How to Fix/Remove the Meta Robots Line?
Thankfully fixing this issue is easy and should not be delayed. First you need to locate where this line comes from.
In WordPress, the first thing you should do is go to Dashboard > Settings > Reading.
Make sure that the checkbox for Search Engine Visibility is unchecked.
If this didn’t fix the issue, then you should check if this line is hard-coded in the theme.
To check this, you should go to Appearance > Editor and then choose “Theme Header header.php” from the list of files on the right (valid for most themes).
Look through this file and make sure that there is no META ROBOTS tag with a harmful value. If there is – remove it and click the blue “Update File” button.
In Conclusion
This line of code can cause a great deal of headache, loss of revenue and negative long-term SEO impact.
Is your website affected by NOINDEX? Check today!
We also have a detailed meta robots guide available, check it out.
Updates from 29/08/2018:
I wanted to take the time and mention the new figures from 29th of August 2018.
The number of websites that I analyze has increased dramatically. The original article referred to the data extracted from 50,000 hotel websites. Now I analyze ~875,000 unique hotel websites (unique domains).
The findings from these 875,000+ hotel websites shows that 1.502% of hotel websites use NOINDEX or NONE as their meta robots value.
The percentage doesn’t seem high, but that’s over 13,000 of hotel websites that effectively block all search engine bots from indexing their websites.
Granted, some of them are doing this while performing maintenance on their websites. Other websites are actually trying to keep a low profile and display their business only via direct linking.
But I’ve taken the time to manually visit 200+ of random websites from this list and I would quickly estimate that 3/4 of them are actually using NOINDEX by mistake. So there’s that.
1 Comment
I’ve seen it a few times where people have complained that their website is not appearing in search engines – only to find that they have the ‘discourage search engines’ ticked. Sometimes a company will have a new website developed and pay little care to this setting as well as not respecting their old permalinks and failing to setup their redirects. To me these are some of the most important things to get right. You can have the best looking website, but if you ignore these basic settings you’ll lose all visibility.