From time to time we re asked to resolve urgent and immediate problems, and one such query came in jus yesterday. This is how the story went.
Why have my online sales dropped?
Let’s take a look at the analytics, we said. And so we did… We took a deep dive and went away from looking at the standard Channels, which are often inaccurate, and instead looked at Source/Medium. We had to compare two different Properties within GA4 because the client had moved away from WordPress, and moved onto Magento2, but to us, the data gave a clear and obvious cause. Both Google and Bing had plummeted.
Why has my Organic traffic dropped?
So now we knew that Organic traffic was the issue, and nothing to do with Paid Ads on Google or Meta, and nothing to do with any email marketing. This lead us to believe that it had to be a site issue. And so we start the next stage of investigations, into Google Search Console and Bing Webmasters. Whilst we left Bing to run a site audit, we looked at the number of indexed pages in Google. We saw an increase in the number of indexed pages, and then a gradual decline over a 3 month period, but nothing significant. However, the non-indexed pages told a different story
Why are my web pages not being indexed?
Looking in more detail at the non-indexed pages, we could see that the majority were “blocked from indexing”. Our first stop was to check the /robots.txt file, and we found this to be fine, with nothing obscure blocking the pages. So we dived a little deeper and looked at an example of a page that was “being blocked”. The front end of the website was showing fine. The home page, when we looked at View Source, was showing as “index,dofollow” which is the instruction to the search engines to please index this page, and please follow any links to discover more URLs.
However, when we looked at a page that was “blocked”, and we viewed the Source Code there, we discovered that the page (and many more pages) were showing “noindex,nofollow”, which is the instruction to the search engines to not index the page and not follow any links. Now we knew the cause of the problem.
How do I fix a Noindex,nofollow problem?
In this instance, we did not build the website, and so we gave clear instructions to the client to ask their web developer to investigate and resolve the issue. We had seen the issue before on a Magento 2 website (this site was also built using Magento 2), and so we knew it was possibly a setting in the back end of the website (most likely) or a legacy setting from when the site was on a staging server, that had carried over when it was made live. Clearly, you do not want the staging server to show and compete within the Search Engine results Pages (SERPs).
For Magento 2, this is what we know:
To fix or implement noindex,nofollow in Magento 2, you can modify the robots.txt file or use the Layout Update XML field in the admin panel for specific pages, categories, or products. You can also set default robots meta tags in the store configuration.
Methods to Implement noindex,nofollow in Magento 2:
- robots.txt – This file controls how search engine crawlers interact with your site. You can add specific disallows for certain URLs or paths to prevent indexing.
- Layout Update XML – For individual pages, categories, or products, navigate to the Design section and use the Layout Update XML field to add the following code:
This code will be inserted into the - Store Configuration – For site-wide settings, you can configure the default robots meta tag in the Magento admin under Stores -> Configuration -> General -> Design -> Search Engine Robots -> Default Robots.
- Event/Observer (for more complex scenarios)- For dynamic pages like product detail pages, you can use an observer to add the noindex,nofollow tag based on specific conditions.
Example using Layout Update XML:
Go to CMS -> Pages (or Catalog -> Categories/Products for categories/products).
Select the page you want to modify.
Go to the Design tab.
Find the Layout Update XML field.
Add the code above to the field.
Save the page.
Checking if noindex,nofollow is implemented:
View the page source code (Ctrl+U or right-click -> Inspect).
Look for the tag within the
If you are having issues with your website, or not receiving as many online sales as you were expecting, please feel free to contact us
