Instead of a 404 message for pages not found, some websites serve a regular page with custom messaging, or even search results included.
The robots.txt file is accessible via the https://mywebsite/robots.txt URL of your website and contains a set of rules for search engines that describe which pages should be included or excluded from indexing.
However, there are many other factors affecting Magento SEO, as well. In a previous post, we made the case for customer experience as the most crucial driver of optimal website indexing.
Right, we’ve also heard this story many times. Site performanсe is a critical component of search engine friendliness.
The “robots” meta tag is the part of a page’s source code that tells search engines what pages should be indexable, and what pages should be excluded from the index. There are several possible values for this tag.
Search engines might apply penalties in such a case because of duplicated content. In this example, because the URLs are different while the page is the same, you would need to ensure that both pages have the identical canonical link – in this case, https://mywebsite/product, which points to the original page.
There are many technical aspects you’ll need to pay attention to in order to optimize Magento SEO. This article provides a Magento SEO checklist of tasks you can perform that will make search engine robots love your website. Having an incorrect record among these rules might disable the indexing of the entire website. Therefore, you should ensure the correct set of rules (and path to the sitemap) are configured in the file. More information about robots.txt in Magento can be retrieved from the official documentation.
To edit titles and meta descriptions in Magento, you can utilize the built-in Magento functionality for CMS pages, categories, products, etc., where you’ll find corresponding fields on the entity edit form in the admin panel. For example, the same product details page could be accessible by two separate URLs, one with a category in the path, and one without:
Your sitemap is the list of all website pages that you want search engines to index. Refer to the official documentation to use Magento’s built-in functionality to generate a sitemap.
Trailing Slash Redirects
According to search engines, https://mywebsite/product and https://mywebsite/product/ are different URLs, so the aforementioned penalty for duplicated content would also be applicable here unless you ensure that you configure a redirect from https://mywebsite/product/ to https://mywebsite/product (or vice versa), rather than showing the same page for both URLs.
For some of these items, you may eventually need the technical assistance of a Magento developer. However, for others, you can simply use the Magento admin panel, or a 3rd-party plugin to help automate the process. If we’ve left anything out, let us know in the comments. If you Magento setup require a more deeper SEO improvements, consider ordering Magento SEO Audit from Atwix.
Two of the primary items a search engine checks on your site are the page title and meta title. The meta title is a short description of the content on a page The page title entity is self-explanatory.
Page Title and Meta Tags
Here’s another example:
Have you ever heard someone say, “I want to have a lightning-fast website because I need to have a good indexing among search engines!”?
You can also exclude pages using the robots.txt file mentioned above. If so, a rule in the robots.txt file has priority. For example, if a page is excluded in the robots.txt, the record in the “robots” meta tag will have no effect (ie, it’s ignored).
Well, for example, it’s always a good practice to exclude login, checkout, cart, and other non-content pages from the search index, simply because it’s typically not beneficial to show them in the search result. For example, NOINDEX, FOLLOW is the ideal configuration for the cart page.
Soft 404 Pages
So, there you have it – our shortlist of the most crucial components affecting your website’s SEO.
The checklist is simple but powerful. While it won’t cover all the technical details (which would require a book instead of a post), we hope it helps you better understand Magento technical SEO.
While it’s important to keep both the title and meta descriptions simple, it is important that every page of your website has them. . It’s also highly recommended to ensure that each title and meta description is unique for every page of your website.
Sitemap and robots.txt are Present and Configured Correctly
It’s important to note that if you don’t specify the “robots” meta tag for a page, it will automatically be tagged as INDEX, FOLLOW.
The more customers that use your website, the better ranking you will achieve. So, performance is far from your only consideration when discussing search engine optimization.
Nowadays, Google and other search engines consider mobile customer experience first – before desktop experience – when ranking your website. Therefore, it is critical that you ensure pages aren’t broken for mobile resolutions.
Robots Meta Tag
Another common issue arises when different domains point to the same page.
- NOINDEX, NOFOLLOW – do not index the page, do not follow the links on the page (do not allow search bots to go further through the links found on this page)
- NOINDEX, FOLLOW – do not index the page, but follow the links on the page
- INDEX, NOFOLLOW – index the content of the page, but do not follow the links
- INDEX, FOLLOW (most common) – index the page and follow the links
Google and other search engines utilize sophisticated algorithms to identify this kind of SEO “trick.” Therefore, you should never try to hide 404 links. Rather, if a page is organically no longer available, then the 404 status must be returned.
So, why do you need a “robots” tag, at all?
These are different URLs that might point to the same page, as the same rule with the canonical link is applicable for this case.
Refer to the official Adobe documentation for more details on how canonical links work, and how to configure them properly in Magento.
Canonical links exist in the source code of every webpage and point to the original URL of the content. Sound complex? Let’s elaborate on that. In other words, you can have different URLs pointing to a single page.
These SEO practitioners are effectively attempting to hide the 404 page-not-found. However, this type of content is not highly regarded by search engines. In fact, quite the opposite.
- If your category pages are indexable, make sure they contain, not only a list of products, but additional textual content that describes the category.
- If the search results page on your website is indexable, this may not be the most optimal solution (however, there are exceptions).
- Attempt to utilize only one H1 tag per page. And keep the hierarchy of the H tags intact (eg, H1 > H2 > H3 …)
- Avoid pages with a low text ratio. There are tools available that check you HTML-to-text ratio. Or more simply, avoid pages with just a few sentences if the page is indexable by search engines.
- Check your website using SEO tools (including Google PageSpeed Insights and google search console) regularly to find and address issues in a timely fasion.
- Use additional rich snippets. Rich snippets are metadata recognizable by search engines. Search engines may show additional information on the search result page by reading metadata. Common examples include ratings, prices, product availability, etc. Magento provides many rich snippets out-of-the-box, and you can also use 3rd-party extensions to extend the data. Feel free to check out our Rich Snippets extension that adds additional metadata to your website, while improving the look and feel of the search results.
When you perform a search in Google, you see the title and meta description of each page returned in the search result.
For example, https://mywebsite/product and https://www.mywebsite/product have the same final destination, which is another case of duplicated content. Therefore you would need to manage redirects from one domain to another, similar to the previous examples.
There are several recommendations that are less critical or have a content-related nature. It’s worth mentioning them here, too.