Tools and Resources for all Your Ecommerce Pain Points
Explore Now

How to Configure Magento Robots.txt

Search Engine Optimization (SEO) is one of the most important aspects that you should follow for making your Magento store a successful one. To get organic traffic on to your store, it must rank within the top Google search results. For that, there are many SEO practices involved and a robots.txt file is one of them. By using this file, you allow search engines to index your store, and you can also disallow pages which you don’t want to index.

Today, I will tell you how you can easily configure Magento Robots.txt file by following some easy steps. In this comprehensive guide, I will also show how to configure Robots.txt in Magento 1.x as well as for Magento 2,x:

Configure Magento Robots.txt

First and foremost, you need to understand how to create the Robots.txt file. To begin with, create a Robots.txt file in the root directory of your Magento store and then add the following declaration in your Magento Robots.txt file to enable robots.txt rules for all crawlers:

User-agent: *

Next, declare your Magento store directories which you do not want to be indexed by Google. Each declaration statement will start with ‘Disallow’:

Now disallow specific clean URLs which you do not want to be indexed by Google. It’s a great way to avoid duplicate content issues. Also, if you want to disallow any specific page URLs of your store indexed, then you can add it here.

The next step for configuring Magento robots.txt requires you to exclude common Magento files to be indexed which are in the root directory:

First disallow your included and structural files by type, such as .js, .css and .php files. And then disallow page URLs, search result URLs, and pager limit URLs that are dynamically generated by Magento:

At last, just reference your sitemap.xml by adding the following line:

Sitemap: http://www.example.com/sitemap.xml

Your final Magento Robots.txt file will be:

The process to configure Magento robots.txt is now complete. You can now move onto the next phase.

Configure Magento 2 Robots.txt

In order to configure Magento 2 robots.txt, you have to first navigate to CONTENT → Configuration from the admin panel of your Magento 2 store:

content-config magento robots.txt

Now click on Edit:

edit magento 2 robots.txt

Then scroll the page and you will see Search Engine Robots section, unfold it:

search engine robots Magento robots.txt

  • Default Robots fields have four options, choose one according to your need:

INDEX, FOLLOW: If you want web crawlers to index your store and also check back later for changes.
NOINDEX, FOLLOW: If you don’t want web crawlers to index Magento 2 store but want it to check back later for changes.
INDEX, NOFOLLOW: If you want web crawlers to index your store once, but don’t want it to check back later for changes.
NOINDEX, NOFOLLOW: If you don’t want web crawlers to index your store and also don’t want it to check back later for changes.

  • Edit custom Instruction of Robots.txt file: This file is used to add custom instructions. I will discuss it later in this tutorial.
  • Reset To Defaults button: Clicking it will remove your custom instructions and restore the default instructions.

Now just click on Save Configuration to configure Magento 2 Robots.txt.

save config magento robots.txt

Custom Instructions for Magento 2 Robots.txt

For Allowing Full Access

For Disallowing Access to All Folders

Default Instructions

To Disallow Duplicate Content:

To Disallow User Account and Checkout Pages

To Disallow CMS Directories:

To Disallow Catalog and Search Pages

To Disallow URL Filter Searches

The process to configure Robots.txt in Magento 2.x is now complete.

Final Words

If you are trying to configure robots.txt file for the very first time, don’t worry. It is easier than it sounds. This is done to help improve the SEO of your Magento store, which will result in boosting sales and conversions.

Read: 27 SEO Professionals Share Their Ecommerce SEO Tips

I hope that after following this tutorial, you have learned how to easily configure Magento Robots.txt and Magento 2 Robots.txt. Still, if you have any confusion or want to discuss anything related to this guide, just use the comment box below!

 

About Author

Syed Muneeb Ul Hasan is an expert in PHP and Magento, he prefers to educate users in implementing and learning Magento. When not working, he loves to watch cricket.