Home Blogging How Robots.txt file helps Search Engine Bots to crawl your Blog

How Robots.txt file helps Search Engine Bots to crawl your Blog

by editor
Google robots

Doing SEO for Blog/Website is not so easy task, we need to always wonder for new term and search engines algorithm to find and define new SEO strategies.

“Robots.txt” file customization is also cover under SEO process, this file helps to crawl a website for search engine’s bots. As this file is customizable, so admin can define what are the parts of the site can crawl or not.

When I was started to search custom setup parameters for robots.txt file I found so many questions which are asked by bloggers on the web for sites SEO and robots.txt file importance for SEO process.

You will see problem questions like:

  • Why my blog is not crawled?
  • Why crawling process is slow for my blog?
  • Why my admin folder is also indexed by Google?
  • No-index is not working on Robots.txt file.

Above mentions that entire problem you can solve by using the robot.txt file.

Now, you must be wondering to know more about Robots.txt file, where it should be, how to create and how can we use for search engine optimization?

I’ll explain everything one by one in this article, and after reading this, you can create and use this file for your website.

web Robots

What is the Robots.txt file?

A Robots.txt is simple text file located in hosting root directory of your website and it indicates those parts of your website you don’t want accessed by any search engine crawlers. This file uses a protocol which is set of commands to indicate access to your site by search engines web crawlers called Robots Exclusion Standard.

You can get more details about Robots.txt file is here.

How to create a robots.txt file?

To create a robots.txt file, you need access to the root directory of your hosting server. If any problems for accessing or not sure about your access permissions, you can contact your web hosting service providers.

Generally, robots.txt file uses two keywords. User-agent and Disallow. User-agents are referring for search engine bots or web crawler applications. To use this keyword value you need to check those value should list in the Web Robots Database. Disallow keyword refer a command for the user-agent that gives instruction to not to access a particular URL.

If you want to give instruction to user-agent to access another directory which is a subdirectory of the disallowing directory then you need to use the third keyword “Allow”.

After creation and saved in your website root directory you can check your robots.txt file at browser by using this URL http://<your domain name>/robots.txt.

You can get more ideas on Robots.txt file creation and commands here.

How to test your Robots.txt file?

This is a very important step of robots.txt file creation. Your blog/website get properly indexed in search engines and gives a better result in google search by using this file. So, be sure created robots.txt file has all entries as per the standard and instruction.

Google provides a tool “robots.txt” file tester, you can use this tool to identify all entries of your created file and correct if there is any problem.

You can get better idea to test your robots.txt file here.

How to submit your Robots.txt file to Google?

There are two options to submit your new robots.txt file to google.

One is by using “submit” function of the robots.txt tester tool allow you to put the file in place and ask google to crawl quickly and index new robots.txt file for your site.

Another one is to put your robots.txt file manually at the root directory of your website and use the “verify live version” option to see your live robots.txt  file.

You can check the google documentation for better understanding by following this link.

Feel free to leave your own opinion, and any extra tips and ideas are always much appreciated.

Don’t forget to subscribe to GUTANN email newsletter to keep updating yourself with such posts in future.

Dear learners if you need any help for your blog, you can get the suggestion by commenting below or contact me by using the contact form.

You may also like


Mabel January 15, 2016 - 3:41 am

Usually I don’t learn article on blogs, but I would like to say that this write-up very
pressured me to try and do it! Your writing style has been surprised me.
Thank you, quite nice post.

Satish Kumar January 15, 2016 - 12:45 pm

Thanks, Mabel……:-)


Leave a Comment