Robots.txt file

What or why Robots.txt file and how to add it to blog?

There are many bloggers who do not understand Robots.txt option well or never bother about this option. They feel that this option is of no use so leave it blank. Actually this option plays a very important role for search engine optimization. What or why Robots.txt file and how to add it to blog? 

 

 

 

 

 

And if you continue blogging without doing search engine optimization, chances of getting enough visitors to your blog will decrease. However, while activating this option, if you unknowingly turn on the other Robots.txt file copy option, then it may not be good. For this reason you need to know more about it before activating it.

 

 

 

 

 

 

 

Robots.txt file:

Every search engine has its own web robots. You might be wondering if this is like Rajinikanth’s Hindi movie Robot. Actually it is nothing of the kind. This is a type of web function, called a robot, for checking all types of websites that search engines have. And through the Robots.txt file, those robots are instructed whether to crawl and index your blog/website. You can use this Robot.txt file to allow robots to crawl and index or not. Again, if you want, you can allow some posts to be crawled and indexed, and some posts may not be allowed to be crawled and indexed.

 

 

 

 

 

 

How does this Robots.txt file work?

The Robots.txt file is like a flight announcer at an airport. The way he tells passengers to board the plane on time when it’s time to fly, the Robots.txt file tells search engines to index new blog posts when it’s time to crawl. As a result, your newly posted articles are easily found in search engines.

 

 

 

 

 

 

 

Most blogs have Robots.txt files like this. Maybe you have used it on your blog without even realizing it or still doing it. I want you to fully understand about this and then add it to the blog. I will try to explain it separately by dividing it into two parts. First I will discuss these parts and then the symbols they contain.

 

 

 

 

 

User-agent:

Mediapartners-Google: First of all, robots are indicated through User-agent. Here Mediapartners-Google is a robot of Google Adsense. If you are using Google Adsense on your blog then it needs to be added. If you disallow this option then Adsense robot will not get any information about your blog ads. You are not using Google Adsense, then delete these two red colored lines.

 

 

 

 

User-agent:* refers to all types of robots. When you use the * symbol after user-agent, it means that you are directing all types of robots.

 

 

 

 

 

 

 

 

Disallow:

search with this command to disallow the keyword. That is, your blog’s search links are being asked not to be crawled and indexed. For example, if you look at the Label links of your blog, you will see that there is this word Search before the link of each Label. For this the robot is instructed not to crawl the Label links. Because Label links do not need to be indexed in search engines.

 

 

 

 

 

 

Allow:

This allows the keyword Allow. This ‘/’ symbol means that robots will crawl and index your blog’s home page. For example, after submitting your site to Google Webmaster Tools, you will see that Google Webmaster Tools is always indexing one post more than your post. Actually not much, it is also counting your home page.

 

 

 

 

 

Sitemap:

This will tell the robots to index new posts when you make new posts. Every default blogger has a sitemap. But by default it does not index more than 25 posts. For this, this Sitemap link needs to be added to the Robots.txt file and submitted to Google Webmaster Tools.

 

 

 

/

Name description

User-agent This refers to robots.
Disallow means that no robot will crawl the directory after it. Note: If no directory is given after Disallow then Disallow will work as Allow.
* This symbol refers to all types of robots.
Allow means that the robot will crawl the directory after that.

 

 

 

 

Name description

User-agent This refers to robots.
Disallow means that no robot will crawl the directory after it. Note: If no directory is given after Disallow then Disallow will work as Allow.
* This symbol refers to all types of robots.
Allow means that the robot will crawl the directory after that.

 

 

 

 

 

Name description

User-agent This refers to robots.
Disallow means that no robot will crawl the directory after it. Note: If no directory is given after Disallow then Disallow will work as Allow.
* This symbol refers to all types of robots.
Allow means that the robot will crawl the directory after that.

 

 

 

 

 

How to add to the blog?

From the Blogger dashboard click on Settings > Search Preference.
Then click on the Edit button of Custom Robots.txt.

 

 

 

After that, if you click on Yes of Enable custom robots.txt content, you will see a box. Copy and paste the above codes in this box.
Then click Save to exit. That’s all

Leave a Comment

Your email address will not be published. Required fields are marked *