How to add custom robot for blogger

Custom robots.txt is a way for you to instruct the search engine that you don’t want it to crawl certain pages of your blog (“crawl” means that crawlers, like Googlebot, go through your content, and index it so that other people can find it when they search for it). For example, let’s say there are parts of your blog that have information you would rather not promote, either for personal reasons or because it doesn’t represent the general theme of your blog -- this is where you can clarify these restrictions.

However, keep in mind that other sites may have linked to the pages that you’ve decided to restrict. Further, Google may index your page if we discover it by following a link from someone else's site. To display it in search results, Google will need to display a title of some kind and because we won't have access to any of your page content, we will rely on off-page content such as anchor text from other sites. (To truly block a URL from being indexed, you can use meta tags.)

To exclude certain content from being searched, go to Settings | Search Preferences and click Edit next to "Custom robots.txt." Enter the content which you would like web robots to ignore. For example:

User-agent: *

Disallow: /about

You can go to this link to generate it by yourself....

Click on generate Sitemap and you will see this...

The Robots.txt Blogger are very important because they can communicate with the search engines to index what we want and what not and just be good for positioning remember that the more completely you have your blog will have a better position clear that the matter content and a lot ... but let's start.What is a Robot.txt?The robots.txt are simple lines of code with which we indicate to our content crawlers as to index and how it should crawl to an improvement in searches.

For example, many times we have some pages that do not want to be indexed by the search engines either because that page does not have much content or for the simple fact that we do not want to be indexed such as labels on our blog or website Welcome or any other product that is not as important for indexing.Here's something to note is that the first thing that crawlers revised when going through your site are the robots.txt then you will see the importance of them.Include the lines of code in robots.txt is important even if you do not want to even use them for non-indexing of any content especially since the simple fact of robots.txt include code in our blog, we will giving the signal to search engines that are free to investigate and index our content.The robots.txt my blog is this: So you can get your own code you have to put robots.txt in your browser the address of your blog more this: /robots.txt

For Example:

https://khmerueteams.blogspot.com/robots.txt

and your code should be found as an example below:

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://khmerueteams.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Just as is the code we are telling us spiders or robots that can index and search all our websites and content.

This code lines is best left as it is, if you know it does not use, auque robots.txt desacernos can help us duplicate content, there is a more effective way to do this as it is the goal of "rel canonical "that later will talk about it.
It is best to copy and paste your robots.txt lines as shown and then you'll see what duplicate content.

Share on Google Plus

About KhmerueTeams Official

Welcome to KhmerueTeams Officials Blogger. I am Mr. Chhuye The Admin of this blogger. If you have quetion contact me : +855968555664 / +855976235033 or watch my video on ​

YOUTUBE HERE