Targeting search engines using robots.txt file in blogger would be helpful. But its complicated to use robots.txt file and custom robots header tags in blogger. Search engines crawl your site using this robots.txt file. So not using it effectively, makes your blog not listed in search engines. Robots.txt( also known as Wanderers, Crawlers, or Spider) tell webmaster what to crawl in your site. Setting can be added to stop tips to market blog online from search engines.
- Login into Blogger
- Select your blog if you have many
- Then migrate to SETTINGS –> Click on “Search Preferences“
- In Crawlers and Indexing, for custom robots.txt, click on Edit and check yes.
Then copy the below code and paste it.Click on Save Changes
Allow: /Sitemap: http://www.yourdmomain.com/feeds/posts/default?orderby=updated
Explanation to robots.txt file
User-agent : Medidapartners-Google : This part is about adsense. If you are not using adsense in your blog then just remove it. If you use adsense then no worries.User-agent : User agent calls search engines to crawl his/her site. * indicates to include all search engines like google, yahoo and bing ect.
Disallow: /search?q=* : Telling search engines not to crawl search pages.
Disallow: /*?updated-max=* : Telling search engines not to index or crawl labels and navigation pages.
Allow: / : Allowing search engines to index your homepage
Sitemap : Must use to index your ever new post. Tells search engines to update when, every new post comes.
How to add a page link to disallow search engines to crawl?
The below is sample code to disallowing to crawl privacy page from search engines. Remember one line per one command. While adding a code make sure you have excluded your domain. Ex: www.yourdomain.com
Hope our article helped you, if you have any problem on this issue, comment below.