How To Add Custom Robot.txt In Blogger For Seo

Whats Robots.txt?

When you create a blog, a robot txt file is generated by blogger automatically to inform Robots(crawlers) sent by search engines about your blog and to inform to crawl your blog pages or no.
You would like search engines to index some pages of your blog and some you would rather not to be indexed e.g any demo pages, category page or other. 

How Robots and Crawlers see your blog's Robot text?

Robots.txt is first thing found by crawlers when they find your blog. Your blog's Robots.txt is like a guide which tell crawlers and spiders what to index and what not. When you disallow any page then it crawlers and spiders will not index that page.

How to find your blog's Robots text page?

Initially your blog Robots.txt page in empty because blogger doesn't add any Robots.txt. You can view your blog's Robots text by replacing Yourblog in this link: http://www.yourblog.blogspot.com/robots.txt 

How to find Robots.txt in your blog:

You can find Robots.txt simply. Go to blogger login to your account, go to settings ---> Search preferences. If you can't find Robots.txt then See the below image:

How does Robots.txt look?

If you have not customized your blog's Robots.txt, It should look like below:

User-agent: Mediapartners-GoogleDisallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://Yourblog.blogspot.com/sitemap.xml

Where:

User-agent: Mediapartners-Google is for those who use Google Adsense if you don't use Google Adsense remove it. If you allow this option Robots and Crawlers will be able to see ads on specified pages and if not allowed they will not be able to see ads on specified pages.

User-agent:* specify which potion is for Robots, Crawlers etc.

Disallow: /search is to guide robots and crawler not to index the links which contain /search for example: http://101helper.blogspot.com/search/label/Tips%20%26%20tricks , Robots and Crawlers won't index this link and other like this as it contains /search.

Allow means those thing which Robots can do, adding / means you allow Robots to crawl on main page(homepage) of your blog.

Sitemap refers to the sitemap of your blog it means Robots can index every new post of your blog.


How to Disallow pages or posts which you don't want to
be indexed:


To disallow robots to index a specific page add the link of that page in front of Disallow: but is you want to disallow robots to index a specific post then add /year/month/post-link add year and moth of the post in which it is published with the link of that post.

How to Add custom Robot.txt to your blog?

Follow the below steps to add custom robot text to your blog:

(1). Go to blogger and log-in to your account.

(2). Go to Settings ---> Search preferences and find crawlers and indexing.

(3). Click edit and check Yes in order to edit it.

(4). Add the below code after replacing Yourblog. Make other changes if you want such as disallowing any page or post to be indexed or any thing else.

User-agent: Mediapartners-GoogleDisallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://Yourblog.blogspot.com/sitemap.xml

(5). After following step 4 click save and you are done!

You can check your blog's Robots.txt by this link http://yourblog.blogspot.com/robots.txt.

You may like: How To Use Robots Text For Better Seo In Blogger

Thanks for visiting 101Helper. I hope this was helpful if you have any problem feel free to ask me. Follow for new posts. Share with others.

Tags:  How to add custom Robot.txt in blogger,101Helper,boost search engine index rate blogspot,boost blog traffic, show posts in search results blogger,seo optimization guide,Show blog post on google,bing,yandex,yahoo