No products in the cart.
Robots.txt is a simple but very important file of the page which controls crawling access of the bots or crawlers.
In this Blogger tutorial, I will explain the robots.txt file in detail like what is robots.txt file, how to generate or create it, and then how to submit it. After this tutorial, you don’t need any robots.txt generator for generating your robots.txt file for your blogger website.
So first we need to know what is robots.txt file?
What is a Robots.txt file?
Robots.txt is a simple text file (.txt) that is located in the root directory of your website and it tells search engine crawlers or bots which page or file you want (or don’t want) to crawl.
You can check your website robots.txt file just by adding the /robots.txt slug after your website URL like this:
This is My website: https://www.houseofblogger.com
then my robots.txt file will be located at: https://houseofblogger.com/robots.txt
You Can Allow or disallow different Crawling access for a particular bot like Googlebot, yahoo bot, etc. So you might be curious now about what happens if I don’t use the robots.txt file on my website?
But is it really a necessary file to use or what happens if we are not using the robots.txt file on our website?
Is the Robots.txt file necessary?
This Answer simply depends on your requirements and needs but robots.txt is not a must-have file for your website.
Confused about when we should use the robots.txt file and when we don’t need the robots.txt file especially if our website is hosted on the blogger platform.
let’s clear your confusion…
When we must have a robots.txt file?
- You want to block specific pages and files from the search engine.
- You want that only Particular bots can access or crawl your website.
When we don’t need a robots.txt file?
- When you want to index everything available on your website.
- You don’t want to block any page or file blocked from the search engine.
Now let’s move on to how to create a robots.txt file if you need it.
How to create a Robots.txt file for blogger?
Creating a robots.txt file is really easy but many bloggers use a robots.txt generator to generate robots.txt files for blogger websites.
But I will show you how you can create a robots.txt file for your website by yourself.
So First look at the format of the robots.txt file.
This is the default format of any robots.txt file and you need to add rules for crawlers in the same manner.
Here, User-agent refers to the specific bot, and disallow refers to the content that we want to block from crawling.
Asterisk (*) Refers to all of the bots and crawlers.
If you want to block your about and contact page for Googlebot you will use:
Let’s have one more example:
Suppose You want to block label archives from indexing but you want to allow particular label archives to index so your robots.txt will be like this: (Suppose you want to index only 1 label named API)
I hope you got it.
Let’s move on next part “What is the perfect robots.txt file for a blogger website”
Perfect Robots.Txt for Blogger
In Blogger We don’t have a login page, user details, and dashboard pages like WP and others to block from crawlers.
but we should block search queries from crawlers.
like if someone searches the “Median” keyword in our website’s search box so blogger will generate this type of URL: https://www.website.com/search?q=median
but we don’t want to index these types of links in the search engine.
So after disallowing these search query links your Robots.txt will look like this:
User-agent: * Disallow: /search?q= Allow: / Sitemap: https://www.yourwebsite.com/sitemap.xml Sitemap: https://www.yourwebsite.com/sitemap-pages.xml Sitemap: https://www.yourwebsite.com/atom.xml?redirect=false&start-index=1&max-results=500
www.yourwebsite.com with your
Just Copy this Code and paste it into your robots.txt section.
To open the robots.txt section:-
- Go to Blogger Dashboard.
- Click on setting > scroll down.
- Find Custom Robots.Txt and enable them.
- Paste the above code in that section.
Important links and resources
Today I shared a full guide for creating robots.txt to upload it on your website for blogger. I hope you find this helpful.
Drop a like and share with your blogger friend because-
Great Satisfaction Comes from sharing with others.