How to add a robots.txt file?
In this article, I am going to give a guide on adding roborts.txt to your website and check whether indexing is working on your site or not make sure to read each paragraph to understand properly about robots.txt.
What is Robots.txt
Robots.txt is a protocol, which has been made to give instruction to search engines to crawl the necessary page of the site and prevent from crawling the sites which we don’t want to index.
When the search engine bots are crawling our sites. They tend to look at the administrative page, backend portal, tag pages and make user-visible these things which owner of the site doesn’t want to make visible, So Robots.txt files give commands not to visit these sites.
In simple language, Robots.txt tells crawlers what to crawl on the site.
Creating a Robots.txt file
We can use any plain text editor to make the file[Remeber only plain text editor like notepad, Textedit]. Don't use other editors which add characters, curly brackets which make errors for crawlers.
In my case, I am going to use notepad
1. Open any editor and, Write [User-agent: *]
Remember, everything written here is case sensitive write properly otherwise errors may happen
First-line means everything written on the page can be crawled due to [*] sign
2. Write Disallow: /
This second line means anything written at disallowed won't be crawled.
3. Adding sitemap and saving as Robots.txt
It is not necessary at all but for crawlers to understand where to go . Write your webpage name in example.com.
Make sure to save as Robots.txt and encode UTF-8 as done in the image. Believe me or not that's simple making robots.txt
Where to Locate Robots.txt on our site?
The Robots.txt file can only be located in the site directory. For example:[example.com/robots.txt]. This is the right way to locate.
It can’t be located in our subdirectory[example.com/pictures/robots.txt].
If you are unable to go to the root file then contact your web hosting for permission.
We can use robots.txt in our subdomain [hello.example.com/robots.txt].
Importance of Robots.txt on our SEO based website
- For Hiding backends, the administrative page from the user.
2. To limit the crawl rate limit and Crawl Demand
According to google,
3. For not to crawl duplicate content
Duplicate content is one of the bad things that degrade our Seo optimization. So adding duplicate links at dis allowed is best for both for crawler and for our website.
