hello friends today you are going to read a tutorial on what is robots.txt
and how to set custom robots.txt in blogger.
robots.txt is very important thing for any site
Web site owners use the /robots.txt file to give instructions
about their site to web robots; this is called The Robots Exclusion Protocol.
It works like a robot wants to vists a Web site URL,
say http://92tricks.blogspot.com
Before it does so, it firsts checks for http://92tricks.blogspot.com/robots.txt, and finds
The User-agent: * means this section applies to all robots. The Disallow: /
tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:
robots can ignore your /robots.txt. Especially malware robots that scan
the web for security vulnerabilities,
and email address harvesters used by spammers will pay no attention.
the /robots.txt file is a publicly available file.
Anyone can see what sections of your server you don't want robots to use.
if you want To allow all robots to complete access
then you can use
and how to set custom robots.txt in blogger.
robots.txt is very important thing for any site
what is robots.txt
Web site owners use the /robots.txt file to give instructions
about their site to web robots; this is called The Robots Exclusion Protocol.
How it works??
It works like a robot wants to vists a Web site URL,
say http://92tricks.blogspot.com
Before it does so, it firsts checks for http://92tricks.blogspot.com/robots.txt, and finds
User-agent: *
Disallow: /The User-agent: * means this section applies to all robots. The Disallow: /
tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:
robots can ignore your /robots.txt. Especially malware robots that scan
the web for security vulnerabilities,
and email address harvesters used by spammers will pay no attention.
the /robots.txt file is a publicly available file.
Anyone can see what sections of your server you don't want robots to use.
if you want To allow all robots to complete access
then you can use
User-agent: *
Disallow:or just create an empty robots.txt file
To exclude all robots from the entire server
User-agent: *
Disallow: /To exclude a single robot.
use this robots.txt
User-agent: BadBot
Disallow: /if want to allow just one robot to access your site
use below txt in this robots.txt i allow google to access my site
User-agent: Google
Disallow:
User-agent: *
Disallow: /i hope now you understand about robots.txt if not then don't worry
i am here to teach you.
you can ask me in comments.
how to set custom robots.txt in blogger?
login to blogger.com
click on blog title in which blog you want to set
custom robots.txt
now click on settings
search petferences
click edit custom robots.txt like bellow screen shoot.
check yes to enable robots.txt
when you check yes you will see text box just past your robots.txt here
and click save changes.
here is robots.txt that you can use as your robots.txt
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.92tricks.blogspot.com/sitemap.xml
Dear readers, after reading the Content please ask for advice and to provide constructive feedback Please Write Relevant Comment with Polite Language.Your comments inspired me to continue blogging. Your opinion much more valuable to me. Thank you. Conversion Conversion Emoticon Emoticon