How To Install A Robots.txt File Part 2

How To Install A Robots.txt File Part 2

What does the Code Mean ?

User-agent: *
The asterisk (*)  represents a special value and means any robot can visit your website (all are welcome).

Disallow:
The Disallow: line without a / (forward slash) tells the search engine robots (spiders) that they can index all of the contents within your entire website.

The Disallow: line without the trailing slash (/) tells all robots to index everything. If you have a line that looks like this:

Disallow: /membership/

It tells the robot that it cannot index the contents of that /membership/ directory.

Other robots.txt Protocol - robots.txt file

To allow all robots complete access:

User-agent: *
Disallow:

To exclude all robots from the server:

User-agent: *
Disallow: /

To exclude all robots from parts of a server:

User-agent: *
Disallow: /membership/
Disallow: /images-membership/
Disallow: /images-membership/

To exclude a single robot from the server:

User-agent: Google Bot (name of bot)
Disallow: /

To exclude a single robot from parts of a server:

User-agent: Google Bot (name of bot)
Disallow: /membership/
Disallow: /images-membership/
Disallow: /pictures-membership/

Note: The asterisk (*) in the User-agent field has the meaning "any Robot& and therefore is the only one needed until you fully understand how to set up different User-agents.

If you want to Disallow: a particular file within the directory, your Disallow: line might look like this one:

Disallow: /membership/for-my-eyes-only.htm

Keep in mind that using the above example excludes for-my-eyes-only.htm but will not exclude the entire /membership/ directory.

I would personally use only the very first example until you are more familiar with the different robots.txt codes. Most people only need to use this one.

Another way to have a protected area within your website is to not link any hyperlinks to that protected webpage, the robots will not find this particular webpage and it will remain hidden from the search engine spiders.