How to Write robots.txt

Rules Explanation

User-agent: *

Disallow:
The asterisk (*) means this rule applies to “all robots”. Because ‘Disallow’ is empty, everything is allowed.
User-agent: *
Disallow: /tmp/
Disallow: /lib/
All robots can visit every directory except ‘tmp’ and ‘lib’.
User-agent: BadRobot
Disallow: /
BadRobot is blocked for every directory. ‘/’ means it’s for all directories under the root directory.
User-agent: BadRobot
Disallow: /
User-agent: *
Disallow: /lib/
The blank line in between means a new command is followed. BadRobot is blocked for every directory and all other robots can visit every directory except ‘lib’.
- Don’t miss ‘:’ between indicator and value.
- Use ‘*’ for all user agents.
- Use ‘/’ for all directories.

No Comments

Leave a comment