| Rules |
Explanation |
User-agent: *
Disallow:
|
The asterisk (*) means this rule applies to “all robots”. Because ‘Disallow’ is empty, everything is allowed. |
User-agent: *
Disallow: /tmp/
Disallow: /lib/
|
All robots can visit every directory except ‘tmp’ and ‘lib’. |
User-agent: BadRobot
Disallow: / |
BadRobot is blocked for every directory. ‘/’ means it’s for all directories under the root directory. |
User-agent: BadRobot
Disallow: /
User-agent: *
Disallow: /lib/ |
The blank line in between means a new command is followed. BadRobot is blocked for every directory and all other robots can visit every directory except ‘lib’. |
- Don’t miss ‘:’ between indicator and value.
- Use ‘*’ for all user agents.
- Use ‘/’ for all directories.
|
