How to Write robots.txt

Rules Explanation
User-agent: *
Disallow:
The asterisk (*) means this rule applies to "all robots". Because 'Disallow' is empty, everything is allowed.



User-agent: *
Disallow: /tmp/
Disallow: /lib/
All robots can visit every directory except 'tmp' and 'lib'.



User-agent: BadRobot
Disallow: /
BadRobot is blocked for every directory. '/' means it's for all directories under the root directory.



User-agent: BadRobot
Disallow: /

User-agent: *
Disallow: /lib/
The blank line in between means a new command is followed. BadRobot is blocked for every directory and all other robots can visit every directory except 'lib'.



- Don't miss ':' between indicator and value.
- Use '*' for all user agents.
- Use '/' for all directories.