How to Write robots.txt

Buginess Forms

Rules	Explanation
User-agent: * Disallow:	The asterisk (*) means this rule applies to "all robots". Because 'Disallow' is empty, everything is allowed.
User-agent: * Disallow: /tmp/ Disallow: /lib/	All robots can visit every directory except 'tmp' and 'lib'.
User-agent: BadRobot Disallow: /	BadRobot is blocked for every directory. '/' means it's for all directories under the root directory.
User-agent: BadRobot Disallow: / User-agent: * Disallow: /lib/	The blank line in between means a new command is followed. BadRobot is blocked for every directory and all other robots can visit every directory except 'lib'.
- Don't miss ':' between indicator and value. - Use '*' for all user agents. - Use '/' for all directories.