Posts tagged: Robots.txt

How to Write robots.txt

Rules Explanation User-agent: * Disallow: The asterisk (*) means this rule applies to "all robots". Because 'Disallow' is empty, everything is allowed. User-agent: * Disallow: /tmp/ Disallow: /lib/ All robots can visit every directory except 'tmp' and 'lib'. User-agent: BadRobot Disallow: / BadRobot is blocked for every directory. '/' means it's for all directories under the root directory. User-agent: BadRobot Disallow: / User-agent: * Disallow: /lib/ The blank line in between means a new command is followed. BadRobot is blocked for every directory and all other robots can visit every directory except 'lib'. - Don't miss ':' between indicator and value. - Use '*' for all user agents. - Use '/' for all directories.