# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: User-agent: * Disallow: /u/ User-agent: Googlebot-Image Disallow: / User-agent: msnbot-media Disallow: / User-agent: psbot Disallow: / User-agent: Yahoo-MMCrawler Disallow: /