dave.io

User-Agent: * Disallow: /music? Disallow: /widgets/radio? Disallow: /show_ads.php Disallow: /affiliate/ Disallow: /affiliate_redirect.php Disallow: /affiliate_sendto.php Disallow: /affiliatelink.php Disallow: /campaignlink.php Disallow: /delivery.php Disallow: /music/+noredirect/ Disallow: /harming/humans Disallow: /ignoring/human/orders Disallow: /harm/to/self Allow: /

Someone at last.fm has been reading a bit too much Asimov - this is the contents of their robots.txt file.

For anyone confused, a robots.txt file tells search engines that respect it what they shouldn’t index, preventing excessive load, giving the search engines cleaner results and otherwise sanitising their behaviour.