Smit Shah

Block All Spiders except for good ones

I started to notice that my server was having performance issues.
Based on the traffic and the server size that I have this should not be the case.
Upon review of the logs I noticed that there were crawlers munching up on my webpages.

This is what my robots.txt looks like now.

Only allow the good ones.
User-agent: Googlebot
Disallow: 
User-agent: Mediapartners-Google
Disallow: 
User-agent: MSNBot
Disallow: 
User-agent: Slurp
Disallow: 
User-agent: Teoma
Disallow: 
User-agent: twiceler
Disallow: 
User-agent: Gigabot
Disallow: 
User-agent: Scrubby
Disallow: 
User-agent: Robozilla
Disallow: 
User-agent: Nutch
Disallow: 
User-agent: ia_archiver
Disallow: 
User-agent: baiduspider
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: 
User-agent: psbot
Disallow: 
User-agent: asterias
Disallow: 
User-agent: yahoo-blogs/v3.9
Disallow: 
User-agent: *
Disallow: /
                                                                                         
This is a personal weblog. The opinions expressed here represent my own and not those of my employer. For accuracy and official reference refer to MSDN/ TechNet/ BOL /Other sites that are authorities in their field. Me or employer do not endorse any tools, applications, books, or concepts mentioned on the site. I have documented my personal experience on this site. The information on this site may not be up to date or accurate at times, if you are not sure or have a question, please contact me to verify information.