# allow only important bots User-agent: * User-agent: Mediapartners-Google* User-agent: ArchitextSpider User-agent: Baiduspider User-agent: Googlebot User-agent: Googlebot-Image User-agent: Mediapartners-Google User-agent: msnbot User-agent: msnbot-media User-agent: Scooter User-agent: Slurp User-agent: Teoma User-agent: Yahoo-Blogs User-agent: Yahoo-MMCrawler Allow: / Disallow: /cgi-bin/ Disallow: /AdTests/ Disallow: /AltAds/ Disallow: /artwine/ Disallow: /Business/ Disallow: /files/ Disallow: /photos/ Disallow: /graphics/ Disallow: /graphicstemp/ Disallow: /hotelgraphics/ Disallow: /vansphotos/ User-agent: 62.24.104.2 #Kenya spider Disallow: / User-agent: 63-217-30-178.pccwglobal.net Disallow: / User-agent: 68.180.151.76 #Phishing URI, abbey-bnk-plc.com Disallow: / User-agent: 98.139.168.220 #Phishing URI, yahoo.com Disallow: / User-agent: 98.138.19.88 #Phishing URI, pictsforeby.com Disallow: / User-agent: 98.138.19.88 #Phishing URI, rumbaxu.com Disallow: / User-agent: 206.219.84.250 Disallow: / User-agent: office.newtechbeverage.com Disallow: / User-agent: spider59.yandex.ru Disallow: / User-agent: tpiol.tpiol.com Disallow: