# Block all spiders, except where noted #User-agent: * #Disallow: / # Allow Google to index EM User-agent: googlebot Disallow: # Allow MSN to index EM User-agent: MSNBot Disallow: # Allow Yahoo to index EM User-agent: Slurp Disallow: # Allow Avnet Google To Index EM User-agent: gsa-crawler Disallow: