User-agent: bot* Disallow: / User-agent: robot Disallow: / User-agent: crawl Disallow: / User-agent: spider Disallow: / User-agent: \wbot[\/\-] Disallow: /