User-agent: Slurp
Crawl-delay: 0.5
Allow: /