User-agent:* Disallow: /js/ Disallow: /n/ Crawl-delay: 10 #Request-rate: 60/1m