User-agent: Scrapy Allow: / # Allow crawling of all content User-agent: * Disallow: