This allows all robots to crawl all files. User-agent: * Disallow: /technology/ Disallow: /scripts/ Disallow: /staff/ Disallow: /scheduler/ Disallow: /help/ Disallow: /signin/ Disallow: /traffic/ Disallow: /calendar/