All toolsGenerated result
robots.txt Generator
Generate robots.txt directives from user-agent names, allowed and disallowed paths, and a sitemap URL.
Waiting for input
Input and result notes
robots.txt controls crawling, not access permissions, and does not guarantee exclusion from search. Blocking a crawl can prevent a search engine from seeing a page's noindex directive. Protect private content with authentication.