robots.txt Generator
Create Allow and Disallow rules and add a sitemap with clear technical warnings.
robots.txt controls crawling; it does not protect private information or replace noindex.
Generated robots.txt
—What robots.txt controls
robots.txt communicates crawling rules to compatible bots. It is published at the host root, such as https://example.com/robots.txt.
Disallow is not noindex
Blocking crawling does not guarantee that a URL disappears from results when discovered elsewhere. Use appropriate indexing controls such as noindex without blocking the access needed to read it.
It is not security
Paths in the file are public. Never use robots.txt to protect dashboards, personal data, or confidential files; apply real authentication and authorization.
Explore all web development tools