Free tools · 0 AI · runs in browser
Generate a basic robots.txt allowing or blocking paths. Download instantly.
Choose allow-all or custom disallow paths and download a robots.txt. Pair with clean indexable product pages before you chase directory links.
User-agent: * Allow: /
Yes - accidentally disallowing important paths blocks crawling. Keep rules minimal unless you know why.
Related: Directory submission guide · All free tools · Directory database