Robots.txt Generator
Robots.txt Generator is a free seo tool that runs directly in your browser with no account required.
About this tool
Generate a minimal, valid robots.txt with a universal user-agent block, one Allow rule, one Disallow rule, and a Sitemap declaration. Good for launching a new site or replacing a broken file with a known-safe baseline.
How to use
- Enter your sitemap's full URL in the Sitemap URL field.
- Set the Allow path (usually /) and the Disallow path (such as /admin/).
- Copy the generated file and save it as robots.txt at the root of your domain.
Frequently asked questions
Where does robots.txt need to live?
At the root of the host, e.g. https://example.com/robots.txt. Crawlers do not look for it in subdirectories, and each subdomain needs its own file.
Can I add multiple Disallow rules or user agents?
The generator outputs one User-agent: * block with a single Allow and Disallow. After copying, you can add more Disallow lines or agent-specific blocks by hand; the syntax is one directive per line.
Does Disallow keep a page out of Google?
Not reliably. It blocks crawling, but a URL can still be indexed from external links. Use a noindex meta tag or header (on a crawlable page) to keep content out of the index.
Why include the Sitemap line?
It tells any crawler where your XML sitemap lives without requiring separate submission in each search engine's console, and it must be an absolute URL.
Is anything about my site sent to a server?
No. The file is assembled from your inputs in the browser; you copy the text out yourself.