Free Robots.txt Generator
Build a valid robots.txt from safe presets: user-agents, allow and disallow rules, crawl-delay, preferred host and sitemap links. Download and drop it at your site root.
Start from a safe default, then refine the disallow list below.
Use * for every crawler, or a name like Googlebot for one.
Turn output into a monitored workflow
A Novaverb workspace crawls your live site, verifies these tags and files are actually served, tracks changes over time, and turns findings into assigned work.
Robots.txt Generator
robots.txt tells crawlers which parts of your site they may request. This generator builds a valid file from a safe preset plus your own allow/disallow rules, crawl-delay, preferred host and sitemap links.
How to use it
- Pick a preset. 'Allow all' is right for most public sites; 'Block all' is for staging environments you don't want indexed.
- Add paths you want to keep out of search under Disallow (e.g. /admin/, /cart/).
- Link your sitemap so crawlers discover your URLs faster.
- Download the file as robots.txt and upload it to your site root, at /robots.txt.
A word of caution
robots.txt controls crawling, not indexing — a blocked URL can still appear in results if it's linked elsewhere. To keep a page out of the index, use a noindex meta tag instead. Novaverb's Site Audit flags pages where these signals conflict.
Robots.txt Generator FAQ
Where does the file go?
At the root of your domain, reachable at https://your-domain/robots.txt. It must be at the root — a robots.txt in a subfolder is ignored.
Is crawl-delay respected by Google?
Googlebot ignores crawl-delay (set the rate in Search Console instead), but Bing and several other crawlers honour it.
Is my data uploaded anywhere?
No. This generator runs fully client-side in your browser. Nothing you type is sent to a server, so it is safe to use on private content.
Is it really free?
Yes, with no account. A Novaverb workspace is only needed when you want to verify, monitor, and track these files on your live site over time.