Allow public crawling by default, add custom rules, or explicitly block crawling for a temporary environment.
smart_toyFree generator · runs in your browser
Start free
Free Robots.txt Generator & Rule Tester
Create an RFC-aligned robots.txt with User-agent, Allow, Disallow and Sitemap rules. Test example paths, copy or download the file free—no signup.
boltInstant, no account
shield_lockNothing is uploaded
content_copyCopy or download the result
tune
Your inputRobots.txt Generator
Choose a base policy, then add only the paths that need an exception.
warningDo not use this to protect private content. A blocked URL can still be discovered or indexed, and compliant crawling is voluntary. Use authentication for private environments and noindex for indexing control.
Use * for all crawlers or one exact product token such as Googlebot.
Use * or a product token containing only letters, hyphens and underscores.
Enter paths on this host. A missing leading slash is added automatically.
Use a more specific Allow path inside a broader blocked area.
Remove comments, line breaks or control characters from rule values.
Use complete http:// or https:// URLs. Cross-host sitemap URLs are permitted.
Every sitemap must be a complete HTTP or HTTPS URL.
Would this path be crawlable?Simulates the current group; test the live file again after publishing.
AllowedNo matching restriction — crawling is allowed.
1Active rules
0Valid sitemaps
0Input warnings
Generated robots.txtrobots.txt
Publish at /robots.txtsmart_toy
Correct the highlighted field to generate the file.
verified_userPrivate by design: this generator runs entirely in your browser — your text is never sent to Novaverb or any server.
Turn output into a monitored workflow
A Novaverb workspace crawls your live site, verifies these tags and files are actually served, tracks changes over time, and turns findings into assigned work.
How to use it
Robots.txt Generator
What is a robots.txt generator?
A robots.txt generator creates a plain-text crawl policy for automated user-agents on one website host. This tool builds User-agent, Allow, Disallow and Sitemap records in your browser and lets you test a path before publishing.
User-agentAllowDisallowSitemap
Create and verify robots.txt in five steps
The file and tester update instantly without uploading your rules.
Use * for all compliant crawlers or enter the exact product token for one crawler.
Disallow broad paths and use more-specific Allow exceptions only where needed.
Try public, blocked and exception paths. The preview identifies the winning rule.
Upload robots.txt to the host root, then fetch the live URL and confirm the response is current.
How rule matching works
The tester follows the key matching behavior defined by the Robots Exclusion Protocol.
A longer matching path takes precedence over a shorter, broader rule.
When equivalent Allow and Disallow rules both match, the Allow rule should be used.
/Folder/ and /folder/ are different paths. Test the same casing your server uses.
Use the right control for the job
robots.txt is a crawl preference, not access control or a guaranteed indexing command.
check_circleGood uses
- Reduce crawling of duplicate filters, carts and low-value utility paths.
- Expose one or more absolute sitemap URLs.
- Create specific rules for crawlers that identify themselves and honor the protocol.
warningNot guaranteed by robots.txt
- Keeping sensitive content private — use authentication and authorization.
- Removing a URL from search — use an indexable noindex response, removal workflow or access control as appropriate.
- Forcing every crawler or AI system to comply — the protocol is advisory.
travel_exploreGenerated is not the same as live
A correct draft can still be uploaded to the wrong host, cached, redirected or served with an error. After publishing, fetch the real file, verify its HTTP status and confirm that the Sitemap directives resolve.
Common questions
Robots.txt Generator FAQ
What does robots.txt do?
robots.txt tells compliant crawlers which URL paths they may request on a specific scheme, host and port. It manages crawling; it is not authentication and does not guarantee that a discovered URL will stay out of an index.
Where should robots.txt be placed?
Place it at the root of the exact host, for example https://example.com/robots.txt. A file in a subfolder does not control the whole host, and rules do not automatically transfer between subdomains.
Does robots.txt prevent Google indexing?
No. A blocked URL can still be discovered from links and may appear without a snippet. Use an appropriate noindex directive when you need indexing control, and do not block the crawler from reading that directive.
Does Google support Crawl-delay?
No. Google documents User-agent, Allow, Disallow and Sitemap as supported robots.txt fields and does not support Crawl-delay, so this generator intentionally does not add it.
Which rule wins when Allow and Disallow both match?
The most specific matching path wins. If an Allow and Disallow rule are equivalent, Allow should win. Matching is case-sensitive.
Can I add a sitemap to robots.txt?
Yes. Add one or more complete HTTP or HTTPS sitemap URLs. The Sitemap value must be absolute, and it may point to a different host.
Can robots.txt block AI crawlers?
You can target a crawler that identifies itself with a product token, but only crawlers that recognize and honor the Robots Exclusion Protocol will comply. Do not treat the file as a guarantee against access, training or reuse.
Is my data uploaded anywhere?
No. This generator runs fully client-side in your browser. Nothing you type is sent to a server, so it is safe to use on private content.
Is it really free?
Yes, with no account. A Novaverb workspace is only needed when you want to verify, monitor, and track these files on your live site over time.
More free tools
Explore all Novaverb Free Tools
code_blocks
Meta Tag GeneratorCreate valid SEO meta tags for titles, descriptions, canonicals, …
arrow_forwardaccount_treeXML Sitemap GeneratorTurn a list of URLs into a valid XML …
arrow_forwardmatch_wordWord & Character CounterCount words, characters, sentences, paragraphs and reading time live, …
arrow_forwardmatch_caseConvert CaseInstantly convert text to Sentence case, lower case, UPPER …
arrow_forwardimageImage ConverterConvert JPG, PNG, WebP, GIF, BMP, AVIF and ICO …
arrow_forwardcompressImage CompressorCompress JPG, PNG, WebP and AVIF images in your …
arrow_forwardaspect_ratioImage ResizerResize up to 20 JPG, PNG, WebP or AVIF …
arrow_forward
languageWebsite SEO CheckerCrawl coverage, indexable pages, internal links
arrow_forward
query_statsKeyword Research ToolCheck keyword search intent and view available measured search …
arrow_forwardshieldWebsite Security CheckerCheck a public website's SSL certificate, security headers, server …
arrow_forwardspeedServer Response Time CheckerMeasure server-side HTTP fetch time and response status for …
arrow_forwardlinkBacklink CheckerCheck the latest measured backlinks and referring domains available …
arrow_forwardfact_checkRobots.txt & Sitemap CheckerCheck robots.txt and the default sitemap.xml for free. Inspect …
arrow_forwardcodeMeta Tag CheckerCheck any live page's title tag, meta description, H1 …
arrow_forwardrouteRedirect CheckerCheck a URL's complete HTTP redirect chain for free. …
arrow_forwardboltHTTP/2 TestCheck whether a website supports HTTP/2 with a live …
arrow_forwardrocket_launchHTTP/3 TestCheck whether a website supports HTTP/3 over QUIC with …
arrow_forwardnetwork_checkWebsite Performance TestMeasure a URL's full connection waterfall — DNS lookup, …
arrow_forwardsmart_toyGEO CheckerCheck whether generative AI answer engines — ChatGPT, Perplexity, …
arrow_forward
check_circleCopied to clipboard