Allow public crawling by default, add custom rules, or explicitly block crawling for a temporary environment.
Free generator, runs in your browser
Start free
Free Robots.txt Generator for SEO & AI Crawlers
Create and test a plain RFC 9309-aligned robots.txt with CMS publishing guidance and optional AI crawler controls. Copy or download it free - no signup.
Instant, no account
Nothing is uploaded
Copy or download the result
Your inputRobots.txt Generator
Choose a base policy, then add only the paths that need an exception.
Plain robots.txtStarts with a valid allow-all group. Your path and AI choices are added to the generated file in real time.
Do not use this to protect private content. A blocked URL can still be discovered or indexed, and compliant crawling is voluntary. Use authentication for private environments and noindex for indexing control.
Use * for all crawlers or one exact product token such as Googlebot.
Use * or a product token containing only letters, hyphens and underscores.
Enter paths on this host. A missing leading slash is added automatically.
Use a more specific Allow path inside a broader blocked area.
Remove comments, line breaks or control characters from rule values.
Use complete http:// or https:// URLs. Cross-host sitemap URLs are permitted.
Every sitemap must be a complete HTTP or HTTPS URL.
AI crawler preferencesAdd separate, named groups without changing Google Search crawling.
No named AI groups will be added. All crawlers fall back to your main group.
Would this path be crawlable?Simulates the current group; test the live file again after publishing.
AllowedNo matching restriction - crawling is allowed.
1Active rules
0Valid sitemaps
0Input warnings
Generated robots.txtrobots.txt
Publish at /robots.txtCorrect the highlighted field to generate the file.
Private by design: this generator runs entirely in your browser - your text is never sent to Novaverb or any server.
Turn output into a monitored workflow
A Novaverb workspace crawls your live site, verifies these tags and files are actually served, tracks changes over time, and turns findings into assigned work.
How to use it
Robots.txt Generator
What is a robots.txt generator?
A robots.txt generator creates a plain-text crawl policy for automated user-agents on one website host. This tool builds User-agent, Allow, Disallow and Sitemap records in your browser and lets you test a path before publishing.
User-agentAllowDisallowSitemap
Create and verify robots.txt in five steps
The file and tester update instantly without uploading your rules.
Use * for all compliant crawlers or enter the exact product token for one crawler.
Disallow broad paths and use more-specific Allow exceptions only where needed.
Try public, blocked and exception paths. The preview identifies the winning rule.
Upload robots.txt to the host root, then fetch the live URL and confirm the response is current.
How rule matching works
The tester follows the key matching behavior defined by the Robots Exclusion Protocol.
A longer matching path takes precedence over a shorter, broader rule.
When equivalent Allow and Disallow rules both match, the Allow rule should be used.
/Folder/ and /folder/ are different paths. Test the same casing your server uses.
RFC 9309 compliance checklist
These are protocol requirements and crawler-handling rules, not optional SEO folklore.
§2.3Correct file and locationServe lowercase /robots.txt at the top level of each scheme, host and port as UTF-8 text/plain.
§2.2.1–2.2.3Groups and matchingUser-agent tokens use letters, hyphens or underscores. Matching is case-sensitive; the longest rule wins and Allow wins an equal tie.
§2.3.1HTTP results matterCrawlers should follow at least five redirects. A 4xx file may mean crawl is allowed; a 5xx or network failure means complete disallow while unreachable.
§2.4–2.5Cache and parser limitsA cached file normally should not be used beyond 24 hours, and parsers must support at least 500 KiB.
Start from a transparent CMS baseline
Selecting a platform loads a visible, editable starter into Generated robots.txt. Novaverb never adds hidden directives; verify the installed CMS version before replacing a platform-managed file.
/robots.txt, /wp-sitemap.xmlWordPress can serve both endpoints virtually. Inspect the current response first because plugins can replace the core output; only add rules you have verified for this site.
WordPress core referenceweb/robots.txtDrupal Composer Scaffold manages the root file. Append or patch project-specific rules through scaffold configuration so core updates do not silently erase them.
Drupal scaffold guiderobots.txt.dist → robots.txtUse the plain file shipped with your exact Joomla release as the current platform source. Do not import a path list from an older release or another website.
Joomla core filetemplates/robots.txt.liquidShopify already serves an SEO-oriented default. Customize the Liquid template only when needed; replacing the complete generated file can remove future platform updates.
Shopify guidanceapp/robots.tsUse the MetadataRoute.Robots convention for typed static or dynamic output, or add app/robots.txt for a plain static file.
Next.js robots referencepublic root or explicit /robots.txt routeThese frameworks do not imply one universal SEO rule set. Serve the generated text from the public root or a text/plain route and test the production host.
Choose AI access by purpose
Search visibility, model development and user-requested retrieval are different decisions. Provider names and policies can change, so recheck the linked source before publishing.
OAI-SearchBot, Claude-SearchBot, PerplexityBotBlocking these can reduce discovery, snippets, citations or search-result quality in the corresponding AI products.
GPTBot, ClaudeBot, Google-ExtendedThese named controls relate to potential model training, model improvement or Gemini grounding. Google-Extended does not affect Google Search inclusion or ranking.
A user-agent rule is not identity verification, access control or a universal opt-out. Review provider documentation and validate real traffic separately.
Use the right control for the job
robots.txt is a crawl preference, not access control or a guaranteed indexing command.
Good uses
- Reduce crawling of duplicate filters, carts and low-value utility paths.
- Expose one or more absolute sitemap URLs.
- Create specific rules for crawlers that identify themselves and honor the protocol.
Not guaranteed by robots.txt
- Keeping sensitive content private - use authentication and authorization.
- Removing a URL from search - use an indexable noindex response, removal workflow or access control as appropriate.
- Forcing every crawler or AI system to comply - the protocol is advisory.
Generated is not the same as live
A correct draft can still be uploaded to the wrong host, cached, redirected or served with an error. After publishing, fetch the real file, verify its HTTP status and confirm that the Sitemap directives resolve.
Common questions
Robots.txt Generator FAQ
What does robots.txt do?
robots.txt tells compliant crawlers which URL paths they may request on a specific scheme, host and port. It manages crawling; it is not authentication and does not guarantee that a discovered URL will stay out of an index.
Where should robots.txt be placed?
Place it at the root of the exact host, for example https://example.com/robots.txt. A file in a subfolder does not control the whole host, and rules do not automatically transfer between subdomains.
Does robots.txt prevent Google indexing?
No. A blocked URL can still be discovered from links and may appear without a snippet. Use an appropriate noindex directive when you need indexing control, and do not block the crawler from reading that directive.
Does Google support Crawl-delay?
No. Google documents User-agent, Allow, Disallow and Sitemap as supported robots.txt fields and does not support Crawl-delay, so this generator intentionally does not add it.
Which rule wins when Allow and Disallow both match?
The most specific matching path wins. If an Allow and Disallow rule are equivalent, Allow should win. Matching is case-sensitive.
Can I add a sitemap to robots.txt?
Yes. Add one or more complete HTTP or HTTPS sitemap URLs. The Sitemap value must be absolute, and it may point to a different host.
Can robots.txt block AI crawlers?
You can target a crawler that identifies itself with a product token, but only crawlers that recognize and honor the Robots Exclusion Protocol will comply. Do not treat the file as a guarantee against access, training or reuse.
Which AI crawlers should I allow or block?
Decide by purpose. OAI-SearchBot, Claude-SearchBot and PerplexityBot support search discovery or citations; GPTBot, ClaudeBot and Google-Extended provide separate controls related to model development or Gemini grounding. Policies change, so verify each provider’s current documentation before publishing.
Should I replace the default robots.txt in WordPress, Drupal, Joomla or Shopify?
Usually no. Start from the file or virtual output maintained by the installed platform version, then add only verified project-specific changes. Shopify recommends preserving its generated Liquid defaults, while Drupal Composer Scaffold can manage and update the root file.
Is my data uploaded anywhere?
No. This generator runs fully client-side in your browser. Nothing you type is sent to a server, so it is safe to use on private content.
Is it really free?
Yes, with no account. A Novaverb workspace is only needed when you want to verify, monitor, and track these files on your live site over time.
More free tools
Explore all Novaverb Free Tools
Meta Tag GeneratorCreate valid SEO meta tags for titles, descriptions, canonicals, …
XML Sitemap GeneratorCreate a valid XML sitemap or sitemap index from …
Word CounterCount words, characters, sentences, paragraphs and reading time instantly. …
Convert CaseConvert text to sentence case, lowercase, UPPER CASE, capitalized …
Image ConverterConvert up to 20 browser-supported images to JPG, PNG, …
Image CompressorCompress JPG, PNG, WebP and AVIF images in your …
Image ResizerResize up to 20 JPG, PNG, WebP or AVIF …
AI Content CheckerCheck writing signals associated with AI-generated text in a …
Plagiarism CheckerFind potential matching sources for selected sentences via a …
Online HTML EditorWrite and edit HTML with an instant live preview, …
SEO ROI CalculatorEstimate organic conversions, gross profit, monthly ROI and break-even …
UTM Campaign BuilderBuild consistent UTM campaign URLs for ads, email, social …
Campaign Brief BuilderTurn an objective, audience, offer and measurement plan into …
SEO Traffic ForecastModel organic clicks, conversions and revenue over a planning …
Content Calendar PlannerCreate a practical multi-week content calendar from one topic, …
Marketing KPI CalculatorCalculate CTR, CPC, conversion rate, CPA, revenue and ROAS …
Website SEO CheckerCrawl coverage, indexable pages, internal links
Keyword Research ToolCheck the exact query's available search volume, keyword difficulty …
SERP CheckerInspect the returned organic results for a keyword and …
Website Security CheckerAudit website security posture, TLS/SSL certificates, HTTP security headers, …
WordPress Security Configuration CheckerCheck eight externally observable WordPress configuration areas: XML-RPC, wp-login.php, …
Server Response Time CheckerMeasure server Time to First Byte (TTFB), DNS lookup, …
Backlink CheckerExplore backlinks, referring domains, dofollow links, and domain authority …
Robots.txt CheckerTest and validate robots.txt rules, User-Agent directives, blocked paths …
Sitemap CheckerDiscover sitemap declarations, inspect the root document and fetch …
Meta Tag CheckerCheck page title length, meta description, H1 heading structure, …
HTTP Status & Redirect CheckerTrace HTTP status codes (200, 301, 302, 404, 500) …
Website MonitorRun one live availability check and retain an evidence …
HTTP/2 TestCheck whether the exact submitted hostname negotiates HTTP/2 through …
HTTP/3 TestTest whether your web server supports HTTP/3 over QUIC …
Website Performance TestCompare HTTP response timing from available probe locations and …
GEO CheckerInspect observable page signals that support retrieval, answer extraction, …
Core Web Vitals CheckerCheck 75th-percentile real-user LCP, INP and CLS from Chrome …
PageSpeed CheckerRun one Lighthouse lab audit to inspect performance, accessibility, …
Website Safety CheckerCheck whether a domain or URL is flagged for …
Knowledge Graph CheckerLook up matching entities for a brand, person, product …
Keyword Gap CheckerFind ranking keywords observed for a competitor and not …
Competitor Top PagesFind the pages with the highest estimated organic traffic …
Copied to clipboard