Allow public crawling by default, add custom rules, or explicitly block crawling for a temporary environment.
Free generator, runs in your browser
Start free
Free Robots.txt Generator for SEO & AI Crawlers
The Novaverb robots.txt generator builds a plain crawl policy file in your browser from User-agent, Allow, Disallow and Sitemap records, with optional AI crawler groups and a path tester to use before you publish.
Instant, no account
Nothing is uploaded
Copy or download the result
Your inputRobots.txt Generator
Choose a base policy, then add only the paths that need an exception.
Plain robots.txtStarts with a valid allow-all group. Your path and AI choices are added to the generated file in real time.
Do not use this to protect private content. A blocked URL can still be discovered or indexed, and compliant crawling is voluntary. Use authentication for private environments and noindex for indexing control.
Use * for all crawlers or one exact product token such as Googlebot.
Use * or a product token containing only letters, hyphens and underscores.
Enter paths on this host. A missing leading slash is added automatically.
Use a more specific Allow path inside a broader blocked area.
Remove comments, line breaks or control characters from rule values.
Use complete http:// or https:// URLs. Cross-host sitemap URLs are permitted.
Every sitemap must be a complete HTTP or HTTPS URL.
AI crawler preferencesAdd separate, named groups without changing Google Search crawling.
No named AI groups will be added. All crawlers fall back to your main group.
Would this path be crawlable?Simulates the current group; test the live file again after publishing.
AllowedNo matching restriction - crawling is allowed.
1Active rules
0Valid sitemaps
0Input warnings
Generated robots.txtrobots.txt
Publish at /robots.txtCorrect the highlighted field to generate the file.
Private by design: this generator runs entirely in your browser - your text is never sent to Novaverb or any server.
Turn output into a monitored workflow
A Novaverb workspace crawls your live site, keeps what it finds, tracks changes over time and turns findings into assigned work.
Definition
What is a robots.txt generator?
A robots.txt generator creates a plain-text crawl policy for automated user-agents on one website host. The Novaverb Robots.txt Generator builds User-agent, Allow, Disallow and Sitemap records from a form, adds optional named groups for AI crawlers, and lets you test a path against the rules before the file is published.
Scope
What does the robots.txt generator create?
The robots.txt generator creates one robots.txt file with a main crawler group, its Disallow and Allow lines, optional groups for six named AI crawlers and any Sitemap lines you supply. A platform starter for WordPress, Drupal, Joomla, Shopify, Next.js, Laravel or Django sets a visible, editable baseline. No Crawl-delay line is added.
Who it is for
Who is the robots.txt generator for?
The robots.txt generator is for people who control what crawlers may request on a host. A developer drafts the file for a new site or a temporary environment, an SEO specialist limits crawling of filters, carts and utility paths, and a publisher sets a separate preference for AI search and model-training crawlers.
Why it matters
Why use a robots.txt generator?
A robots.txt generator catches a rule mistake before a crawler reads it. One wrong Disallow line can block a whole section, and an Allow exception may not win the way you expect. Testing a path against the draft shows the winning rule first, and invalid tokens or sitemap URLs are counted as input warnings.
Method
How does the robots.txt generator work?
The robots.txt generator works in five steps that end with a live fetch. Choose a starting policy, target a crawler group with an asterisk or an exact product token, add Disallow paths and Allow exceptions, test representative paths in the tester, then publish at the host root and fetch the live URL. The file rebuilds as you type.
What is a robots.txt generator?
A robots.txt generator creates a plain-text crawl policy for automated user-agents on one website host. This tool builds User-agent, Allow, Disallow and Sitemap records in your browser and lets you test a path before publishing.
User-agentAllowDisallowSitemap
Create and verify robots.txt in five steps
The file and tester update instantly without uploading your rules.
Use * for all compliant crawlers or enter the exact product token for one crawler.
Disallow broad paths and use more-specific Allow exceptions only where needed.
Try public, blocked and exception paths. The preview identifies the winning rule.
Upload robots.txt to the host root, then fetch the live URL and confirm the response is current.
How rule matching works
The tester follows the key matching behavior defined by the Robots Exclusion Protocol.
A longer matching path takes precedence over a shorter, broader rule.
When equivalent Allow and Disallow rules both match, the Allow rule should be used.
/Folder/ and /folder/ are different paths. Test the same casing your server uses.
RFC 9309 compliance checklist
These are protocol requirements and crawler-handling rules, not optional SEO folklore.
§2.3Correct file and locationServe lowercase /robots.txt at the top level of each scheme, host and port as UTF-8 text/plain.
§2.2.1–2.2.3Groups and matchingUser-agent tokens use letters, hyphens or underscores. Matching is case-sensitive; the longest rule wins and Allow wins an equal tie.
§2.3.1HTTP results matterCrawlers should follow at least five redirects. A 4xx file may mean crawl is allowed; a 5xx or network failure means complete disallow while unreachable.
§2.4–2.5Cache and parser limitsA cached file normally should not be used beyond 24 hours, and parsers must support at least 500 KiB.
Start from a transparent CMS baseline
Selecting a platform loads a visible, editable starter into Generated robots.txt. Novaverb never adds hidden directives; verify the installed CMS version before replacing a platform-managed file.
/robots.txt, /wp-sitemap.xmlWordPress can serve both endpoints virtually. Inspect the current response first because plugins can replace the core output; only add rules you have verified for this site.
WordPress core referenceweb/robots.txtDrupal Composer Scaffold manages the root file. Append or patch project-specific rules through scaffold configuration so core updates do not silently erase them.
Drupal scaffold guiderobots.txt.dist → robots.txtUse the plain file shipped with your exact Joomla release as the current platform source. Do not import a path list from an older release or another website.
Joomla core filetemplates/robots.txt.liquidShopify already serves an SEO-oriented default. Customize the Liquid template only when needed; replacing the complete generated file can remove future platform updates.
Shopify guidanceapp/robots.tsUse the MetadataRoute.Robots convention for typed static or dynamic output, or add app/robots.txt for a plain static file.
Next.js robots referencepublic root or explicit /robots.txt routeThese frameworks do not imply one universal SEO rule set. Serve the generated text from the public root or a text/plain route and test the production host.
Choose AI access by purpose
Search visibility, model development and user-requested retrieval are different decisions. Provider names and policies can change, so recheck the linked source before publishing.
OAI-SearchBot, Claude-SearchBot, PerplexityBotBlocking these can reduce discovery, snippets, citations or search-result quality in the corresponding AI products.
GPTBot, ClaudeBot, Google-ExtendedThese named controls relate to potential model training, model improvement or Gemini grounding. Google-Extended does not affect Google Search inclusion or ranking.
A user-agent rule is not identity verification, access control or a universal opt-out. Review provider documentation and validate real traffic separately.
Use the right control for the job
robots.txt is a crawl preference, not access control or a guaranteed indexing command.
Good uses
- Reduce crawling of duplicate filters, carts and low-value utility paths.
- Expose one or more absolute sitemap URLs.
- Create specific rules for crawlers that identify themselves and honor the protocol.
Not guaranteed by robots.txt
- Keeping sensitive content private - use authentication and authorization.
- Removing a URL from search - use an indexable noindex response, removal workflow or access control as appropriate.
- Forcing every crawler or AI system to comply - the protocol is advisory.
Generated is not the same as live
A correct draft can still be uploaded to the wrong host, cached, redirected or served with an error. After publishing, fetch the real file, verify its HTTP status and confirm that the Sitemap directives resolve.
Input
What do you enter into the robots.txt generator?
Enter a starting policy, a platform, a crawler user-agent, Disallow and Allow paths one per line, sitemap URLs and an AI policy into the robots.txt generator. A missing leading slash is added automatically. An invalid crawler token, a rule containing a comment or control character, or an incomplete sitemap URL is flagged and blocks the output until corrected.
Result
What does the robots.txt generator output contain?
The robots.txt generator output contains the generated file, a tester and a summary. The file starts with a comment naming the starter, then the main group, any AI crawler groups and the Sitemap lines. The summary counts active rules, valid sitemaps and input warnings. Copy and Download save the text as robots.txt.
Reading it
How do you use the robots.txt generator output?
Use the robots.txt generator output by first typing real paths into the tester: a public page, a blocked path and an exception. Each shows Allowed or Blocked and names the winning rule. Then upload the file to the root of the exact host, because a file in a subfolder does not control the host.
Source
Where does the robots.txt generator run?
The robots.txt generator runs in your browser. The file and the tester update instantly from the form without uploading your rules, and no live site is fetched. The tester therefore simulates the current draft group only. The published file has to be tested again once it is live on the host.
Limits
What does the robots.txt generator not tell you?
The robots.txt generator does not tell you what your live host serves or how any crawler will behave. A correct draft can still be uploaded to the wrong host, cached, redirected or served with an error. The file is advisory, so it cannot keep content private or remove a URL from search.
Evidence
What does a robots.txt generator result prove?
A robots.txt generator result proves that the draft is built from valid records and that a tested path is allowed or blocked under its matching rules. The result cannot prove that a crawler honors the file, that a named AI crawler is who it claims to be, or that a blocked URL stays out of an index.
Standards
Which standards does the robots.txt generator apply?
The robots.txt generator applies RFC 9309, the Robots Exclusion Protocol. User-agent tokens use letters, hyphens or underscores. Path matching is case-sensitive, the longest matching rule wins and Allow wins an equal tie. The file belongs at lowercase /robots.txt at the top level of each scheme, host and port, served as UTF-8 plain text.
Verification
How can you verify robots.txt generator output yourself?
Verify robots.txt generator output after publishing by fetching the real file at the root of the host. Confirm the HTTP status is successful, that the response is current and not a cached or redirected copy, and that every Sitemap line resolves. Repeat the path tests against the published rules, because the tester only simulated the draft.
Example
What does a typical robots.txt generator result look like?
A typical robots.txt generator result is a WordPress starter that disallows the admin path and allows the admin-ajax file inside it, plus one sitemap URL. Testing the admin-ajax path returns Allowed, with the Allow line named as the winning rule, because the more specific match beats the broader Disallow.
Cost
Is the robots.txt generator free?
The robots.txt generator is free and needs no account for the form, the platform starters, the AI crawler options, the path tester or the copy and download buttons. A Novaverb account is needed only to verify the file on your live site and track it over time.
In a workspace
What does a Novaverb workspace add to the robots.txt generator?
A Novaverb workspace adds evidence from the live site to the robots.txt generator, which only drafts a file. Crawl Explorer continues the job: it lists every URL captured in your project crawl with its HTTP status, indexability signals and depth, so the effect of the published rules can be checked against captured pages.
When to go further
When is a free robots.txt generator not enough?
A free robots.txt generator is not enough when you need to know what the rules do across a real site. Seeing which sections a crawl no longer reaches, catching a platform update that overwrote the file, or confirming that each subdomain serves the right file takes a crawl and repeated checks.
Next step
What should you do after using the robots.txt generator?
After using the robots.txt generator, compare the draft with the file your platform already serves, since replacing a platform-managed file can lose its defaults. Upload the result to the host root, fetch the live URL, confirm the status and the sitemap lines, and test the same paths again against the published file.
Related tools
Which free tools pair with the robots.txt generator?
Free tools that pair with the robots.txt generator cover the checks a draft cannot make. The Robots.txt Checker fetches the live file and reads its directives. The XML Sitemap Generator creates the sitemap that a Sitemap line points to. The Meta Tag Generator outputs the page-level robots directive used for indexing control.
Meta Tag GeneratorFree meta tag generator: build title, description, canonical, robots, …
XML Sitemap GeneratorFree XML sitemap generator: build a valid sitemap.xml or …
Word CounterFree word counter: count words, characters, sentences and paragraphs, …
Convert CaseFree convert case tool: switch text between sentence case, …
Redirect Rule GeneratorFree redirect rule generator: turn a list of old …
Schema Markup GeneratorFree schema markup generator: JSON-LD for FAQ, Article, Organization, …
Text CompareFree text compare tool: see every added, removed and …
Image ConverterFree image converter: change up to 20 pictures at …
Image CompressorFree image compressor: shrink up to 20 images in …
Image ResizerFree image resizer: set up to 20 images to …
AI Content CheckerFree AI content checker: measures repeated word forms, sentence …
Plagiarism CheckerFree plagiarism checker: searches the longest sentences of a …
Online HTML EditorFree online HTML editor: write or paste HTML and …
SEO ROI CalculatorFree SEO ROI calculator: turn investment, organic sessions, conversion …
UTM Campaign BuilderFree UTM builder: create a tagged campaign URL from …
Campaign Brief BuilderFree campaign brief builder: turn an objective, audience, offer, …
SEO Traffic ForecastFree SEO traffic forecast: model organic clicks, conversions and …
Content Calendar PlannerFree content calendar planner: turn a topic, audience, format …
Marketing KPI CalculatorFree marketing KPI calculator: work out CTR, CPC, conversion …
Website SEO CheckerHomepage status, indexability, title, headings and links from one live scan
Keyword Research ToolFree keyword research tool: monthly searches, cost per click, …
SERP CheckerFree SERP checker: the organic results for a keyword …
Website Security CheckerFree website security checker: a limited outside check of …
WordPress Security Configuration CheckerFree WordPress security checker: eight public configuration areas on …
Server Response Time CheckerFree server response time checker: one live request from …
Backlink CheckerFree backlink checker: see the referring domains, link totals, …
Robots.txt CheckerFree robots.txt tester: fetch a site's live robots.txt and …
Sitemap CheckerFree sitemap checker: find a site's XML sitemaps from …
Meta Tag CheckerFree meta tag checker: a page's live title, meta …
HTTP Status & Redirect CheckerFree redirect checker: trace every hop of a URL's …
Website MonitorFree website status checker: one live request to your …
HTTP/2 TestFree HTTP/2 test: opens one live TLS connection to …
HTTP/3 TestFree HTTP/3 test: attempts a real QUIC connection to …
Website Performance TestFree website performance test: time one request to your …
GEO CheckerFree GEO checker: retrieve, quote, cite and graph signals …
Core Web Vitals CheckerFree Core Web Vitals checker: real-user LCP, INP and …
PageSpeed CheckerFree PageSpeed checker: a Lighthouse lab audit of a …
Website Safety CheckerFree website safety checker: see whether a URL is …
Knowledge Graph CheckerFree Knowledge Graph checker: look up a brand or …
Keyword Gap CheckerFree keyword gap analysis: keywords a competitor ranks for …
Competitor Top PagesFree competitor top pages tool: a domain's leading pages …
Questions
What do people ask about the robots.txt generator?
People ask the robots.txt generator FAQ what robots.txt does, where the file should be placed, and whether blocking a URL prevents indexing, which it does not. They also ask about Crawl-delay, which rule wins when Allow and Disallow both match, adding a sitemap, blocking AI crawlers, and replacing a platform default file.
What does robots.txt do?
robots.txt tells compliant crawlers which URL paths they may request on a specific scheme, host and port. It manages crawling; it is not authentication and does not guarantee that a discovered URL will stay out of an index.
Where should robots.txt be placed?
Place it at the root of the exact host, for example https://example.com/robots.txt. A file in a subfolder does not control the whole host, and rules do not automatically transfer between subdomains.
Does robots.txt prevent Google indexing?
No. A blocked URL can still be discovered from links and may appear without a snippet. Use an appropriate noindex directive when you need indexing control, and do not block the crawler from reading that directive.
Does Google support Crawl-delay?
No. Google documents User-agent, Allow, Disallow and Sitemap as supported robots.txt fields and does not support Crawl-delay, so this generator intentionally does not add it.
Which rule wins when Allow and Disallow both match?
The most specific matching path wins. If an Allow and Disallow rule are equivalent, Allow should win. Matching is case-sensitive.
Can I add a sitemap to robots.txt?
Yes. Add one or more complete HTTP or HTTPS sitemap URLs. The Sitemap value must be absolute, and it may point to a different host.
Can robots.txt block AI crawlers?
You can target a crawler that identifies itself with a product token, but only crawlers that recognize and honor the Robots Exclusion Protocol will comply. Do not treat the file as a guarantee against access, training or reuse.
Which AI crawlers should I allow or block?
Decide by purpose. OAI-SearchBot, Claude-SearchBot and PerplexityBot support search discovery or citations; GPTBot, ClaudeBot and Google-Extended provide separate controls related to model development or Gemini grounding. Policies change, so verify each provider’s current documentation before publishing.
Should I replace the default robots.txt in WordPress, Drupal, Joomla or Shopify?
Usually no. Start from the file or virtual output maintained by the installed platform version, then add only verified project-specific changes. Shopify recommends preserving its generated Liquid defaults, while Drupal Composer Scaffold can manage and update the root file.
Is my data uploaded anywhere?
No. This generator runs fully client-side in your browser. Nothing you type is sent to a server, so it is safe to use on private content.
Is it really free?
Yes, with no account. A Novaverb workspace is only needed when you want your live site crawled, the results kept and changes tracked over time.
Copied to clipboard