1. Source
Live robots.txt fetch. The result identifies where its evidence came from.
Free tool, no account required
Free Robots.txt Tester & Validator
The Novaverb robots.txt tester fetches the live robots.txt file at a site's root and reports its HTTP status, user-agent groups, Allow and Disallow counts, sitemap declarations and any site-wide Disallow rule.
Also known as: robots.txt tester, validator
Evidence shown with sourceNo account requiredNo invented metrics
Run a real check above
Submit a public URL, domain, or keyword. Novaverb will show only the evidence this tool can actually retrieve or measure.
Definition
What is a robots.txt tester?
A robots.txt tester is a tool that fetches a site's robots.txt file and shows the crawl rules written in it. The Novaverb Robots.txt Checker reads the live file at the root of the exact hostname you submit and summarizes its status, user-agent groups, Allow and Disallow rules and sitemap lines.
Scope
What does the robots.txt tester check?
The robots.txt tester checks the HTTP status of the file, the user-agent values it names, the number of Allow and Disallow directives, every Sitemap line and the file size in bytes. A literal Disallow: / rule is flagged as a block on every path for its group, and the fetched file is displayed with its comments preserved.
Who it is for
Who is the robots.txt tester for?
The robots.txt tester is for developers who just deployed and need to confirm a staging block did not ship, SEO specialists auditing which crawler groups a site addresses, site owners deciding how to treat AI crawlers named in the file, and migration leads checking that the new host serves the file at its root.
Why it matters
Why use a robots.txt tester?
A robots.txt tester catches a crawl rule mistake before crawlers act on it. One Disallow: / line left in place can stop compliant crawlers from fetching site content, a file returning an error status gives crawlers no explicit guidance, and a missing Sitemap line removes a discovery route. A visitor using a browser notices none of these.
Method
How does the robots.txt tester work?
The robots.txt tester works by keeping the exact hostname you submitted, resolving it to a public address and fetching /robots.txt from the root with safe redirect handling. The returned text is then counted by User-agent, Allow, Disallow and Sitemap directive. An empty Disallow value stays permissive, and only the literal root rule is flagged as a block.
- We preserve the exact submitted hostname and fetch /robots.txt from its root with safe redirect handling.
- We count User-agent, Allow, Disallow and Sitemap directives from the captured file.
- We flag a literal Disallow: / root block and keep an empty Disallow value permissive; we do not simulate a named crawler's full policy.
Input
What do you enter into the robots.txt tester?
Enter a domain, or any URL on it, into the robots.txt tester. The host is taken from the address and the file is fetched from the root, the only location the specification recognizes. A robots.txt placed in a subfolder is not read and is reported as missing. Hosts that do not resolve to a public address are refused.
Use it like this
yourdomain.comAny address form works - we fetch /robots.txt at the host root for you. http or https, with or without www, a bare domain or a full path - we normalize it for you.https://www.yourdomain.com/anythingEven a deep URL is fine; we resolve to the host and read its root robots.txt.Avoid this
Expecting a subfolder robots.txt to countrobots.txt is only honored at the host root; a copy inside a subfolder is ignored by crawlers.Result
What does the robots.txt tester report contain?
A robots.txt tester report contains a header with the fetched URL and status, counts for user-agent groups, Disallow and Allow rules and declared sitemaps, a warning when a Disallow: / rule exists, the list of user-agent values, the file as served, each Sitemap directive found and an analysis panel with findings and priority fixes.
Reading it
How do you read a robots.txt tester result?
Read a robots.txt tester result by checking the status first: 200 means the file was served and its rules apply, while 404 means no file exists and crawlers assume full access. Next look for the Disallow: / warning and find which user-agent group owns the rule. Then confirm that at least one sitemap is declared.
Robots.txt explained
Test your robots.txt without overclaiming what it does
robots.txt tells compliant crawlers which paths they may request. It is the first file most crawlers fetch, so a single wrong line can quietly keep a whole site out of search - or let private paths be crawled.
What to check
- Location - the file must live at the domain root and return HTTP 200 as plain text.
- User-agent groups - each group's Allow/Disallow rules apply only to the agents it names; an empty Disallow allows every path, and Disallow: / blocks every path for that group.
- Declared sitemaps - publish the sitemap's absolute URL here so crawlers can discover it.
- Validation boundary - this page checks HTTP reachability and directive evidence; it is not a full RFC 9309 parser or a Search Console inspection.
Why it matters for SEO and AI discovery
A clear robots.txt reduces ambiguity for search and AI crawlers. Novaverb can use this signal as part of a broader crawl and technical evidence model, alongside indexability, canonicals and internal links.
Source
Where does the robots.txt tester get its data?
The robots.txt tester gets its data from a live fetch of /robots.txt on the exact hostname you submit, made when you run the check. The rules shown are the ones in that served file. No crawl history or search engine report is consulted, so the result reflects what the server returned to this single request.
Limits
What does the robots.txt tester not tell you?
The robots.txt tester does not tell you whether a named crawler may fetch a specific URL, because it does not simulate any crawler's matching, caching or policy behavior. Robots.txt is a crawl directive, not access control or index removal, so the result says nothing about whether a blocked page is private or absent from search.
Evidence
What does a robots.txt tester result prove?
A robots.txt tester result proves what the root file contained when it was fetched: its status, the user-agent values, the Allow and Disallow counts, the sitemap declarations and any literal Disallow: / rule. That supports conclusions about what crawlers are told. What a particular crawler then does with those rules cannot be concluded from the summary.
2. Boundary
The checker fetches /robots.txt and reads its directives. robots.txt is a crawl directive, not access control or index removal; this tool does not simulate a specific crawler's policy.
3. Next action
Use the finding to verify a problem, then connect a workspace when you need history, monitoring, or site-wide analysis.
Standards
Which standards does the robots.txt tester apply?
The robots.txt tester applies RFC 9309, the Robots Exclusion Protocol published by the IETF. The specification defines the syntax used to identify User-agent, Allow, Disallow and Sitemap lines, and the tool treats an empty Disallow as permissive. The summary does not claim to implement crawler-specific matching, caching or policy extensions beyond the fields it displays.
IETFRFC 9309
Robots Exclusion Protocol
Identifies captured User-agent, Allow, Disallow and Sitemap directives and treats an empty Disallow as permissive.
Read the specificationWe list a standard only where this tool genuinely reads or measures against it. Where a signal is outside a live check, the result says so instead of implying coverage.
Verification
How can you verify a robots.txt tester result yourself?
Verify a robots.txt tester result by loading the same hostname followed by /robots.txt in a browser and comparing the text with the file shown in the report. Count the Disallow lines by hand for a short file. To confirm the status code, request the file with a command line client and read the first response line.
Example
What does a typical robots.txt tester finding look like?
A typical robots.txt tester finding is a file that returns 200 with a single group for all crawlers, a few Disallow rules for admin and cart paths, and no Sitemap line. The analysis marks the file reachable, adds a warning that no sitemap is declared and suggests a Sitemap line so crawlers discover the sitemap directly.
Cost
Is the robots.txt tester free?
The robots.txt tester is free and no account is required. You can check any public hostname from the form. A free Novaverb account is needed for the step the file alone cannot answer: crawling the site to see every URL these rules allow or block, keeping the evidence and rechecking after the file changes.
In a workspace
What does a Novaverb workspace add to the robots.txt tester?
A Novaverb workspace adds a site crawl to the robots.txt tester, so the rules are seen against real URLs and important pages kept out of search by accident become visible. Crawl Explorer is the product that continues the job, letting you inspect every URL captured in the project crawl and turn each supported technical finding into an action.
When to go further
When is a free robots.txt checker not enough?
A free robots.txt checker run is not enough when you need to know which of your actual pages a rule affects, since that takes a crawl of the whole site. One check is also a single moment. A file overwritten by a later deploy is only noticed when the same check is repeated and compared over time.
Next step
What should you do after running the robots.txt tester?
After running the robots.txt tester, remove any site-wide Disallow: / rule that is not deliberate. If the file returned an error status, publish one at the host root, even a permissive one. Add a Sitemap line with the full sitemap URL. Then run the check again and confirm the status, counts and sitemap list changed.
Related tools
Which free tools pair with the robots.txt tester?
Related free tools pair with the robots.txt tester. The Robots.txt Generator creates an RFC 9309-aligned file with optional AI crawler controls. The Sitemap Checker follows the Sitemap lines declared in robots.txt and inspects those documents. The Website SEO Checker reads the homepage for a noindex directive alongside its robots.txt and sitemap rows.
Website SEO CheckerHomepage status, indexability, title, headings and links from one live scan
Keyword Research ToolFree keyword research tool: monthly searches, cost per click, ad …
SERP CheckerFree SERP checker: the organic results for a keyword in …
Website Security CheckerFree website security checker: a limited outside check of a …
WordPress Security Configuration CheckerFree WordPress security checker: eight public configuration areas on a …
Server Response Time CheckerFree server response time checker: one live request from Germany, …
Backlink CheckerFree backlink checker: see the referring domains, link totals, follow …
Sitemap CheckerFree sitemap checker: find a site's XML sitemaps from robots.txt …
Meta Tag CheckerFree meta tag checker: a page's live title, meta description, …
HTTP Status & Redirect CheckerFree redirect checker: trace every hop of a URL's redirect …
Website MonitorFree website status checker: one live request to your address …
HTTP/2 TestFree HTTP/2 test: opens one live TLS connection to a …
HTTP/3 TestFree HTTP/3 test: attempts a real QUIC connection to a …
Website Performance TestFree website performance test: time one request to your URL …
GEO CheckerFree GEO checker: retrieve, quote, cite and graph signals from …
Core Web Vitals CheckerFree Core Web Vitals checker: real-user LCP, INP and CLS …
PageSpeed CheckerFree PageSpeed checker: a Lighthouse lab audit of a public …
Website Safety CheckerFree website safety checker: see whether a URL is listed …
Knowledge Graph CheckerFree Knowledge Graph checker: look up a brand or name …
Keyword Gap CheckerFree keyword gap analysis: keywords a competitor ranks for and …
Competitor Top PagesFree competitor top pages tool: a domain's leading pages by …
Questions
What do people ask about the robots.txt tester?
People ask what the robots.txt tester checks, what robots.txt is, whether an empty Disallow blocks the whole site, how Allow differs from Disallow, which HTTP status the file should return, whether it should list a sitemap, whether it can protect private content and whether the checker simulates how one named crawler reads the file.
What does the Robots.txt Checker check?
It fetches your /robots.txt file, reports the HTTP status, and lists crawl rules grouped by user-agent, counting every Allow and Disallow directive plus any declared Sitemap lines so you see exactly what crawlers are told.
What is robots.txt?
Robots.txt is a plain-text file at your domain root that follows the Robots Exclusion Protocol (RFC 9309). It tells cooperating crawlers like Googlebot which paths they may or may not request, grouped by user-agent.
Does an empty Disallow rule block the whole site?
No. A line containing Disallow: with no path allows all paths for that group. Disallow: / is the rule that blocks the root and every path beneath it.
Does the checker decide whether one named crawler may fetch a URL?
No. It lists the groups and directives captured in the file. It does not reproduce a named crawler's complete matching, caching or policy behavior for a submitted URL.
Can robots.txt protect private content?
No. The file is public and its rules are voluntary. Protect private paths with authentication and authorization; use robots.txt only to communicate crawl preferences to cooperating crawlers.
What is a good HTTP status for robots.txt?
A 200 OK means the file was served and its rules apply. A 404 means no file exists, so crawlers assume full access. Persistent 5xx errors can cause crawlers to pause crawling entirely.
What's the difference between Allow and Disallow?
Disallow lists paths crawlers should not request; Allow re-permits a sub-path inside a broader Disallow. The most specific matching rule wins, so Allow can carve exceptions out of a blocked directory.
Should robots.txt list my sitemap?
Yes. A Sitemap: directive with the full URL helps crawlers discover your XML sitemap independently of any submission. The checker reports every Sitemap line it finds so you can confirm it is declared.
Is robots.txt a security or access-control tool?
No. Robots.txt is a public, voluntary instruction for cooperating crawlers, not access control. Anyone can read it, and it cannot protect private content. Use authentication or server rules to actually restrict access.
Does this checker simulate exactly how Googlebot reads my file?
It parses and reports your directives as written, but it does not replicate any single crawler's full internal matching policy. Treat the grouped rules and counts as an accurate readout, not a crawler-specific simulation.