Free tool, no account required

Free Robots.txt Tester & Validator

Test and validate robots.txt rules, User-Agent directives, blocked paths and AI bot access policies for search engines and AI crawlers.

Also known as: robots.txt tester, validator

Evidence shown with sourceNo account requiredNo invented metrics

Run a real check above

Submit a public URL, domain, or keyword. Novaverb will show only the evidence this tool can actually retrieve or measure.

Evidence model

Know what the result proves

This check captures the root robots.txt file for the exact submitted hostname and reports its HTTP status, non-comment line count, user-agent values, Allow and Disallow counts, sitemap declarations and any literal Disallow: / rule. It does not simulate a named crawler against a specific URL.

1. Source

Live robots.txt fetch. The result identifies where its evidence came from.

2. Boundary

The checker fetches /robots.txt and reads its directives. robots.txt is a crawl directive, not access control or index removal; this tool does not simulate a specific crawler's policy.

3. Next action

Use the finding to verify a problem, then connect a workspace when you need history, monitoring, or site-wide analysis.

Robots.txt explained

Test your robots.txt without overclaiming what it does

robots.txt tells compliant crawlers which paths they may request. It is the first file most crawlers fetch, so a single wrong line can quietly keep a whole site out of search - or let private paths be crawled.

What to check

Location - the file must live at the domain root and return HTTP 200 as plain text.
User-agent groups - each group's Allow/Disallow rules apply only to the agents it names; an empty Disallow after User-agent: * can block everything.
Declared sitemaps - publish the sitemap's absolute URL here so crawlers can discover it.
Validation boundary - this page checks HTTP reachability and directive evidence; it is not a full RFC 9309 parser or a Search Console inspection.

Why it matters for SEO and AI discovery

A clear robots.txt reduces ambiguity for search and AI crawlers. Novaverb can use this signal as part of a broader crawl and technical evidence model, alongside indexability, canonicals and internal links.

Get the right answer

What to submit - and what to avoid

Submit the domain, or any URL on it; the host is resolved and robots.txt is fetched from the root, which is the only location the specification recognises. A file placed in a subdirectory is not a robots.txt and will be reported as missing, which is the same conclusion every real crawler reaches.

Use it like this

yourdomain.comAny address form works - we fetch /robots.txt at the host root for you. http or https, with or without www, a bare domain or a full path - we normalize it for you.

https://www.yourdomain.com/anythingEven a deep URL is fine; we resolve to the host and read its root robots.txt.

Avoid this

Expecting a subfolder robots.txt to countrobots.txt is only honored at the host root; a copy inside a subfolder is ignored by crawlers.

Public methodology

Exactly how this result is produced

The exact hostname is resolved to a public address and /robots.txt is fetched with safe redirect handling. The returned text is counted by directive and user-agent, while empty Disallow values remain permissive and only the literal root rule is flagged as a site-wide block.

We preserve the exact submitted hostname and fetch /robots.txt from its root with safe redirect handling.
We count User-agent, Allow, Disallow and Sitemap directives from the captured file.
We flag a literal Disallow: / root block and keep an empty Disallow value permissive; we do not simulate a named crawler's full policy.

Built on public standards

The international standards this check applies

RFC 9309 defines the Robots Exclusion Protocol syntax used to identify User-agent, Allow, Disallow and Sitemap lines. This summary does not claim to implement crawler-specific matching, caching or policy extensions beyond the fields it displays.

IETFRFC 9309

Robots Exclusion Protocol

Identifies captured User-agent, Allow, Disallow and Sitemap directives and treats an empty Disallow as permissive.

Read the specification

We list a standard only where this tool genuinely reads or measures against it. Where a signal is outside a live check, the result says so instead of implying coverage.

Common questions

Robots.txt Checker FAQ

What does the Robots.txt Checker check?

It fetches your /robots.txt file, reports the HTTP status, and lists crawl rules grouped by user-agent, counting every Allow and Disallow directive plus any declared Sitemap lines so you see exactly what crawlers are told.

What is robots.txt?

Robots.txt is a plain-text file at your domain root that follows the Robots Exclusion Protocol (RFC 9309). It tells cooperating crawlers like Googlebot which paths they may or may not request, grouped by user-agent.

Does an empty Disallow rule block the whole site?

No. A line containing Disallow: with no path allows all paths for that group. Disallow: / is the rule that blocks the root and every path beneath it.

Does the checker decide whether one named crawler may fetch a URL?

No. It lists the groups and directives captured in the file. It does not reproduce a named crawler's complete matching, caching or policy behavior for a submitted URL.

Can robots.txt protect private content?

No. The file is public and its rules are voluntary. Protect private paths with authentication and authorization; use robots.txt only to communicate crawl preferences to cooperating crawlers.

What is a good HTTP status for robots.txt?

A 200 OK means the file was served and its rules apply. A 404 means no file exists, so crawlers assume full access. Persistent 5xx errors can cause crawlers to pause crawling entirely.

What's the difference between Allow and Disallow?

Disallow lists paths crawlers should not request; Allow re-permits a sub-path inside a broader Disallow. The most specific matching rule wins, so Allow can carve exceptions out of a blocked directory.

Should robots.txt list my sitemap?

Yes. A Sitemap: directive with the full URL helps crawlers discover your XML sitemap independently of any submission. The checker reports every Sitemap line it finds so you can confirm it is declared.

Is robots.txt a security or access-control tool?

No. Robots.txt is a public, voluntary instruction for cooperating crawlers, not access control. Anyone can read it, and it cannot protect private content. Use authentication or server rules to actually restrict access.

Does this checker simulate exactly how Googlebot reads my file?

It parses and reports your directives as written, but it does not replicate any single crawler's full internal matching policy. Treat the grouped rules and counts as an accurate readout, not a crawler-specific simulation.

More free checks