Kiểm tra & Xác thực robots.txt miễn phí

It fetches your /robots.txt file, reports the HTTP status, and lists crawl rules grouped by user-agent, counting every Allow and Disallow directive plus any declared Sitemap lines so you see exactly what crawlers are told.

Robots.txt is a plain-text file at your domain root that follows the Robots Exclusion Protocol (RFC 9309). It tells cooperating crawlers like Googlebot which paths they may or may not request, grouped by user-agent.

No. A line containing Disallow: with no path allows all paths for that group. Disallow: / is the rule that blocks the root and every path beneath it.

No. It lists the groups and directives captured in the file. It does not reproduce a named crawler's complete matching, caching or policy behavior for a submitted URL.

No. The file is public and its rules are voluntary. Protect private paths with authentication and authorization; use robots.txt only to communicate crawl preferences to cooperating crawlers.

A 200 OK means the file was served and its rules apply. A 404 means no file exists, so crawlers assume full access. Persistent 5xx errors can cause crawlers to pause crawling entirely.

Disallow lists paths crawlers should not request; Allow re-permits a sub-path inside a broader Disallow. The most specific matching rule wins, so Allow can carve exceptions out of a blocked directory.

Yes. A Sitemap: directive with the full URL helps crawlers discover your XML sitemap independently of any submission. The checker reports every Sitemap line it finds so you can confirm it is declared.

No. Robots.txt is a public, voluntary instruction for cooperating crawlers, not access control. Anyone can read it, and it cannot protect private content. Use authentication or server rules to actually restrict access.

It parses and reports your directives as written, but it does not replicate any single crawler's full internal matching policy. Treat the grouped rules and counts as an accurate readout, not a crawler-specific simulation.