robots.txt Checker
Fetch, parse and test any website's robots.txt — allow/disallow rules, crawl-delay, sitemaps and URL blocking for Googlebot & Bingbot.
About the robots.txt Checker
This tool retrieves the robots.txt file served at the root of any public website and parses it into structured directives grouped by User-agent. It reports Allow and Disallow rules, Crawl-delay, declared Sitemaps, and the full redirect chain the fetch followed. It also runs a small set of URL tests against the rules to show whether a given path is allowed for the wildcard group, Googlebot, and Bingbot.
What it shows
- HTTP status, final URL, byte count and redirect chain of the
robots.txtfetch. - Parsed groups: every
User-agentblock with its Allow, Disallow, Crawl-delay and Sitemap lines. - Sample URL tests using Google-style longest-match wins (with
*and$wildcards). - Warnings for malformed lines and directives outside a group.
How to use the robots.txt Checker
- Enter any domain or URL (e.g.
example.comorhttps://example.com/page). - Complete the Cloudflare Turnstile verification.
- Press Check robots.txt and read the parsed rules and test results.
Frequently asked questions
What is robots.txt?
A plain text file at the root of a website that tells search engine crawlers which paths they may or may not access, via Allow and Disallow rules grouped by User-agent.
Does this tool crawl the site?
No. It only fetches /robots.txt itself and tests sample paths against the parsed rules locally. It does not crawl the site's pages.
Which user-agents are tested?
Sample paths are tested against the wildcard * group, plus Googlebot and Bingbot groups when they exist.
How are Allow/Disallow conflicts resolved?
Following Google's rule: the longest matching rule wins. On a tie, Allow wins.
Is it free?
Yes, completely free. Cloudflare Turnstile is verified once per scan to prevent automated abuse.