robots.txt is a plain-text file at the root of a domain (e.g., https://example.com/robots.txt) that tells search-engine crawlers which pages they may or may not visit. It uses simple User-agent + Allow/Disallow rules.

How does the URL tester work?

The tester picks the user-agent group that best matches the bot you specify, then applies Google's longest-match rule — if an Allow pattern is more specific than a Disallow, the URL is allowed. Empty Disallow means allow all.

What does "Disallow: /" mean?

Disallow: / for User-agent: * means the entire site is blocked from all crawlers. If unintentional, this is the biggest possible SEO mistake — the tool flags it prominently.

Do all crawlers respect robots.txt?

robots.txt is a voluntary standard. Major search engines (Google, Bing, DuckDuckGo, Yandex, Baidu) respect it. Malicious scrapers and some AI training crawlers may ignore it — use server-side auth or IP blocking for hard restrictions.

Robots.txt Analyzer & Tester — Parse Crawl Rules

About the Robots.txt Analyzer

The Robots.txt Analyzer fetches any domain's /robots.txt file, parses it into a structured tree of user-agent groups, allow/disallow rules, sitemap references, and crawl-delay directives, then lets you test whether any URL path is blocked for any specific bot. The parser follows the Google Robots Exclusion Protocol (RFC 9309) and Google's longest-match rule for allow/disallow precedence.

All parsing runs on the server (to avoid CORS restrictions when fetching arbitrary origins) but the URL-tester runs entirely in your browser once parsed — instant results, no round-trip.

What the analyzer reports

📊 Summary statistics

Total lines, directive count, group count, sitemap references, and warning count — at-a-glance health of the file.

🚨 Critical warnings

The most dangerous mistake — User-agent: * + Disallow: / — blocks the entire site from all crawlers. The tool tags any group in this state with a red "Blocked All" badge.

👥 User-agent groups

Each group is shown with its user-agents, then all its rules in file order (Allow, Disallow, Crawl-delay). Stacked user-agent lines are correctly merged into one group per Google's spec.

🗺️ Sitemaps

Every Sitemap: directive is extracted and linked. Sitemaps are global (they apply to the whole site, not to any specific user-agent group).

⚠️ Syntax warnings

Missing colons, unknown directives, rules that appear before any user-agent — all flagged with line numbers so you can find them fast.

The URL tester

Enter a URL path (e.g., /private/, /admin/settings, /blog/post-42) and pick a bot (Googlebot, Bingbot, DuckDuckBot, YandexBot, or the wildcard *). The tester finds the most specific matching user-agent group, then applies each rule in order:

Empty Disallow: allows everything (this is the default state before any rule).
Wildcards * match any sequence of characters.
$ at the end of a pattern anchors it to the end of the URL.
The longest matching pattern wins — so a specific Allow: /public/whitelist.html can override a broader Disallow: /public/.

The result shows which rule matched, on which line, and whether the bot is allowed or blocked.

Robots.txt best practices

Always add a Sitemap directive — Sitemap: https://example.com/sitemap.xml. It's the fastest way for crawlers to discover your URL list.
Never disallow CSS or JS — Google needs them to render your page correctly for ranking.
Don't use robots.txt to hide sensitive URLs — the file is public. Anyone can read your Disallow list. Use auth for real security.
Prefer noindex meta over Disallow for pages you want out of the index but that should still be crawlable (e.g., internal search results).
Test after every change — one stray Disallow: / under User-agent: * can de-index a site overnight.

Privacy & limits

Fetching runs server-side, SSRF-guarded (private IPs blocked), 10-second timeout, 1 MB response cap, rate-limited to 30 requests per minute per IP. Nothing is stored.

🤖 Robots.txt Analyzer & Tester