Run this tool instantly in your browser. No signup and no server-side processing.

Robots.txt Tester

Parse and test robots.txt rules against URLs to check crawler access.

robots.txt2 rule group(s) parsed

Parsed Rules

User-agent: *

Allow: /

Disallow: /admin/

Disallow: /private/

Disallow: /api/

Crawl-delay: 1

User-agent: crawler-01

Allow: /api/public/

Disallow: /api/

Disallow: /tmp/

Test URLs

Allowed - Allow: /api/public/ (agent: crawler-01)

Blocked - Disallow: /api/ (agent: crawler-01)

Blocked - Disallow: /admin/ (agent: *)

Allowed - Allow: / (agent: *)

Related Tools

DNS Lookup

Run live DNS-over-HTTPS lookups and inspect real resolver answers.

User-Agent Parser

Decode raw User-Agent strings to identify browser, engine, OS, CPU, and device details.

HTTP Status Codes

Searchable reference for all HTTP status codes with descriptions.

URL Parser & Query Editor

Parse URLs into components and interactively edit query parameters.

Robots.txt Tester: complete usage guide

Test robots.txt rules against target URLs to verify crawler allow/deny behavior before SEO rollouts and production deployments.

What this tool does

It evaluates robots.txt directives for user-agents and paths so you can confirm crawl policy outcomes.

It helps detect accidental blocking of important pages before search visibility is impacted.

It supports SEO QA by making rule precedence and wildcard behavior easier to validate.

Typical use cases

Verify that canonical tool pages are crawlable before release.
Audit staging-to-production robots.txt changes during launch windows.
Test rule precedence when multiple user-agent blocks exist.
Document crawl policy decisions in SEO runbooks and review notes.

Input examples

Rule set

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Target URL

https://example.com/admin/help/robots-guide

Crawler agent

crawler-01

Output examples

Evaluation result

Allowed: matched Allow /admin/help/ over broader Disallow /admin/

Rule trace

Applied user-agent block: * ; winning directive: Allow /admin/help/

SEO note

Re-test after each robots.txt update and before cache propagation completes.

Common errors and fixes

Blocking critical pages with broad disallow

Add specific allow rules for required indexable paths.

Assuming robots controls indexing alone

Combine robots rules with meta robots and canonical strategy.

User-agent block not matching expected crawler

Verify exact agent precedence and fallback to wildcard block.

Forgetting to deploy updated robots file

Check production response and CDN cache invalidation status.

Security and privacy notes

Rule testing is local and should use non-sensitive URLs when possible.
Do not expose private staging domains in external screenshots.
Sanitize proprietary directory names before sharing diagnostics.

Step-by-step workflow

Start with a minimal input for Robots.txt Tester and verify baseline behavior first.
Make assumptions explicit: encoding, delimiter strategy, and timezone.
Change one variable at a time and compare output diffs.
Keep one verified output snapshot as a team reference.

Quality checklist before sharing output

Confirm Robots.txt Tester output is deterministic across repeated runs with identical input.
Check boundary inputs (empty values, long fields, invalid characters).
Remove sensitive data before sharing output externally.
Verify final rendering on both desktop and mobile viewports.