Your files never leave your device. All processing happens locally in your browser.
How do I test whether a URL is allowed or blocked by robots.txt without uploading it to a server?
Open AnyTool’s Robots.txt Tester, paste a site’s robots.txt (or load the sample), type a test URL (a path like /admin/ or a full URL) and pick a user-agent — Googlebot, Googlebot-Image, Bingbot, * or a custom token. A pure engine parses the file into groups and applies Google’s exact matching rules: it selects the most-specific (longest) user-agent group case-insensitively, falling back to *, then among that group’s rules the LONGEST matching path wins, with Allow beating Disallow on an equal-length tie, and supports the * (any run of characters) and $ (end-of-URL) wildcards. You get a big ALLOWED / BLOCKED verdict, the exact deciding rule (with its line and match length), the group used, and — when several rules match — how the winner was chosen. A syntax-validation panel flags rules before any user-agent, missing colons, unknown directives and non-absolute Sitemap URLs, and a parsed-groups view lists every user-agent, Allow/Disallow rule and sitemap. Everything runs in your browser — nothing is uploaded and no site is crawled. Remember: robots.txt controls crawling, not indexing — a Disallowed URL can still appear in search if it’s linked, so use a noindex tag to deindex; verify in Google Search Console.
Paste robots.txt + a test URL + a user-agent → ALLOWED / BLOCKED with the matched rule
Google’s real rules: most-specific user-agent group, longest path wins, Allow beats Disallow on a tie
Supports * (any run of chars) and $ (end-of-URL) wildcards; empty Disallow = allow, Disallow: / = block all
Syntax validation (rule before user-agent, missing colon, unknown directive, bad Sitemap) + parsed-groups view
A robots.txt tester decides whether a given crawler (user-agent) is permitted to crawl a given URL under a site’s robots.txt, which is a plain-text file at the site root that lists Allow and Disallow path rules grouped by User-agent. Under Google’s interpretation the crawler uses the group with the most specific matching user-agent (case-insensitive, with * as the default), and within that group the rule whose path pattern is the longest match wins — with Allow beating Disallow when two rules tie in length. The path patterns support two wildcards: * matches zero or more of any character and $ anchors the end of the URL. An empty Disallow value allows everything, Disallow: / blocks the whole site, and Sitemap and Crawl-delay are separate directives (Google ignores Crawl-delay). robots.txt governs crawling, not indexing.
No. robots.txt controls crawling, not indexing. A Disallowed URL can still be indexed and shown (usually without a snippet) if other pages link to it — use a noindex tag to keep it out.
robots.txt is a crawl directive, not an index directive. Disallowing a URL tells well-behaved crawlers not to fetch it, but Google can still index the URL — showing it in search results, typically without a description — when it discovers the link elsewhere, because it never crawled the page to see any noindex. To actually keep a page out of the index you must let it be crawlable and add a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header; if you Disallow the page as well, Google can’t crawl it to see the noindex, so the block backfires. robots.txt is also a public file and only well-behaved bots obey it, so never rely on it to hide private URLs — use authentication.
Google uses the most specific rule — the one whose path pattern is the longest match. If an Allow and a Disallow match with the same length, Allow wins.
For the crawler’s matched user-agent group, Google evaluates every Allow and Disallow rule whose path pattern matches the URL and applies the most specific one, measured by the number of characters in the rule path. A longer matching pattern beats a shorter one, so Allow: /admin/public/ (a longer match) overrides Disallow: /admin/ for /admin/public/help. When an Allow and a Disallow match with exactly the same length, the least-restrictive rule wins — Allow beats Disallow. If no rule matches the URL at all, it is allowed by default. This tester shows the deciding rule and, when several rules match, lists them longest-first so you can see exactly how the winner was chosen.
* matches zero or more of any character (including /), and $ anchors the end of the URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf.
Google (and Bing) support two special characters in robots.txt path patterns. An asterisk * matches zero or more of any character, including slashes, which is handy for query strings and dynamic segments — Disallow: /*?sessionid= blocks any URL containing that parameter. A dollar sign $ anchors the end of the URL, so Disallow: /*.pdf$ blocks URLs that end in .pdf but not /file.pdf?download=1. Without a trailing $, a rule is a prefix match: Disallow: /fish also blocks /fishheads and /fish/salmon. This tester implements both wildcards exactly, and its validator warns when $ appears anywhere but the end (where it would match a literal $) or when a redundant ** is used.
No — a browser can’t fetch another site’s robots.txt (CORS blocks it, and there’s no server proxy). Open https://yoursite.com/robots.txt and paste its contents in.
The tester is 100% client-side and privacy-first, so it never contacts a server on your behalf — and a browser can’t fetch another origin’s /robots.txt anyway, because the Same-Origin Policy / CORS blocks cross-origin requests and there is deliberately no server proxy. Instead you paste the robots.txt you want to test: open https://yoursite.com/robots.txt in a tab, copy the text, and drop it in (or load the built-in sample). Everything you paste stays on your device, which makes it safe to test staging or unreleased robots.txt files. Because the matching follows Google’s rules, other crawlers such as Bing or various AI bots may interpret some directives slightly differently, so confirm live behaviour in Google Search Console’s robots.txt report.
Detailed Explanation
📖How It Works
Robots.txt Tester — Allowed / Blocked with the Matched Rule
AnyTool’s Robots.txt Tester takes a robots.txt the user PASTES plus a test URL and a user-agent, and decides — entirely in the browser — whether that crawler may crawl that URL, following Google’s documented matching rules. It parses the file into user-agent groups (each a set of Allow/Disallow path rules), selects the group whose user-agent token is the most specific case-insensitive match (falling back to the * default group), then applies the rule whose path pattern is the LONGEST match, with Allow beating Disallow on an equal-length tie. The result is a clear ALLOWED / BLOCKED verdict with the exact deciding rule (its line and match length), the group used, and — when several rules match — a longest-first ranking that shows how the winner was chosen.
Inputs: a pasted robots.txt + a test URL (path or full URL) + a user-agent (Googlebot, Googlebot-Image, Bingbot, *, or custom)
Group selection is case-insensitive and most-specific; the * group is the fallback and loses to any specific match
Precedence: longest matching path wins; on an equal-length tie, Allow beats Disallow; no match = allowed by default
Empty Disallow value = allow everything; Disallow: / = block the whole site
Shows the deciding rule, the group applied, and all matching rules ranked longest-first
🔧Technical Details
Wildcards, Sitemaps, Crawl-delay and Syntax Validation
The engine implements the two wildcards Google supports: * matches zero or more of any character (including slashes), and $ anchors the end of the URL — so Disallow: /*.pdf$ blocks URLs ending in .pdf, while a rule without a trailing $ is a prefix match. Sitemap directives are parsed independently of any group and must be absolute URLs; Crawl-delay is surfaced but flagged as non-standard because Google ignores it (Bing and Yandex honour it). A live syntax validator flags the real mistakes: an Allow/Disallow rule before any User-agent line, a missing colon, an unknown directive, a non-absolute Sitemap URL, a misplaced $ (anywhere but the end), a redundant **, and duplicate * groups. A parsed-groups view lists every user-agent, Allow/Disallow rule and sitemap so the file’s structure is transparent.
* = any run of characters (incl. /), $ = end-of-URL; only these two are special
Sitemap is independent of groups and must be an absolute http(s) URL
Crawl-delay is shown but noted as ignored by Google (Bing/Yandex honour it)
Validation: rule-before-user-agent, missing colon, unknown directive, bad Sitemap, misplaced $, redundant **, duplicate * groups
Parsed-groups view highlights the group actually applied to your test
⚠️Limitations
Crawling vs Indexing, CORS and Privacy
The tool is explicit that robots.txt controls CRAWLING, not INDEXING: a Disallowed URL can still be indexed and shown in Google Search (usually without a snippet) if other pages link to it, because Google may index the URL without crawling the page. To keep a page out of the index you must let it be crawlable and add a noindex meta tag or an X-Robots-Tag: noindex header — Disallowing it prevents Google from ever seeing that noindex. Because it is 100% client-side, the tester cannot and does not fetch another site’s /robots.txt (the Same-Origin Policy / CORS blocks cross-origin requests and there is no server proxy), so the user pastes the file contents; nothing is uploaded and no site is crawled, which makes it safe for staging or unreleased robots.txt files. Matching follows Google’s interpretation, so other crawlers may read some rules differently — verify in Google Search Console’s robots.txt report.
robots.txt = crawl directive, NOT an index directive — use noindex (meta / X-Robots-Tag) to deindex
Never Disallow a page you also want de-indexed, or Google can’t crawl it to see the noindex
A browser can’t fetch another origin’s /robots.txt (CORS, no proxy) — the user pastes the contents
100% client-side: nothing uploaded, no crawl; safe for staging / unreleased files; works offline once cached
Follows Google’s rules — Bing, Yandex and AI bots may differ; confirm in Google Search Console
Robots.txt testing: in-browser (AnyTool) vs online / server-side testers
Explicit: robots.txt ≠ noindex; use noindex to deindex
Rarely explained
Cost / signup
Free, no signup
Often gated or ad-heavy
AnyTool tests a robots.txt you PASTE using Google’s matching rules, entirely in your browser, and uploads nothing — which is safe for staging or unreleased files. A browser cannot fetch another site’s /robots.txt for you (CORS blocks cross-origin requests and there is no server proxy), so you paste the contents. Matching follows Google’s interpretation; other crawlers (Bing, Yandex, AI bots) may read some rules differently, and robots.txt controls crawling, not indexing — verify with the live robots.txt report in Google Search Console. Comparison as of July 2026.
