Your files never leave your device. All processing happens locally in your browser.
How do I create a robots.txt file for my website?
Open AnyTool’s Robots.txt Generator and build groups: each group targets one or more User-agents (Googlebot, Bingbot, or “*” for all) and lists the paths they may not (Disallow) and may (Allow) access. Add absolute Sitemap URLs and an optional Host, or pick a preset — Allow all, Block all, Block AI crawlers, WordPress, or Block cart/admin. The file is assembled live with proper grouping, a built-in linter flags common mistakes, and you can copy it or download robots.txt. Place the file at your domain root (https://yoursite.com/robots.txt). Everything runs in your browser, nothing uploaded.
Build any number of groups: User-agent(s) + Allow/Disallow paths + optional Crawl-delay
Presets: Allow all, Block all, Block AI crawlers, WordPress, Block cart/admin
Block-AI buttons use a verified June-2026 token list (GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider…)
Live validation: missing leading “/”, accidental Disallow: /, Allow/Disallow conflicts, relative sitemaps, Crawl-delay misuse
100% client-side — copy or download robots.txt; nothing is uploaded
What is
robots.txt
robots.txt is a plain-text file placed at the root of a website (https://example.com/robots.txt) that tells web crawlers which paths they may or may not request, following the Robots Exclusion Protocol (RFC 9309). It is organised into groups, each starting with one or more “User-agent:” lines followed by “Allow:” and “Disallow:” rules, with site-wide “Sitemap:” lines. It is a voluntary directive that well-behaved crawlers honor — it is publicly visible and does not enforce access control, so it must never be used to hide private URLs.
Each group starts with one or more “User-agent:” lines, then “Allow:” and “Disallow:” rules whose values are URL paths starting with “/”. “Sitemap:” lines use absolute URLs and go at the end.
A robots.txt file is a set of groups. Each group begins with at least one “User-agent:” line (a crawler token such as Googlebot, or “*” for all bots) and is followed by “Disallow:” and “Allow:” rules. Field names are case-insensitive but path values are case-sensitive and must start with “/”. Two wildcards are supported: “*” matches any run of characters and “$” anchors the end of the URL. An empty “Disallow:” value means “allow everything” for that group, and crawlers pick the single most specific (longest-matching) rule. “Sitemap:” and “Host:” are site-wide directives written once, conventionally after the groups, and Sitemap must be an absolute https URL. The generator assembles all of this for you with correct grouping and a blank line between groups.
Add a group for each AI training bot with “Disallow: /”. The generator’s Block-AI buttons add the verified 2026 tokens — GPTBot, ClaudeBot, CCBot, Google-Extended and more — in one click.
To ask AI training crawlers not to use your site, add a separate group per bot, for example “User-agent: GPTBot” then “Disallow: /”. The Robots.txt Generator includes a verified June-2026 list of AI training tokens — GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended, Applebot-Extended, Bytespider, Amazonbot, meta-externalagent and others — each added as its own fully-disallowed group, while AI-search crawlers that keep you citeable (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are kept separate so you can choose. Important: this only works for bots that actually honor robots.txt — malicious or undeclared scrapers can ignore it entirely.
No. robots.txt is a public, voluntary directive — it does not enforce or secure anything. Use authentication or a noindex response for private pages.
robots.txt is a polite request that well-behaved crawlers choose to follow; it has no enforcement power. The file is served publicly, so anyone can read it, and listing a path under Disallow actually advertises that the path exists. Malicious bots and scrapers can ignore the file completely, and a page blocked from crawling can still be indexed from external links — Google just cannot see a noindex tag it is not allowed to crawl. To truly keep a page out of search or away from visitors, protect it with authentication or return a noindex header, and only use robots.txt to manage crawl budget and politely steer compliant crawlers.
No. The robots.txt is built entirely in your browser. Nothing you type — user-agents, paths, sitemaps — is sent to a server, logged or stored.
AnyTool’s Robots.txt Generator runs completely client-side. Every group, path, sitemap and preset updates the assembled file and the live validation locally, with no server round-trip, so your crawler rules never leave your device. No AI-bot list or URL is ever requested by the tool. The page works offline once cached, and closing the tab discards your draft. You simply copy the result or download robots.txt and upload it to your own site root.
Detailed Explanation
⚙️Methodology
How the Robots.txt Generator Builds and Groups the File
AnyTool’s Robots.txt Generator models a robots.txt file (the Robots Exclusion Protocol, RFC 9309) as a set of groups plus site-wide directives, and a pure, clock-free engine assembles the text deterministically. Each group carries one or more User-agent tokens and lists of Disallow and Allow paths plus an optional non-standard Crawl-delay; the engine writes every User-agent line, then the Allow rules, then the Disallow rules (emitting an empty “Disallow:” when a group should allow everything), separating groups with a blank line and appending Host and absolute Sitemap lines at the end. Path values are URL paths beginning with “/”, with the two robots.txt wildcards supported — “*” matches any run of characters and “$” anchors the end of the URL — and user-agent-specific groups are kept separate from the global “*” group, exactly as crawlers treat them.
An empty Disallow value is emitted to mean “allow everything” for that group
Wildcards supported: “*” (any characters) and “$” (end-of-URL anchor)
Host and absolute Sitemap directives are written site-wide, after the groups
All assembly is pure client-side JavaScript — no server, no CDN, no fetch
🔧Technical Details
Presets and the Verified 2026 AI-Crawler Block List
One-click presets populate correctly-grouped rules: Allow all, Block all (Disallow: / for “*”), WordPress (admin blocked, admin-ajax allowed), Block cart/admin for e-commerce, and Block AI crawlers. The block-AI feature uses a token list verified in June 2026 and deliberately split by role: training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Amazonbot, meta-externalagent, FacebookBot, PetalBot) are the ones the preset disallows, each as its own “User-agent: token” / “Disallow: /” group; AI-search crawlers that keep a site citeable (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) are shown separately so authors can keep them. The 2026 reality the tool reflects is that “block all AI” is no longer wise — blocking training bots no longer blocks the search bots that drive AI visibility.
Presets: Allow all, Block all, Block AI crawlers, WordPress, Block cart/admin
AI tokens verified June 2026, split into training / AI-search / user-triggered roles
Block-AI preset disallows training bots only, leaving search engines untouched
Per-bot toggle buttons add or remove a single fully-disallowed group
Reflects 2026 guidance: blocking training crawlers does not block AI-search crawlers
⚠️Limitations
Live Validation and the Directive-Not-Security Reality
A built-in linter checks the model against the mistakes documented in the major SEO guides and flags them as errors, warnings or notes: Allow/Disallow values that do not start with “/”, an accidental “Disallow: /” that removes the whole site, the same path appearing in both Allow and Disallow, blocked CSS/JS or asset folders that break Google’s rendering, redundant “**” or a misplaced “$”, multiple un-mergeable “*” groups, a non-numeric or very high Crawl-delay (which Google ignores entirely), and Sitemap entries that are not absolute https URLs. Crucially, the page is honest that robots.txt is a polite, voluntary directive rather than access control: it must live at the domain root, it is publicly visible, well-behaved bots may honor it but malicious ones can ignore it, and listing a path under Disallow advertises rather than hides it — so private pages need authentication or noindex, not robots.txt.
