Generate a valid robots.txt, then test it before you ship.
Pick your rules with checkboxes, block AI crawlers in one click, and see the file validated line by line. No login, no upload, no guesswork. Everything runs right here in your browser.
Use * to apply the rules below to every crawler.
Generated robots.txt
User-agent: * Allow: /
Validator
1 warningValidating the generated file above. Paste an existing robots.txt to test it instead.
No Sitemap line found. Adding your sitemap URL helps search engines discover all your pages.
One file is easy. Catching the next one that breaks is not.
A single bad Disallow can quietly deindex a whole section of your site.
Doing this by hand for 300 pages? Murkuz watches your crawl and indexing signals in Google Search Console, catches the technical issues that suppress rankings, drafts the fix, pushes it to your CMS after you approve, and proves the recovery with real numbers.
What a robots.txt file actually does
A robots.txt file lives at the root of your domain and gives crawlers a set of instructions about which URLs they should and should not request. It is the first file most search engines look for when they visit your site. Used well, it keeps crawlers away from low-value or private areas such as carts, search results, and admin paths, so they spend their crawl budget on the pages you actually want ranked.
Disallow does not mean noindex
This is the mistake that trips up most people. A Disallow rule stops a crawler from fetching a page, but it does not remove that page from the search index. If other sites link to a blocked URL, Google can still list it, usually with no description. To truly keep a page out of search results, you must let it be crawled and add a noindex meta tag or an X-Robots-Tag header. Blocking a page in robots.txt actually prevents Google from seeing that noindex tag.
Common robots.txt rules explained
- User-agent: * applies the rules below to every crawler that reads robots.txt.
- Disallow: /cart/ tells crawlers not to request anything under the cart path.
- Disallow: /*?* blocks URLs that contain a query string, useful for faceted navigation and tracking parameters.
- Allow: / explicitly permits crawling, and can override a broader Disallow for a specific path.
- Sitemap: points crawlers to your full sitemap so they can discover every page you want indexed.
Should you block AI crawlers?
That is a business decision, not a technical one. Blocking crawlers like GPTBot, ClaudeBot, Google-Extended, and CCBot can stop your content from being used to train large language models or answer questions inside AI assistants. The trade-off is that you may also reduce your visibility inside AI answers, which is becoming its own source of traffic. This tool makes both choices one click apart so you can decide deliberately. Remember that robots.txt is voluntary: it only affects crawlers that choose to honor it.
Frequently asked questions
What is a robots.txt file?+
A robots.txt file is a plain text file at the root of your domain, for example example.com/robots.txt, that tells search engine crawlers which parts of your site they may or may not request. It is a request, not a lock. Well-behaved crawlers like Googlebot and Bingbot respect it, but it does not block malicious bots or hide a page from search on its own.
Does robots.txt stop a page from appearing in Google?+
No. Disallow only stops crawling, not indexing. A URL that is blocked in robots.txt can still appear in search results if other pages link to it, often with no description. To keep a page out of the index, allow crawling and use a noindex meta tag or an X-Robots-Tag header instead.
How do I block AI crawlers with robots.txt?+
Add a User-agent block for each AI crawler with Disallow: / underneath. This tool has a one-click option that adds groups for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, PerplexityBot and others. Note that this is voluntary: it only stops crawlers that choose to honor robots.txt.
Where does the robots.txt file go?+
It must live at the root of your domain and be reachable at yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored. Each subdomain needs its own file, so blog.example.com and example.com are treated separately.
Does Google respect Crawl-delay?+
No. Googlebot ignores the Crawl-delay directive. You control Google's crawl rate in Google Search Console instead. Crawl-delay is still honored by some other crawlers such as Bingbot and Yandex, so this tool keeps it available.
Is this robots.txt generator free?+
Yes. It is completely free, requires no login, and runs entirely in your browser. Nothing you type is uploaded. Generate your file, validate the syntax, copy it or download it, and drop it at the root of your site.
Write one robots.txt here. Let Murkuz watch the whole site.
Doing this by hand for 300 pages? Murkuz detects the crawl and indexing issues that quietly suppress rankings, drafts the fix, publishes it to your CMS, and proves the recovery. Autonomously, on repeat.
Free tool. No login. Murkuz is the autonomous version, running across your whole site.