Generate a valid robots.txt, then test it before you ship.
Pick your rules with checkboxes, block AI crawlers in one click, and see the file validated line by line. No login, no upload, no guesswork. Everything runs right here in your browser.
Use * to apply the rules below to every crawler.
Generated robots.txt
User-agent: * Allow: /
Validator
1 warningValidating the generated file above. Paste an existing robots.txt to test it instead.
No Sitemap line found. Adding your sitemap URL helps search engines discover all your pages.
Writing the file is easy. Seeing what Google did with it is not.
A single bad Disallow can quietly deindex a whole section of your site.
Murkuz puts your Google Search Console data inside Claude or ChatGPT, so the next question is one sentence instead of another export. Ask whether a URL is indexed, which sitemaps Google has on file, and when it last read them. Murkuz can submit a new sitemap for you. That is the only thing it writes.
Inspect_URLList_SitemapsSubmit_SitemapWhat a robots.txt file actually does
A robots.txt file lives at the root of your domain and gives crawlers a set of instructions about which URLs they should and should not request. It is the first file most search engines look for when they visit your site. Used well, it keeps crawlers away from low-value or private areas such as carts, search results, and admin paths, so they spend their crawl budget on the pages you actually want ranked.
Disallow does not mean noindex
This is the mistake that trips up most people. A Disallow rule stops a crawler from fetching a page, but it does not remove that page from the search index. If other sites link to a blocked URL, Google can still list it, usually with no description. To truly keep a page out of search results, you must let it be crawled and add a noindex meta tag or an X-Robots-Tag header. Blocking a page in robots.txt actually prevents Google from seeing that noindex tag.
Common robots.txt rules explained
- User-agent: * applies the rules below to every crawler that reads robots.txt.
- Disallow: /cart/ tells crawlers not to request anything under the cart path.
- Disallow: /*?* blocks URLs that contain a query string, useful for faceted navigation and tracking parameters.
- Allow: / explicitly permits crawling, and can override a broader Disallow for a specific path.
- Sitemap: points crawlers to your full sitemap so they can discover every page you want indexed.
Should you block AI crawlers?
That is a business decision, not a technical one. Blocking crawlers like GPTBot, ClaudeBot, Google-Extended, and CCBot can stop your content from being used to train large language models or answer questions inside AI assistants. The trade-off is that you may also reduce your visibility inside AI answers, which is becoming its own source of traffic. This tool makes both choices one click apart so you can decide deliberately. Remember that robots.txt is voluntary: it only affects crawlers that choose to honor it.
Frequently asked questions
What is a robots.txt file?+
A robots.txt file is a plain text file at the root of your domain, for example example.com/robots.txt, that tells search engine crawlers which parts of your site they may or may not request. It is a request, not a lock. Well-behaved crawlers like Googlebot and Bingbot respect it, but it does not block malicious bots or hide a page from search on its own.
Does robots.txt stop a page from appearing in Google?+
No. Disallow only stops crawling, not indexing. A URL that is blocked in robots.txt can still appear in search results if other pages link to it, often with no description. To keep a page out of the index, allow crawling and use a noindex meta tag or an X-Robots-Tag header instead.
How do I block AI crawlers with robots.txt?+
Add a User-agent block for each AI crawler with Disallow: / underneath. This tool has a one-click option that adds groups for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, PerplexityBot and others. Note that this is voluntary: it only stops crawlers that choose to honor robots.txt.
Where does the robots.txt file go?+
It must live at the root of your domain and be reachable at yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored. Each subdomain needs its own file, so blog.example.com and example.com are treated separately.
Does Google respect Crawl-delay?+
No. Googlebot ignores the Crawl-delay directive. You control Google's crawl rate in Google Search Console instead. Crawl-delay is still honored by some other crawlers such as Bingbot and Yandex, so this tool keeps it available.
Is this robots.txt generator free?+
Yes. It is completely free, requires no login, and runs entirely in your browser. Nothing you type is uploaded. Generate your file, validate the syntax, copy it or download it, and drop it at the root of your site.
The file is written. Now ask Google what it actually indexed.
This tool builds the file. Murkuz connects Google Search Console to Claude or ChatGPT, so you can check a URL against the index, see which sitemaps Google has on file, and submit a new one without opening another dashboard. Read-only over your data, with that one exception.
Free tool. No login. Murkuz has a permanent free plan and never asks for a card to start.