robots.txt: The Gatekeeper File

No, and this is the single most important thing to understand. Robots.txt controls crawling, not indexing. If you disallow a page, Googlebot will not read its content, but Google can still list the URL in search results if other pages link to it, often with no description because it was never crawled. To keep a page out of the index, you need a noindex rule instead, and Googlebot has to be allowed to crawl the page in order to see that rule. Blocking a page in robots.txt and expecting it to drop out of Google is one of the most common SEO mistakes.

Disallow is a robots.txt directive that asks a crawler not to fetch a path. Noindex is a signal, set either as a meta robots tag in the page's HTML head or as an X-Robots-Tag HTTP header, that tells Google not to show the page in search results. They solve different problems. Use Disallow to stop crawling of areas with no search value or to protect server resources. Use noindex to remove a page from search. They also conflict: if you Disallow a page, Google cannot crawl it to discover the noindex, so the two cancel out. Never block a page you are trying to noindex.

No. Google renders pages much like a browser, running JavaScript and applying CSS to see the final layout and content. If your robots.txt blocks those files, Google renders an incomplete version of the page and may miss navigation, content injected by scripts, or structured data. The result can be lower-quality indexing of pages that are otherwise fine. Allow all rendering resources, and if you are unsure, use the URL Inspection tool in Search Console to view the page exactly as Googlebot renders it.

The file must live at the root of each host, for example https://example.com/robots.txt, and it only applies to that exact host, protocol and port. A subdomain or an http versus https version needs its own file. It must be named robots.txt in lowercase, be plain text encoded as UTF-8, and stay within Google's 500 kibibyte limit, beyond which the rest is ignored. Rules are grouped by user-agent, and each group lists its Disallow and Allow paths.

You block them the same way you block any bot: a user-agent group with Disallow in robots.txt. The tokens differ by company, for example Google-Extended for Google's AI training, GPTBot for OpenAI, ClaudeBot for Anthropic and CCBot for Common Crawl. Blocking Google-Extended controls whether your content trains Google's AI models, and Google confirms it has no effect on how you rank in Search, because that is governed by the separate Googlebot. The key is to never block the search crawlers themselves, Googlebot and Bingbot, as that would remove you from search. Remember too that robots.txt is a request that well-behaved crawlers honour, not an enforced wall.

For most sites, crawl budget is not something you need to manage, and Google says so. It becomes relevant on large sites, roughly those with hundreds of thousands of URLs or sections that change very frequently. There, disallowing genuinely low-value paths such as faceted-navigation parameters, internal search results and infinite-scroll variants can stop crawlers wasting time on them. But blocking is only part of it. Returning clean status codes, avoiding long redirect chains, consolidating duplicate URLs and keeping sitemaps current all matter more to how efficiently Google crawls you.

Google generally caches robots.txt for up to 24 hours, so a change can take about a day to take full effect. If you have made an urgent fix, such as removing a rule that was accidentally blocking important pages, you do not have to wait: open the robots.txt report in Search Console to confirm Google can fetch and parse the new file, and the cached version will be refreshed. If Google cannot reach the file at all, it will use the last good version it has for a time before treating the site as fully allowed.