Does Disallow hide a page from Google?

No. It asks crawlers not to fetch the page, but a disallowed URL can still be indexed if other sites link to it — it just appears with no snippet, because Google was not allowed to read it. To keep a page out of the index, allow the crawl and serve a noindex directive.

Where does robots.txt have to live?

At the root of the host, as https://example.com/robots.txt. A file anywhere else is ignored, and every subdomain needs its own — a rule on example.com does not apply to blog.example.com.

How do I block AI training crawlers?

Add a group naming the ones you want to exclude — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot — with Disallow: /. This tool has that as a preset. It is a request rather than a control: a crawler that ignores the convention is unaffected.

robots.txt Generator

Compose crawler rules group by group and see the file as you build it.

Presets

Site

Group 1

User-agent

Comma-separated. One group can address several crawlers.

Disallow

One path per line.

Allow

One path per line.

Crawl-delayseconds between requests

Sitemaps

robots.txt

Valid

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?*sort=

Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?*sort=

Sitemap: https://example.com/sitemap.xml

Processed locally in your browser. The file is built in this tab. No URL is fetched and nothing is submitted.

Reference

What robots.txt actually does

It is a request, not a control. A crawler that respects the convention reads /robots.txt before fetching anything else and skips the paths you disallow. A crawler that does not respect it — or anyone with a browser — reads the page anyway.

Disallowing a path does not hide it. If other sites link to a blocked URL, search engines can still list it, showing the URL with no snippet because they were not allowed to fetch the content. To keep a page out of an index, allow the crawl and serve X-Robots-Tag: noindex or a meta robots tag.

How groups are matched

A crawler picks exactly one group: the one whose User-agent token most specifically matches its own name, falling back to *. It does not merge groups. That is why a second group naming an agent already covered above is dead weight — the tool warns when it spots one.

Within a group, Google and Bing apply the longest matching rule, so Allow: /admin/public/ wins over Disallow: /admin/. Simpler crawlers take the first match, so do not rely on that ordering for anything that matters.

Directives

Directive	Meaning
User-agent	Opens a group. One token per line; repeat the line to address several crawlers with the same rules.
Disallow	A path prefix not to crawl. An empty value means no restriction.
Allow	An exception inside a disallowed prefix. Honoured by Google and Bing.
Sitemap	An absolute URL. Independent of any group, and the most useful line in the file.
Crawl-delay	Seconds between requests. Bing and Yandex honour it; Google ignores it.
$ *	End-of-URL anchor and wildcard. Widely supported but not part of the original convention.

Where it has to live

Only at the root of a host: https://example.com/robots.txt. A file at /blog/robots.txt is ignored. Subdomains are separate hosts and need their own file, and so does each scheme if you serve both — though if you are serving plain http at all, that is the more pressing problem.

Questions

Does Disallow hide a page from Google?: No. It asks crawlers not to fetch the page, but a disallowed URL can still be indexed if other sites link to it — it just appears with no snippet, because Google was not allowed to read it. To keep a page out of the index, allow the crawl and serve a noindex directive.
Where does robots.txt have to live?: At the root of the host, as https://example.com/robots.txt. A file anywhere else is ignored, and every subdomain needs its own — a rule on example.com does not apply to blog.example.com.
How do I block AI training crawlers?: Add a group naming the ones you want to exclude — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot — with Disallow: /. This tool has that as a preset. It is a request rather than a control: a crawler that ignores the convention is unaffected.