robots.txt and AI crawlers: syntax, matching rules and what to block

What robots.txt does and does not do

A Disallow rule tells a crawler “do not fetch this URL”. It does not tell Google “do not show this URL in search”. Those are two different instructions, and mixing them up is the root of most robots.txt problems.

If a page is blocked, Google cannot read its content. But if other pages link to it, Google can still index the bare URL and show it in results without a description. To keep a page out of search, you need noindex in a robots meta tag or an X-Robots-Tag HTTP header, and the page must stay crawlable so Google can see that rule.

robots.txt is also voluntary. Well-behaved crawlers follow it, but it is not access control: anyone, and any bot that ignores the file, can still open a blocked URL. Private content needs a password or other authentication.

What Google officially says

A few more rules from the same documentation, each a direct quote:

• “You must place the robots.txt file in the top-level directory of a site, on a supported protocol.”
• The rules “apply only to the host, protocol, and port number where the robots.txt file is hosted.”
• “Google enforces a robots.txt file size limit of 500 kibibytes (KiB).” Content after that limit is ignored.
• “Google generally caches the contents of robots.txt file for up to 24 hours”.

The robots meta tag documentation adds the point that matters most in practice: if robots.txt blocks a page, Google cannot see the noindex on it, so that noindex is ignored.

Where the file lives

The file must sit at the root of the host and be called exactly robots.txt: https://example.com/robots.txt. A file at https://example.com/pages/robots.txt is not read.

Its scope is narrow. Because rules apply only to the host, protocol and port where the file is served:

• https://example.com/robots.txt covers https://example.com/ only.
• Each subdomain needs its own file: https://blog.example.com/robots.txt. The file on example.com does not apply to blog.example.com.
• http:// and https://, or a non-standard port such as :8080, are separate scopes too. In practice you redirect http to https and maintain one file per host.

The file must be “a UTF-8 encoded plain text file”, and Google reads only the first 500 KiB. If yours is close to that size, it is almost always full of rules you can replace with a few wildcards.

Syntax: user-agent, disallow, allow, sitemap

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and is followed by the Disallow and Allow rules for those crawlers. Sitemap lines stand on their own and do not belong to any group.

# Rules for every crawler without a group of its own
User-agent: * 1
Disallow: /search 2
Allow: /search/help 3
Disallow: /*?sort=
Disallow: /cart/

# OpenAI's training crawler: a separate group
User-agent: GPTBot 4
Disallow: /

Sitemap: https://example.com/sitemap.xml 5

Fields Google understands

• user-agent: which crawler the group is for. * means all crawlers.
• disallow: a path the crawler should not fetch. An empty Disallow: blocks nothing.
• allow: a path the crawler may fetch, used to open up part of a blocked folder.
• sitemap: the full URL of an XML sitemap.

Everything else is ignored by Google. Crawl-delay is the common example: some other search engines read it, Google does not. A Noindex: line in robots.txt does nothing either.

Groups for several crawlers

Google's documentation says: “You can group together rules that apply to multiple user agents by repeating user-agent lines for each crawler.” So this is one valid group that applies to both crawlers:

User-agent: GPTBot
User-agent: CCBot
Disallow: /

The key rule to remember: a crawler obeys only the most specific group that matches its name. If you add a User-agent: Googlebot group, Googlebot stops reading the * group, so any rule you still want for Googlebot must be repeated inside its own group.

Wildcards and matching rules

Rules match the start of the URL path, and Google supports two special characters:

• “* designates 0 or more instances of any valid character.”
• “$ designates the end of the URL.”

Google also states that “The path value must start with / to designate the root and the value is case-sensitive.” /Admin/ and /admin/ are different paths.

When several rules in a group match the same URL, Google uses “the most specific rule based on the length of the rule path”. If rules conflict, “Google uses the least restrictive rule.” In practice: the longer path wins, and when two matching rules are equally specific, Allow beats Disallow.

Worked examples:

A missing file has the same effect as an empty rule: if /robots.txt returns 404, everything is allowed.

What to block and what never to block

Block URLs that waste crawling on pages nobody needs in search. Leave alone everything Google needs to render and understand the pages you want to rank.

A whole-site Disallow: / usually arrives by accident: it protects a staging site and is copied to production at launch. Check the live file every time a new version of the site goes out.

robots.txt vs noindex

The two tools answer different questions. robots.txt answers “may you fetch this URL?”. noindex answers “may you show this page in search?”.

Combining them on the same URL does not work. A blocked page is never fetched, so Google never sees its noindex, ignores it, and may keep the URL in the index without a snippet. If you want a page out of search, let Google crawl it with noindex in place.

AI crawlers: who is who

AI companies publish their own user-agent tokens, and you address them in robots.txt exactly like Googlebot. The important detail is that one company often runs several bots with different jobs: blocking a training crawler is not the same as leaving an AI search product.

Google: Googlebot and Google-Extended

Googlebot crawls for Google Search. Google-Extended is a robots.txt token that controls whether your content is used for Google's Gemini models. Google states that it does not affect inclusion or ranking in Google Search.

AI Overviews and AI Mode are part of Search and use Googlebot, so blocking Google-Extended does not remove your site from them. The only way out of those features via robots.txt would be to block Googlebot, which removes you from Search as a whole.

# Keep content out of Gemini models, stay in Google Search
User-agent: Google-Extended
Disallow: /

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI separates its bots by job. GPTBot is for model training, OAI-SearchBot for ChatGPT search results, and ChatGPT-User fetches pages when a user asks ChatGPT to open them. Blocking OAI-SearchBot can remove you from ChatGPT search answers, so most sites that want to opt out of training block only GPTBot:

# Opt out of training, stay visible in ChatGPT search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Anthropic: ClaudeBot

Anthropic's crawler uses the token ClaudeBot. Check Anthropic's documentation for what it collects and whether other Anthropic agents use separate tokens.

User-agent: ClaudeBot
Disallow: /

Perplexity: PerplexityBot

PerplexityBot indexes pages for Perplexity's answer engine. Blocking it can remove your pages from Perplexity answers, which is a traffic source for some sites, so decide deliberately.

Common Crawl: CCBot

CCBot builds Common Crawl's open web datasets, which are widely used for AI training. Blocking it keeps your pages out of future crawls. It cannot take them out of datasets that were already published.

How errors in robots.txt affect crawling

What Google does depends on the HTTP status code the file returns:

Two practical consequences. A server error on robots.txt can stop crawling of your whole site, so monitor it like any important page. And a 403 on robots.txt, often caused by a firewall or bot protection, does not block anything: Google reads it as “no rules”.

Because Google generally caches the file for up to 24 hours, a fix may take up to a day to be picked up.

Testing and monitoring

1. Open the file in a browser at https://example.com/robots.txt and check that it returns 200 and the content you expect, on every host and subdomain.
2. Check the robots.txt report in Search Console. It shows the versions Google fetched, when, and any errors or warnings it found.
3. Test key URLs with URL Inspection. It shows whether a URL is blocked by robots.txt. Test your main templates: home, category, product, article.
4. Watch the Page indexing report. The status Indexed, though blocked by robots.txt lists URLs Google indexed without crawling them. If you want them in search, unblock them. If you want them gone, unblock them and add noindex.
5. Compare robots.txt with your sitemap. A sitemap should not list URLs that robots.txt blocks. That is a contradictory signal.

Common mistakes

• ✕Disallow: / left over from stagingThe production site blocks every crawler. Check the live file after each launch.
• ✕Blocking CSS, JavaScript or imagesGoogle cannot render the page the way users see it.
• ✕Using robots.txt to remove pages from the indexA blocked URL can stay indexed. Use noindex on a crawlable page instead.
• ✕Noindex: inside robots.txtNot a supported field. Google ignores it.
• ✕Crawl-delay for GoogleNot supported by Google, so it has no effect on Googlebot.
• ✕Wrong letter case in pathsDisallow: /Admin/ does not block /admin/.
• ✕A Googlebot group that silently overrides *A crawler follows only its most specific group. Once a Googlebot group exists, rules in the * group no longer apply to Googlebot unless you repeat them.
• ✕Blocking an AI search bot by mistakeBlocking OAI-SearchBot or PerplexityBot when you only meant to opt out of training.

FAQ

Sources

• Google Search Central — How Google interprets the robots.txt specification
• Google Search Central — Robots meta tag, data-nosnippet and X-Robots-Tag specifications
• Google Search Central — Overview of Google crawlers and fetchers

Quotes checked against the official pages on 29 October 2026.