AI crawlers and llms.txt: which bots to allow, block and verify

Three jobs AI bots do

"AI crawler" is not one thing. When you look at the official documentation of Google, OpenAI, Anthropic and Perplexity, their bots fall into three groups, and the right decision is different for each group.

• Training crawlers collect public pages that may be used to train future models. Examples: GPTBot, ClaudeBot and the Google-Extended token. Blocking them affects what goes into future models. It does not, on its own, remove you from AI answers.
• Search crawlers build the index that AI search answers draw from and link to. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot and, for Google, Googlebot itself. Blocking them costs you visibility and citations in those products.
• User-triggered fetchers load one page because a person asked an assistant to read it or to answer a question. Examples: ChatGPT-User, Claude-User, Perplexity-User, Google-Agent. Several providers say these may not follow robots.txt, because a human started the request.

The split matters because the two goals site owners usually have pull in different directions: you may want to be cited in AI answers but not want your content used for training. With separate tokens you can choose per job. How to write the rules themselves (groups, Disallow, matching) is covered in the lesson "robots.txt and AI crawlers"; this lesson is about strategy, verification and the infrastructure around the file.

What the providers officially say

These are direct quotes from each provider's own documentation.

Bot by bot: Google, OpenAI, Anthropic, Perplexity

Google: Googlebot, Google-Extended, Google-Agent

Google is the odd one out, because its AI features are part of Search. AI Overviews and AI Mode are fed by the normal Search index, which Googlebot builds. There is no separate "AI search" token for Google, and blocking Googlebot to keep content out of AI Overviews would take you out of Google Search altogether.

Google-Extended is different: it is not a crawler but a switch. In Google's words: “Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” Google describes it as “a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models”, and the same page says it also covers grounding in Gemini Apps and Grounding with Google Search on Vertex AI. You will never see Google-Extended in your logs, because no request carries that name.

If you want to limit what Google shows from your pages in AI features, the tools are the snippet controls, not a crawler block: “To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls.” They apply to the normal results too, which is covered in the lesson on robots meta tags.

Google-Agent is listed among Google's user-triggered fetchers: “Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.” For this whole group Google says: “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot

GPTBot is the training crawler. OAI-SearchBot is the search crawler: “OAI-SearchBot is used to surface websites in search results in ChatGPT's search features.” OpenAI recommends allowing it in robots.txt and allowing requests from its published IP ranges if you want to appear. When both are allowed, OpenAI notes: “If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling.” So seeing only one of them in your logs is not necessarily a sign of a problem.

ChatGPT-User visits a page when a ChatGPT user's request needs it. OpenAI says: “Because these actions are initiated by a user, robots.txt rules may not apply.” and “ChatGPT-User is not used to determine whether content may appear in Search.” Blocking it is therefore not a way to leave ChatGPT search; that is what the OAI-SearchBot rule is for. OAI-AdsBot “only visits pages submitted as ads”, so it matters only if you advertise in ChatGPT.

Timing: “For search results, please note it can take ~24 hours from a site's robots.txt update for our systems to adjust.”

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic documents three bots and says its bots honour robots.txt directives. ClaudeBot collects content that could be used for training. For the other two, Anthropic spells out the cost of blocking:

• “Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”
• “Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.”

Two practical details from the same page. Rules are per host: “Please do this for every subdomain that you wish to opt out from.” And Anthropic supports “the non-standard Crawl-delay extension to robots.txt” if you only want to slow its crawling down rather than stop it. Google ignores Crawl-delay, so this works only for bots that support it.

Perplexity: PerplexityBot, Perplexity-User

Perplexity documents no training crawler. PerplexityBot is for search, and Perplexity recommends allowing it and its published IP ranges if you want to appear in results. Perplexity-User fetches pages while answering a user's question; for it, “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Perplexity says changes may take up to 24 hours to be reflected.

The strategic choice: AI answers vs training

Behind every rule there are two separate questions:

1. Do you want to be found and cited in AI answers? Then the search crawlers and, usually, the user-triggered fetchers must be able to load your pages.
2. Do you accept that your content may be used to train models? That is controlled only by the training tokens.

Because the tokens are independent, the answers can be combined freely. Blocking training does not remove you from ChatGPT search, Claude, Perplexity or Google's AI features, as long as the search bots stay allowed. Blocking a search bot, on the other hand, has a direct cost in visibility in that product, and there is no ranking or citation benefit from blocking anything.

In practice the choice follows the business model:

• E-commerce, local businesses, service and B2B sites usually allow everything. Being named in an answer is a marketing channel, and their content has little value as training data on its own.
• Documentation and SaaS sites usually allow everything too, because developers ask assistants how to use the product. These are also the sites where llms.txt is most useful.
• News publishers and content businesses often block training tokens and keep search bots, so they still get links and visits from AI answers.
• Paywalled and licensed content keeps premium sections away from all AI bots and protects them with real authentication, because robots.txt is only a request.

Example policies for common goals

Four ready-made policies for example.com. One rule from the robots.txt lesson matters a lot here: a crawler obeys only the most specific group that names it. If you give GPTBot its own group, it ignores your User-agent: * group, so repeat any shared rules (such as /admin/) inside it. Anything you do not disallow is allowed by default, so "allow" groups are not needed.

1. Allow everything

# No AI-specific groups: every bot follows the * group
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

2. Cited in AI answers, not used for training

# Training tokens: opt out of the whole site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Search bots and user fetchers fall through to here
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Other training-only tokens, such as Common Crawl's CCBot, can be added to the first group. Check each provider's page first: a token you assume is "training only" may also feed a search product.

3. Google Search only, no other AI

# Every documented AI token except Googlebot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Disallow: /

# Googlebot, Bingbot and the rest
User-agent: *
Disallow: /admin/

This keeps you in Google Search, including AI Overviews and AI Mode, because those use Googlebot. Perplexity-User and other fetchers that generally ignore robots.txt are not covered by this file; if you really need them out, it takes a firewall rule. Think twice before choosing this policy: you give up citations and referral visits from ChatGPT, Claude and Perplexity.

4. A premium section kept away from AI bots

User-agent: Google-Extended
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /reports/
Disallow: /admin/

User-agent: *
Disallow: /admin/

robots.txt here is only the polite layer. The real protection for /reports/ is the server: content that must not leak should only be sent after a login or a paywall check, because user-triggered fetchers and badly behaved scrapers do not ask robots.txt first.

Verifying that a bot is real

Anyone can send a request with GPTBot or ClaudeBot in the user agent. In practice, some requests with AI bot names come from scrapers borrowing a respectable name. Before you allow, block or report anything, check that the request comes from the provider's network.

Google describes three identity signals: “Google's crawlers identify themselves in three ways:” the user-agent header, the source IP address and the reverse DNS hostname of that IP. For a manual check, look up the hostname of the IP, then look up the IP of that hostname. Google's rule for the first step: “Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.” The second lookup must return the original IP. For large volumes, match IPs against Google's published JSON range lists instead.

OpenAI, Anthropic and Perplexity publish JSON files of IP ranges in CIDR format: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, claude.com/crawling/bots.json, perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json. Perplexity's advice applies to all of them: “combine both User-Agent string matching and IP address verification”. Fetch the lists automatically (for example once a day), because ranges change.

Block fake bots at the firewall or server, not in robots.txt: a scraper that pretends to be GPTBot ignores robots.txt anyway. And do not try to opt out of a real provider by blocking its IPs. Anthropic warns that “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.”

CDN and WAF settings that silently block AI search bots

The most common reason a site with a perfectly open robots.txt never shows up in AI answers is not robots.txt at all. It is the CDN or web application firewall in front of the site. The bot reads robots.txt (often cached and served fine), then gets a 403, a challenge page or a 429 for every real page. Nobody notices, because humans in browsers pass the same checks.

Settings to check, in practice:

• Bot-fight or "super bot" modes that challenge anything automated. Perplexity notes: “Cloudflare's Bot Fight Mode can't be skipped with WAF custom rules.” If such a mode is on, an allow rule may not be enough.
• One-click "block AI bots" or "AI scrapers" toggles. Some CDNs offer managed AI-bot blocking and may suggest or enable it for new sites. It often blocks search bots together with training bots, which may not be what you decided.
• JavaScript or managed challenges on whole paths. Bots do not solve them.
• Rate limits tuned for humans. A search crawler that fetches many pages in a row hits them first.
• Country or ASN blocks that exclude the data-centre networks AI providers crawl from.
• Hosting-level security plugins with their own user-agent blacklists, outside the CDN.

The fix is an explicit allow rule for verified bots, matching both the user agent and the published IP ranges, placed before any block or challenge rule. Order matters: as Perplexity puts it for Cloudflare, “A Skip rule only bypasses rules that run after it.” Then confirm in your logs that verified AI bots receive 200 responses, not 403, 429 or 503. Decide your policy in robots.txt first, then make the firewall agree with it, not the other way round.

What llms.txt is

llms.txt is “A proposal to standardise on using an /llms.txt file to provide information to help agents use a website.” It was written by Jeremy Howard, first published on 3 September 2024, and the current text is version 2, last modified on 10 August 2026. The idea is a short Markdown file that gives an AI agent a curated map of your most useful content, so it does not have to dig through navigation, scripts and ads.

Location: “The file can be placed at the site root, or at any path within it, covering the pages under that path.” So https://example.com/llms.txt covers the whole site and https://example.com/docs/llms.txt covers only the docs. The structure, in this order:

1. An optional byte-order mark.
2. “An H1 with the name of the project or site. This is the only required section”.
3. A blockquote with a short summary.
4. Optional paragraphs or lists with more detail (no headings).
5. Optional H2 sections, each a list of links in the form [name](url), optionally followed by a colon and a note.
6. By convention, a section called Optional for secondary links an agent can skip when it is short on context.

The proposal also suggests offering clean Markdown versions of important pages at the same URL with .md added, such as /docs/setup.html.md, and linking to them from llms.txt.

# Example Garden Tools

> Online shop for hand tools and garden equipment, shipping in the EU.
> Prices include VAT. Support answers within one business day.

Orders can be returned within 30 days. Product pages list stock per warehouse.

## Help
- [Shipping and delivery](https://example.com/help/shipping.md): costs, carriers, countries
- [Returns](https://example.com/help/returns.md): how to start a return, refund timing
- [Warranty](https://example.com/help/warranty.md)

## Product guides
- [Choosing a pruning saw](https://example.com/guides/pruning-saws.md): sizes and blade types

## Optional
- [Company history](https://example.com/about.md)

Keep it small and curated. A sitemap lists every page for search engines; llms.txt is a short, hand-picked reading list. The proposal's own test is a good one: give an agent only your llms.txt and ask it questions about your business.

What llms.txt is not: its real status

llms.txt gets a lot of attention, so it helps to be exact about what it is.

• It is a proposal, not a standard. “The llms.txt specification is open for community input.” No standards body has adopted it.
• It is not access control or an opt-out. “robots.txt and llms.txt have different purposes.” Leaving a page out of llms.txt does not stop any bot from crawling it, and listing a page does not override a Disallow.
• It is not a ranking or citation factor that any provider has documented. None of the crawler documentation pages from Google, OpenAI, Anthropic or Perplexity used in this lesson mentions llms.txt as something their crawlers or answers read. Google's guidance for AI features is explicit: “You don't need to create new machine readable files, AI text files, or markup to appear in these features.”
• Publishing is not consuming. The proposal points out that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer documentation. That shows they offer the format for their docs; it is not a statement that their crawlers use other sites' files.

Where it does help is on demand: “llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user.” Coding agents and tools that a user points at a site can follow it. So it is a cheap, low-risk addition for documentation, SaaS and developer sites, and an optional extra for others. If you add one, many CMS and docs platforms can generate it; review the output so it lists only canonical, indexable pages.

Monitoring AI bots in your logs

Your server or CDN logs are the only place where you see what AI bots actually do on your site. Analytics tools do not record bots. A simple routine:

1. Segment by token. Filter requests by GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Agent and Googlebot. Remember that Google-Extended never appears.
2. Verify the IPs against the published lists, and report fake traffic separately.
3. Check status codes per bot. A verified search bot getting 403 or 429 is a firewall problem; many 404s point to stale links in the provider's index.
4. See which URLs they fetch and how often. Search bots should reach the pages you want cited; if they spend their time on filters or parameters, fix that as you would for Googlebot.
5. Watch for changes after a policy edit. Disallowed bots should stop fetching normal pages within a day or so; they will keep reading robots.txt.
6. Connect it to referrals. Visits from chatgpt.com, perplexity.ai and claude.ai in your analytics show whether AI visibility turns into traffic.

How to get the logs, parse them and run these checks at scale is covered in the lesson "Server log analysis".

Common mistakes

• ✕Blocking Googlebot to avoid AI OverviewsGoogle's AI features run on the Search crawl, so this removes the site from Google Search entirely.
• ✕Treating all AI bots as oneBlocking OAI-SearchBot or PerplexityBot "against training" costs visibility and does nothing for training, which they are not used for.
• ✕Thinking Google-Extended affects AI OverviewsIt controls Gemini training and grounding; Search, including AI features, is controlled by Googlebot.
• ✕A CDN toggle that overrides the policyrobots.txt allows search bots, but a "block AI bots" setting or a challenge rule returns 403 to them.
• ✕Named groups that drop the shared rulesA GPTBot group without /admin/ lets GPTBot into a section the * group closed.
• ✕Trusting the user agentAllowlists and reports based on the name alone include scrapers. Verify IPs.
• ✕Using llms.txt as an opt-outIt has no blocking effect. Use robots.txt for policy and authentication for private content.
• ✕Forgetting subdomainsEach host has its own robots.txt, so docs.example.com and shop.example.com need their own rules.

FAQ

Sources

• Google — Common crawlers (Googlebot, Google-Extended)
• Google — Overview of Google crawlers and fetchers
• Google — User-triggered fetchers
• Google — Verify requests from Google crawlers and fetchers
• Google Search Central — AI features and your website
• OpenAI — Overview of OpenAI crawlers
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity — Perplexity crawlers
• llmstxt.org — The /llms.txt file (proposal, v2)

Quotes checked against the official pages on 30 October 2026.