How AI answers choose their sources: Google, ChatGPT, Perplexity and Copilot

How an AI answer is built: retrieval, then generation

A common picture of AI search is a model that “knows” the web and recites it. That is not how cited answers work. When an answer shows sources, the system has run a search at the moment you asked. The usual name for this is retrieval-augmented generation, and in plain words it goes like this:

1. The question is rewritten. Your prompt becomes one or more search queries. A broad question is often split into several narrower ones.
2. Candidates are retrieved. Each query goes to a search index, and the system gets back a set of candidate pages, much like a results page.
3. Passages are selected. The system reads the candidates and picks the pieces that answer the question best: a definition, a number, a step, a comparison.
4. The answer is generated. A language model writes the reply from those passages.
5. Links are attached. The pages that supplied the passages are shown as citations, supporting links or sources.

The practical consequence is simple. The first half of the pipeline is search, so the same things that decide whether a page shows up in search decide whether it is even a candidate: crawl access, indexing, relevance to the query, quality. Only the second half is new. That is why the people who build these systems keep telling site owners that SEO still applies, and why generative engine optimization (GEO) is best understood as an industry label for SEO plus writing passages that can be lifted out cleanly. No provider uses the term in its documentation or publishes a GEO factor list.

Fan-out has a second consequence. Because a broad question is split into narrower searches, a page can be cited for a big question by answering one small part of it well. A page at https://example.com/crm/pricing/ that states prices clearly can be a source for "best CRM for small teams" without being the best general guide to CRMs.

What Google officially says

Google's page AI features and your website covers AI Overviews and AI Mode. Its message is that there is nothing separate to do:

Note the two conditions: indexed and eligible to be shown with a snippet. A page with nosnippet is indexed but does not meet the second condition. And eligibility is not a promise: “Indexing and serving isn't guaranteed.”

On how answers are assembled, Google describes the fan-out step from the diagram above in its own words: “Both AI Overviews and AI Mode may use a "query fan-out" technique — issuing multiple related searches across subtopics and data sources — to develop a response.” It also warns that the two features do not cite the same pages: “AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary.” And not every query gets an AI Overview: “AI Overviews are only shown when our systems determine that it is additive to classic Search, and as such, often don't trigger.”

On files and markup, Google is explicit: “You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add.”

The fundamentals Google lists as still worthwhile include “Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure”, “Making your content easily findable through internal links” and “Making sure that important content is available in textual form”, plus structured data that “matches the visible text on the page”.

Controls are the ordinary Search controls. “AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search.” To limit what is shown, “use nosnippet, data-nosnippet, max-snippet, or noindex controls.” Google-Extended is a separate token: “To limit AI training and grounding in some of Google's other systems, read more about Google-Extended.” Blocking it does not take you out of AI Overviews or AI Mode. Traffic from both features is counted in the Search Console Performance report “within the "Web" search type”, together with classic results. The surfaces themselves are covered in the lesson Classic results, AI Overviews, AI Mode, Discover.

ChatGPT search: OAI-SearchBot

OpenAI runs three separate agents, and only one of them decides whether you can appear in ChatGPT search:

• OAI-SearchBot — “OAI-SearchBot is used to surface websites in search results in ChatGPT's search features.”
• GPTBot — crawls content that may be used to train OpenAI's foundation models. It has nothing to do with search visibility.
• ChatGPT-User — visits a page when a user's request needs it. “ChatGPT-User is not used to determine whether content may appear in Search.” OpenAI notes that “Because these actions are initiated by a user, robots.txt rules may not apply.”

The settings are independent: “Each setting is independent of the others”, so “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models.” Changes are not instant: “For search results, please note it can take ~24 hours from a site's robots.txt update for our systems to adjust.”

OpenAI's help article on searching the web adds two points about where sources come from. First, ChatGPT does not rely on one index of its own alone: “To provide relevant responses to your questions, ChatGPT search sometimes partners with other search providers.” Second, it rewrites the question, just as in the pipeline above: “ChatGPT search typically rewrites your query into one or more targeted queries that it sends those providers.” OpenAI does not list the partners in that article, which is one reason SEOs commonly make sure a site is well indexed in more than one search engine, Bing included.

On ranking, OpenAI says only this: “ChatGPT ranks search results using multiple factors intended to help users find relevant, reliable information. Placement is not guaranteed.” And on eligibility: “To make a website eligible for inclusion, allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses.”

A typical setup that stays in AI search while opting out of training looks like this. The full syntax and the list of tokens are in the lesson robots.txt and AI crawlers:

# Stay in AI search answers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
Disallow: /

Perplexity: PerplexityBot and Perplexity-User

Perplexity's crawler documentation follows the same split between a search crawler and a user-triggered fetcher.

The second agent, Perplexity-User, fetches pages while answering: “When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.” Because it acts on a user's request, Perplexity states: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” As with OpenAI, “Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.”

For you this means PerplexityBot is the agent that decides whether your pages are in Perplexity's search results, and therefore in its pool of citable sources. Blocking it, or letting a firewall challenge it, removes you from that pool.

Microsoft Copilot and Bing

Copilot's answers draw on Bing. Microsoft Advertising states it directly: “Powered by Bing's search index, experiences like Microsoft Copilot, Microsoft Start, and others handle billions of queries each month”. There is no separate Copilot crawler to allow. Being crawled by Bingbot and indexed in Bing is the entry ticket, which in practice means a verified site in Bing Webmaster Tools, a submitted XML sitemap and, if your platform supports it, IndexNow for fast updates.

Microsoft's guidance, Optimizing Your Content for Inclusion in AI Search Answers, is the most detailed of the four on the generation half of the pipeline. It starts from fundamentals: “Crawlability, metadata, internal linking, and backlinks remain essential for ensuring your content is discoverable.” Then it explains what changes:

The same article recommends “Self-contained phrasing: Sentences that make sense even when pulled out of context”, warns “Don't hide important answers in tabs or expandable menus”, because “AI systems may not render hidden content, so key details can be skipped”, and is honest about the limits: “While there's no secret strategy for being selected by AI systems, success starts with content that is fresh, authoritative, structured, and semantically clear.”

Side by side: bots, index and official guidance

Put the four providers next to each other and the pattern is clear: each has one agent that controls search visibility, a separate control for training where one exists, and guidance that comes back to crawlability and indexing.

Eligibility checklist

Before any of the advice about writing matters, a page has to be in the candidate pool. These are the conditions, most of them straight from the providers' own pages:

Two checks are worth doing in your server logs rather than trusting the configuration. Look for requests from OAI-SearchBot, PerplexityBot and Bingbot and check which status codes they get. Many CDNs and security plugins block or challenge unfamiliar bots by default, and a firewall rule can undo a perfect robots.txt. Verify bots by the IP ranges the providers publish, since anyone can fake a user-agent string. Also keep in mind that many AI crawlers do not run JavaScript, so text that only appears after client-side rendering may be invisible to them even when Googlebot sees it.

What tends to get a page cited

Once you are eligible, selection is up to systems nobody documents in detail. What follows is practice that SEOs see work repeatedly, and that matches what Microsoft writes about the generation step. Treat it as raising your chances, not as a formula.

• Answer firstPut the direct answer in the first one or two sentences under a heading that matches the question, then explain. A passage that answers on its own is easy to lift.
• One idea per sectionClear H2 and H3 headings mark where an answer starts and ends. Long, mixed paragraphs are harder to cut into usable pieces.
• Concrete facts in textPrices, dimensions, dates, limits, steps and definitions stated plainly. "Starts at $12 per user per month" is citable; "affordable plans" is not.
• Something only you haveOriginal data, first-hand tests, your own photos and measurements. If ten pages say the same thing, any of them can be the source; if only you have the number, you are the source.
• Freshness you can seeAn accurate "last updated" date and content that is actually kept current, especially for prices, versions and rules.
• Brand mentions elsewhereFor "best X" questions, answers often lean on reviews, comparisons and lists on other sites. Consistent facts about your brand across your site, profiles and third-party pages help you be named, even when the link goes elsewhere.
• Coverage of the sub-questionsFan-out means the narrow follow-up queries matter. A page or section for each real sub-question (pricing, integrations, limits, alternatives) gives retrieval more to find.

Structured data is useful for the rich results it enables and for making facts machine-readable, and Microsoft recommends schema as part of structure. Just keep Google's line in mind: there is no special schema for AI features, and markup has to match the visible text.

Why the sources change from one query to the next

Do not judge your AI visibility from a single screenshot. Sources move for reasons that have nothing to do with your page:

• The wording matters. Each prompt is rewritten into its own set of searches, so two similar questions can retrieve different pages.
• Fan-out varies. The sub-queries are not fixed, so the candidate pool for the same question can change.
• Different products, different models. Google says its two AI features show varying links; ChatGPT, Perplexity and Copilot use different indexes on top of that.
• Context. Location, language, signed-in state and earlier turns in the conversation all feed into the rewritten query. OpenAI, for example, describes rewriting "near me" questions with an approximate location.
• Freshness. New pages enter the index and old ones are updated, so yesterday's source can be replaced today.

OpenAI is candid about quality too: “Search results and citations can be incomplete, outdated, or incorrect.” The only reliable picture comes from a fixed set of prompts tracked over time and from your own referral data, which is the subject of the lesson Measuring AI visibility.

Myths and facts

Common mistakes

• ✕One rule for all AI botsBlocking every AI agent to opt out of training also removes you from ChatGPT search and Perplexity. Block training agents, allow search agents.
• ✕Firewall blocks the search botsrobots.txt says yes, the CDN says 403. Check logs for each bot's status codes.
• ✕Ignoring BingCopilot draws on Bing's index. A site that is not indexed in Bing has no chance there.
• ✕nosnippet on pages you want citedGoogle requires snippet eligibility for supporting links in AI Overviews and AI Mode.
• ✕Key facts hiddenPrices in images, specs in PDFs, answers in tabs or loaded by JavaScript only.
• ✕Vague copyMarketing language with no numbers, dates or definitions gives an answer nothing to quote.
• ✕Buying “AI SEO” files and markupPaying for special files or schema that providers say are not needed.
• ✕Chasing one screenshotRewriting a page because one prompt cited a competitor yesterday.

FAQ

Sources

• Google Search Central — AI features and your website
• OpenAI — Overview of OpenAI crawlers
• OpenAI Help Center — Searching the web with ChatGPT
• Perplexity — Perplexity crawlers
• Microsoft Advertising — Optimizing Your Content for Inclusion in AI Search Answers

Quotes checked against the official pages on 30 October 2026.