How AI answers choose their sources

Your question is not the question that gets searched

When someone asks an assistant for the best project management tool for a small agency, the system does not run that sentence as a search. It decomposes it.

It might search for project management tools for agencies. Then for pricing at small team sizes. Then for time tracking and client billing, because those are the features that distinguish agency use from general use. Then for reviews of the two or three specific products it already associates with the category. Then, if the conversation gave it more to work with, for whatever constraint the person mentioned three messages ago.

Google names this behavior directly and confirms it applies to both AI Overviews and AI Mode. Reporting on the feature puts the number of background searches for one complex question somewhere between eight and twenty. Google does not publish the exact figure and you should treat any specific number you see as an estimate, but the shape is not in dispute: one question in, many searches out, one answer back.

What fan-out does to keyword thinking

You cannot target fan-out queries directly, because you never see them. There is no report, no console, no export. What you can do is cover the neighborhood of sub-questions a reasonable system would generate.

For a product in a considered-purchase category, that neighborhood is fairly predictable:

• The category question. What is this kind of tool, and who is it for?
• The pricing question. What does it cost, at what size, with what excluded?
• The comparison question. How does it differ from the two obvious alternatives?
• The integration question. Does it work with the things I already run?
• The constraint question. Does it handle my specific situation, meaning my industry, my team size, my compliance requirement?
• The disqualifying question. Who is this not for? Assistants use this constantly, because a good recommendation includes a caveat.
• The migration question. What happens if I am already using something else?

Coverage of that neighborhood beats optimization of the headline term. A site that ranks beautifully for "project management software" and has nothing addressing agency client billing will lose the agency answer to a weaker site that wrote one honest page about billing.

The common failure is not bad content. It is content pitched entirely at the top of the funnel, where the model needs no help, and missing at the specific level, where it is actually looking.

Passages get retrieved, not pages

Retrieval systems break documents into chunks and match those chunks against the sub-questions, usually by comparing numerical representations of meaning rather than matching keywords. The practical consequence is unintuitive if you are used to thinking in pages: the model may never see your page. It sees a section of it, lifted out of context, sitting alongside sections from four other sites.

So test every section by reading it alone.

A paragraph opening with "As we saw above, this makes it the obvious choice" fails, because there is no above. A section headed "Pricing" that opens with the actual pricing, stated plainly, with the product named in the sentence, passes. It is a low bar, and most published marketing content does not clear it.

Four habits carry a section through retrieval:

• Name the subject in the sentence rather than relying on a pronoun that points back at the H1.
• Put the answer first and the reasoning after it. Chunk boundaries fall wherever they fall; you want the useful part near the top.
• Keep one idea per section, so the chunk boundary and the idea boundary line up.
• Put numbers, dates, currencies, and units in the text, not only in an image, a chart, or a table cell that loses its header.

Why specifics get reused

The KDD 2024 paper on generative engine optimization tested content modifications against generative answers across roughly ten thousand queries, using a benchmark built for the purpose. Adding relevant statistics, direct quotations, and citations to primary sources measurably increased how visible a source became inside the generated response, with the strongest methods improving the paper's position-adjusted word count metric by up to 41% over baseline.

A passage containing a concrete, attributable fact is better raw material than one containing an adjective. "Fast" cannot be quoted. "Median response time of 240ms across 12 regions in Q2 2026" can be, and it carries its own provenance when it is.

Factors such as length, uniqueness, and presentation of the cited website determine the true visibility of a citation.

Aggarwal et al., GEO: Generative Engine Optimization

Worth noting for when you start measuring: being cited is not binary. A source that contributes one clause and a source that supplies the whole answer both count as "citations," and treating them identically hides the difference.

The crawlers, and the mistake people make with them

AI companies run several crawlers with different jobs, and blocking the wrong one removes you from results you wanted to be in. It is a common mistake: the 2024 wave of AI opt-outs left a lot of robots.txt files with blunt rules in them.

OpenAI's setup is the clearest, because the documentation is explicit.

User agent	What it does	What blocking it costs you
OAI-SearchBot	Builds the index behind ChatGPT search	Your site stops appearing in ChatGPT search answers
GPTBot	Collects content for model training	Content is excluded from training, with no stated effect on search visibility
ChatGPT-User	Fetches a page because a person asked for it in a conversation	Not used to decide what appears in search; robots rules may not apply to these requests

You can allow OAI-SearchBot while disallowing GPTBot. Two separate decisions, two separate consequences.

The common own goal. A team decides it does not want its content training a model. Someone adds a broad block for anything that looks like an AI bot. Six months later the brand has quietly removed itself from ChatGPT search results, and the team is buying a visibility tool to find out why it is missing. If you want the training opt-out without the visibility cost, disallow GPTBot and allow OAI-SearchBot.

The other engines draw the line differently

Do not copy the OpenAI pattern across. Each platform separates things differently.

• Google. AI Overviews and AI Mode are part of Search, so ordinary Googlebot access governs them. The controls are the ones you already have: noindex to stay out entirely, nosnippet or data-nosnippet to keep specific text out of summaries, max-snippet to cap how much is used. Google-Extended is a separate opt-out that governs training and Gemini app grounding, and it does not affect appearance in Search. There is no way to appear in Google Search while opting out of AI Overviews specifically, and Google has been clear that this is deliberate. AI is built into Search rather than bolted onto it.
• Anthropic. ClaudeBot handles crawling for training, with separate agents for user-initiated fetches.
• Perplexity. PerplexityBot handles indexing for search results, with Perplexity-User covering user-initiated visits.
• Microsoft. Copilot's surfaces lean on bingbot and the Bing index, so Bing Webmaster Tools remains the place to check what is actually indexed.

Open your own robots.txt now. For each AI rule in it, ask what it was for and whether the trade it makes is the trade you would make today.

A file that opts out of training while staying visible in AI search looks like this:

# Out of model training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Still findable in AI search
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

The version that quietly costs you visibility usually looks like User-agent: *AI* or a long block list that swept up OAI-SearchBot alongside the training crawlers.

If the content is not in the HTML, assume it is not there

This is the highest-leverage technical fact in AEO, and the most often skipped.

Vercel measured AI crawler behavior across its network and found that the OpenAI and Anthropic crawlers fetch JavaScript files but do not execute them. Across hundreds of millions of GPTBot requests they saw no evidence of JavaScript execution at all. Googlebot does render JavaScript, and Gemini inherits that infrastructure, so a client-rendered page can still make it into AI Overviews. Applebot renders as well.

But if your product descriptions, pricing tables, or comparison content only appear after a client-side fetch, they are invisible to a large share of the engines that would otherwise cite them. Not ranked lower. Absent.

The five-minute version of this check

1. Open your most important commercial page and view source. That means the raw HTML, not the inspector's rendered DOM, which shows you the page after JavaScript has run.
2. Search that source for a sentence you actually care about: your price, your differentiator, the integration name.
3. Do the same with curl or any fetch tool, so nothing in a browser is helping.
4. Repeat for pricing, comparison, integration, and documentation pages. Those are the ones commercial questions reach for.
5. If the sentence is missing, server-render it or include it in the initial HTML payload. This is a development ticket, not a content project.

While you are there, check three related things:

• No consent wall or interstitial stands in front of the text. A crawler will never dismiss one.
• The page is reachable by an internal link, not only through search.
• Any structured data you publish agrees with the visible text rather than contradicting it.

The engines disagree, and the gap is wider than expected

AI search is not one destination with one set of rules. The disagreement starts with how much each engine reads before it answers.

38 sources against 3 is a different activity, not a rounding difference. An engine reading 38 documents has room for a specialist page answering one narrow sub-question well. An engine reading 3 has room for the obvious, canonical answer. Write for the end of that range your buyers actually use.

Then there is what they read. Perplexity is widely described as the Reddit-heavy engine. In our panel it was close to the opposite.

Reddit carried more than a quarter of ChatGPT's citations and between 0.2% and 4.3% of every other engine's. Wikipedia split the same way. A community-first strategy built on the assumption that it pays off across the board would have aimed four fifths of its effort at engines that barely look there.

Why this may contradict studies you have read. Panels differ, and the differences are large. Ours is 135 tracked questions across a specific mix of B2B software and local service brands; a different question mix produces a different source mix. Semrush's 13-week panel of 230,000+ prompts also found these shares moving sharply within a single quarter. Reddit's share of ChatGPT citations fell from around 60% to around 10% inside six weeks. The lesson is not which number is right. It is that citation mixes are unstable enough that inheriting somebody else's average is a bad substitute for measuring your own questions.

What to do with that instability

Measure before you commit budget. Find out where the engines look for your questions before you spend a quarter earning placements. A community strategy is excellent advice for a category ChatGPT answers from forums, and a poor use of a quarter when your buyers ask the engines that lean on communities an order of magnitude less.

Prefer durable work over platform bets. Anything depending on one platform's current preference is a rented position. Content that answers a real sub-question well, and accurate third-party descriptions of what you do, keep working when the mix shifts. Size platform bets as bets.

Citation is not the same as mention

This distinction shapes everything you measure later.

A citation is a link an engine attaches to its answer as evidence. A mention is your brand name appearing in the answer text. They overlap, but not reliably:

• Your page can be cited as evidence for a general claim without your brand being named at all.
• Your brand can be named from the model's own training knowledge, with no page of yours cited anywhere.
• A competitor's page can be cited while the answer recommends you, if the comparison happened to live on their site.

Track both. Mentions tell you about presence in the recommendation. Citations tell you which evidence the engine reached for, and therefore which of your pages, or which third-party pages about you, are doing the work.

Where PromptScout fits in this part

Every answer PromptScout records keeps its cited sources attached and grouped by type, which is how both charts above were produced. You can generate the same view for your own category instead of borrowing ours: if forums dominate one engine and vendor documentation dominates another, that shows up in your source breakdown within a couple of runs.

The source and citation analysis covers the breakdown itself, and coverage across engines covers running the same questions against all five.

What is next

Part 3 turns this into work: the technical prerequisites in order, how to write a passage that survives retrieval, which page types earn citations, where off-site evidence pays, and which widely sold tactics have no evidence behind them.