How AI answers choose their sources

Your question is not the question that gets searched

When someone asks an assistant for the best project management tool for a small agency, it may search for narrower parts of the question as well as the original wording.

It might search for project management tools for agencies. Then for pricing at small team sizes. Then for time tracking and client billing, because those are the features that distinguish agency use from general use. Then for reviews of the two or three specific products it already associates with the category. Then, if the conversation gave it more to work with, for whatever constraint the person mentioned three messages ago.

Google describes query fan-out as a technique AI Overviews and AI Mode may use. It does not establish a fixed search count for every answer. The searches an interface exposes may also be incomplete.

What fan-out does to keyword thinking

Do not infer hidden queries from the final answer alone. Some interfaces expose search activity, but it may be incomplete and differs by product. Use observed buyer questions, available search evidence and the returned answers to identify relevant sub-questions.

For a product in a considered-purchase category, that neighborhood is fairly predictable:

• The category question. What is this kind of tool, and who is it for?
• The pricing question. What does it cost, at what size, with what excluded?
• The comparison question. How does it differ from the two obvious alternatives?
• The integration question. Does it work with the things I already run?
• The constraint question. Does it handle my specific situation, meaning my industry, my team size, my compliance requirement?
• The disqualifying question. Who is this not for? Assistants use this constantly, because a good recommendation includes a caveat.
• The migration question. What happens if I am already using something else?

Cover a sub-question when it adds a useful answer for your buyers. A page explaining agency billing may fit an agency question better than a general product overview, but that relevance is a hypothesis to test, not a guaranteed win over another site.

The common failure is not bad content. It is content pitched entirely at the top of the funnel, where the model needs no help, and missing at the specific level, where it is actually looking.

Passages get retrieved, not pages

Retrieval systems break documents into chunks and match those chunks against the sub-questions, usually by comparing numerical representations of meaning rather than matching keywords. The practical consequence is unintuitive if you are used to thinking in pages: the model may never see your page. It sees a section of it, lifted out of context, sitting alongside sections from four other sites.

So test every section by reading it alone.

A paragraph opening with "As we saw above, this makes it the obvious choice" fails, because there is no above. A section headed "Pricing" that opens with the actual pricing, stated plainly, with the product named in the sentence, passes. It is a low bar, and most published marketing content does not clear it.

Four habits carry a section through retrieval:

• Name the subject in the sentence rather than relying on a pronoun that points back at the H1.
• Put the answer first and the reasoning after it. Chunk boundaries fall wherever they fall; you want the useful part near the top.
• Keep one idea per section, so the chunk boundary and the idea boundary line up.
• Put numbers, dates, currencies, and units in the text, not only in an image, a chart, or a table cell that loses its header.

Why specifics get reused

The KDD 2024 paper on generative engine optimization tested content modifications against generative answers across roughly ten thousand queries, using a benchmark built for the purpose. Adding relevant statistics, direct quotations, and citations to primary sources measurably increased how visible a source became inside the generated response, with the strongest methods improving the paper's position-adjusted word count metric by up to 41% over baseline.

A passage containing a concrete, attributable fact is better raw material than one containing an adjective. "Fast" cannot be quoted. "Median response time of 240ms across 12 regions in Q2 2026" can be, and it carries its own provenance when it is.

Factors such as length, uniqueness, and presentation of the cited website determine the true visibility of a citation.

Aggarwal et al., GEO: Generative Engine Optimization

Worth noting for when you start measuring: being cited is not binary. A source that contributes one clause and a source that supplies the whole answer both count as "citations," and treating them identically hides the difference.

The crawlers, and the mistake people make with them

AI companies run several crawlers with different jobs, and blocking the wrong one removes you from results you wanted to be in. It is a common mistake: the 2024 wave of AI opt-outs left a lot of robots.txt files with blunt rules in them.

OpenAI's setup is the clearest, because the documentation is explicit.

User agent	What it does	What blocking it costs you
OAI-SearchBot	Crawls for ChatGPT search	Opted-out sites are excluded from search answers, but may still appear as navigational links
GPTBot	Collects content for model training	Content is excluded from training, with no stated effect on search visibility
ChatGPT-User	Fetches a page because a person asked for it in a conversation	Not used to decide what appears in search; robots rules may not apply to these requests

You can allow OAI-SearchBot while disallowing GPTBot. Two separate decisions, two separate consequences.

Check the scope of an opt-out. Blocking GPTBot and blocking OAI-SearchBot serve different purposes. OpenAI documents that sites opting out of OAI-SearchBot are excluded from search answers, with a navigational-link exception. Allowing the search bot preserves eligibility; it does not guarantee retrieval or recommendation.

The other engines draw the line differently

Do not copy the OpenAI pattern across. Each platform separates things differently.

• Google. AI Overviews and AI Mode are part of Search, so ordinary Googlebot access governs them. The controls are the ones you already have: noindex to stay out entirely, nosnippet or data-nosnippet to keep specific text out of summaries, max-snippet to cap how much is used. Google-Extended is a separate opt-out that governs training and Gemini app grounding, and it does not affect appearance in Search. There is no way to appear in Google Search while opting out of AI Overviews specifically, and Google has been clear that this is deliberate. AI is built into Search rather than bolted onto it.
• Anthropic. ClaudeBot handles crawling for training, with separate agents for user-initiated fetches.
• Perplexity. PerplexityBot handles indexing for search results, with Perplexity-User covering user-initiated visits.
• Microsoft. Copilot's surfaces lean on bingbot and the Bing index, so Bing Webmaster Tools remains the place to check what is actually indexed.

Open your own robots.txt now. For each AI rule in it, ask what it was for and whether the trade it makes is the trade you would make today.

A file that opts out of training while staying visible in AI search looks like this:

# Out of model training
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Still findable in AI search
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

The version that quietly costs you visibility usually looks like User-agent: *AI* or a long block list that swept up OAI-SearchBot alongside the training crawlers.

Check what relevant crawlers can access

Important public facts should be accessible to the crawlers relevant to your buyers. Check their documented behavior and the page response before attributing a missing citation to JavaScript.

Vercel measured AI crawler behavior across its network and found that the OpenAI and Anthropic crawlers fetch JavaScript files but do not execute them. Across hundreds of millions of GPTBot requests they saw no evidence of JavaScript execution at all. Googlebot does render JavaScript, and Gemini inherits that infrastructure, so a client-rendered page can still make it into AI Overviews. Applebot renders as well.

A client-only price or product description may be unavailable to a crawler that does not execute JavaScript. Serving those facts in the initial HTML reduces that dependency. It does not guarantee that an engine will index or cite them.

The five-minute version of this check

1. Open your most important commercial page and view source. That means the raw HTML, not the inspector's rendered DOM, which shows you the page after JavaScript has run.
2. Search that source for a sentence you actually care about: your price, your differentiator, the integration name.
3. Do the same with curl or any fetch tool, so nothing in a browser is helping.
4. Repeat for pricing, comparison, integration, and documentation pages. Those are the ones commercial questions reach for.
5. If the sentence is missing, server-render it or include it in the initial HTML payload. This is a development ticket, not a content project.

While you are there, check three related things:

• No consent wall or interstitial stands in front of the text. A crawler will never dismiss one.
• The page is reachable by an internal link, not only through search.
• Any structured data you publish agrees with the visible text rather than contradicting it.

The engines disagree, and the gap is wider than expected

AI search is not one destination with one set of rules. Our panel recorded different numbers of source links attached to answers; those lists do not reveal every document retrieved or read.

The median was 38 attached source links for ChatGPT and 3 for Bing Copilot in this panel. That is an observable difference in returned evidence, not proof of how many documents either engine read. It does not establish how much room an engine has for a specialist page.

The domains in those returned source lists also differed. In this panel, Reddit accounted for a smaller share for Perplexity than for ChatGPT.

Reddit accounted for more than a quarter of ChatGPT's returned source links and between 0.2% and 4.3% of every other engine's. Wikipedia split the same way. These source shares do not measure the return on community work or establish how often an engine read a platform without returning its URL.

Why this may contradict studies you have read. Panels differ, and the differences are large. Ours is 135 tracked questions across a specific mix of B2B software and local service brands; a different question mix produces a different source mix. Semrush's 13-week panel of 230,000+ prompts also found these shares moving sharply within a single quarter. Reddit's share of ChatGPT citations fell from around 60% to around 10% inside six weeks. The lesson is not which number is right. It is that citation mixes are unstable enough that inheriting somebody else's average is a bad substitute for measuring your own questions.

What to do with that instability

Inspect evidence before committing budget. Review returned sources for your own questions and decide whether useful participation fits the community and your buyers. Test a scoped contribution with a defined outcome; source share alone does not forecast its return.

Prefer durable work over platform bets. Anything depending on one platform's current preference is a rented position. Content that answers a real sub-question well, and accurate third-party descriptions of what you do, keep working when the mix shifts. Size platform bets as bets.

Citation is not the same as mention

This distinction shapes everything you measure later.

A citation is a link an engine attaches to its answer as evidence. A mention is your brand name appearing in the answer text. They overlap, but not reliably:

• Your page can be cited as evidence for a general claim without your brand being named at all.
• Your brand can be named from the model's own training knowledge, with no page of yours cited anywhere.
• A competitor's page can be cited while the answer recommends you, if the comparison happened to live on their site.

Track both. Mentions tell you about presence in the recommendation. Citations tell you which evidence the engine reached for, and therefore which of your pages, or which third-party pages about you, are doing the work.

Where PromptScout fits in this part

PromptScout records returned source links where available and groups them by domain, provider and page type. Those returned links underpin the source charts above; they are not a separate measurement of displayed citations. Inspect the source breakdown for your own question set and keep the original answer alongside it.

The source and citation analysis covers the breakdown itself, and coverage across engines covers running the same questions against all five.

What is next

Part 3 turns this into work: the technical prerequisites in order, how to write a passage that survives retrieval, which page types earn citations, where off-site evidence pays, and which widely sold tactics have no evidence behind them.