When ChatGPT names three companies in an answer, it is tempting to treat the first one as “ranked number one.” That interpretation imports the mental model of a search-engine results page into a different kind of system.
Traditional search positions are also dynamic, but a search result is built around a visible ordered results interface. A conversational answer is generated for a specific exchange. It may shortlist, compare, qualify, or recommend options in prose, and its wording can change across otherwise similar conversations.
The safer unit of analysis: one answer is one observation produced under a documented set of conditions.
AI recommendations are context-sensitive and variable
A recommendation outcome may change with:
- the buyer's stated need, constraints, or desired outcome;
- prompt wording and follow-up context;
- geography, language, and availability;
- the provider, product surface, visible model, and search setting;
- sign-in state, personalization, or temporary/private modes;
- changes to public evidence or the AI product over time;
- normal response-to-response variation.
This means two responsible studies can observe different outcomes if their collection conditions differ. It also means an isolated screenshot cannot establish a stable market-wide result.
Why repeated observations matter
Repetition helps describe a distribution rather than a single outcome. If a company is recommended in nine of ten independent trials under a frozen buyer context, that is a different signal from appearing once in ten trials—even though either pattern could produce a persuasive screenshot.
Repeated observations can help estimate:
- inclusion and recommendation rates;
- which competitors recur;
- how recommendation order varies;
- whether provider differences are material;
- whether an apparent gap is stable, mixed, or volatile.
Repetition does not remove every limitation. A small sample remains a small sample, and a larger sample collected under poorly defined conditions can still be misleading.
Recommendation order is evidence—not automatically a SERP position
Order can matter. A model may explicitly say one option is its top choice, present a numbered shortlist, or lead with a particular company. That should be preserved in the raw observation.
But analysts should distinguish explicit recommendation language from mere list position. A company appearing first in one answer should not automatically be labeled the permanent “#1 AI result.” Stronger reporting describes the exact outcome and how often it repeated under the study conditions.
| Avoid | Prefer |
|---|---|
| “Company A ranks #1 in ChatGPT.” | “Company A was the primary recommendation in 7 of 10 valid trials for this buyer context.” |
| “ChatGPT never recommends Company B.” | “Company B was not recommended in the 10 valid observations collected under this protocol.” |
A responsible reporting standard
At minimum, preserve the exact prompt, collection timestamp, consumer surface, visible model label when available, sign-in or temporary-chat state, geography and language where relevant, complete raw response, recommendation order, vendors mentioned, visible citations, trial count, validity rules, and exclusions.
Report counts beside percentages. Keep prompt-level and provider-level differences visible. Describe what was observed without inferring hidden retrieval, indexing, training, or reasoning behavior.
This principle is why Haenisch uses repeated, controlled measurements where the decision warrants them—and why remeasurement is required after an intervention. The objective is not to manufacture certainty. It is to make uncertainty measurable enough to support better commercial decisions.