Executive summary
The businesses shown and favored often changed with the buyer request.
We compared 80 pairs of consumer AI answers. Each pair asked about the same local market on the same AI surface, first with a general request and then with a more specific buyer need.
The favored business differed in 63 of 80 pairs, or 78.8%. Only 5 of 80 pairs returned exactly the same set of businesses. The two sets typically shared about one third of the businesses appearing in either answer.
Identical questions also varied when repeated. The study therefore does not show that buyer context caused 78.8% of primary recommendations to change. It shows that one answer was not a stable representation of the recommendation environment, and that inclusion and primary selection need to be measured separately.
The 78.8% figure is an observed difference across matched general and specialized pairs. It is not an estimate of the percentage caused by changing buyer context.
What we tested
We tested whether the businesses included and favored changed when a buyer moved from a general local search to a more specific service and priority.
The study examined Orthodontics and Med Spas across ChatGPT and Gemini. Each of the 80 pairs held the vertical, named location, and surface constant. The second request introduced a more specialized buyer context. Because it changed service need, persona, and stated criteria together, this is an observational matched-context design, not a one-variable causal experiment.
A really good local provider
A specific service plus stated buyer priorities
Five findings in plain English
The clearly favored business differed across 78.8% of matched pairs.
Only 6.3% of matched pairs returned exactly the same businesses.
The typical pair shared about one third of the businesses appearing in either answer.
Primary turnover also appeared when identical prompts were repeated within four separate conditions.
Many businesses appeared in recommendation sets without ever becoming the primary recommendation.
Why this matters
One AI answer is one observation.
A business can appear without being favored. Different buyer needs can coincide with different candidates and a different primary. Repeating the same question can also produce different answers. Measurement therefore needs both relevant decision contexts and repeated observations.
What the study does not prove
It does not isolate buyer context as the cause of the 78.8% turnover rate. It does not establish a stable rank, provider accuracy, business quality, or a universal result beyond the sampled Arizona markets, dates, prompts, and surfaces.
Deeper findings
The sections below preserve the full numerical results and methodological qualifications.
Most matched requests produced different business lists
Only 5 of 80 matched pairs (6.3%) returned identical recommendation sets. Another 5 of 80 (6.3%) had no business in common. The distribution was concentrated at lower overlap values, and median Jaccard similarity was 0.333.
Across the 80 pairs, a mean of 1.94 businesses entered and 2.45 left the set between the general and specialized observations. The context change often coincided with more than a reordering of familiar options: different candidates appeared.
How overlap was measured: Jaccard similarity divides the businesses shared by both lists by the businesses appearing in either list. A value of 1 means identical lists. A value of 0 means no business appeared in both.
The favored business changed in most matched pairs
The primary or clearly favored business differed in 63 of 80 matched pairs (78.8%) and remained the same in 17 of 80 (21.3%). Orthodontics differed in 28 of 40 pairs (70.0%); Med Spas differed in 35 of 40 (87.5%).
ChatGPT differed in 30 of 40 pairs (75.0%); Gemini differed in 33 of 40 (82.5%). These are descriptive results from this sample. They do not establish that one provider was more accurate, stable, or context-sensitive.
Inclusion and primary recommendation were different measures
Inclusion asks whether a business appeared in the extracted recommendation set. Primary selection asks whether the response most clearly favored it. The two measures produced materially different market pictures.
Among 35 normalized Orthodontics businesses included at least once, 18 (51.4%) were primary at least once and 17 (48.6%) were never primary. Among 173 normalized Med Spa businesses included at least once, 53 (30.6%) were primary at least once and 120 (69.4%) were never primary.
These aggregates describe outputs in this corpus, not independent assessments or rankings of the businesses. Identity normalization and the diffuse Med Spa candidate set are important limitations.
This is the empirical step from AI visibility to AI recommendation analysis. Appearing in an answer does not, by itself, establish that the business was selected for the decision.
Vertical and provider findings
| Comparison | Numerator | Denominator | Observed result |
|---|---|---|---|
| Primary differed · Overall | 63 | 80 matched pairs | 78.8% |
| Primary differed · Orthodontics | 28 | 40 matched pairs | 70.0% |
| Primary differed · Med Spas | 35 | 40 matched pairs | 87.5% |
| Primary differed · ChatGPT | 30 | 40 matched pairs | 75.0% |
| Primary differed · Gemini | 33 | 40 matched pairs | 82.5% |
| Same primary across providers | 18 | 80 equivalent-context comparisons | 22.5% |
For equivalent vertical-location-context observations, ChatGPT and Gemini selected the same primary in 18 of 80 comparisons (22.5%). Mean cross-provider set overlap was 0.246, the median was 0.183, and 16 of 80 comparisons (20.0%) had no overlap. Provider environments were not perfectly symmetric, and the study used no independent business-quality ground truth.
The same prompt also produced different answers
A separate 40-observation Orthodontics corpus contained ten trials in each of four identical-prompt surface and condition cells. Within those cells, primary turnover ranged from 16 of 45 pairwise comparisons (35.6%) to 39 of 45 (86.7%). Mean recommendation-set overlap ranged from 0.334 to 0.789.
Some matched-context turnover therefore reflects ordinary run-to-run variation. The repeated-condition rate cannot simply be subtracted from 78.8% because the two corpora have different prompt, persona, service, and geography structures.
Candidate-set overlap across the two repeated contexts was nevertheless lower than within-condition overlap on both surfaces. That makes set reshaping the stronger descriptive result while still stopping short of a causal claim. This is why one ChatGPT answer is not a ranking.
Full interpretation
- One answer is one observation. It should not be treated as a stable market ranking.
- Relevant decisions should be measured separately. Candidate sets and favored businesses can differ across buyer contexts.
- Repeated observations matter. Identical prompts can also produce substantial variation.
- Inclusion is not selection. A business can be visible without being favored for the decision.
Together, these findings support the distinction between AI visibility and Recommendation Intelligence: not only whether a business appears, but how candidate sets, stated criteria, evidence language, and favored outcomes vary across commercially meaningful decisions.
What the findings do not mean
- The study does not establish that buyer context caused 78.8% of primary recommendations to change.
- The study does not establish that the order in an individual answer is a stable search-style rank.
- The study does not establish that one AI surface is more accurate or better than another.
- The study does not establish that any business in the corpus is objectively better or worse.
- The study does not establish that the Arizona results generalize to other markets, verticals, dates, or model versions.
- The study does not establish that visible citations reveal hidden retrieval or causal evidence use.
Method and audit trail
Methodology
Sample construction
The primary analysis used 160 prespecified first-repetition observations: 80 Orthodontics and 80 Med Spa responses forming 80 pairs by vertical, named location, and consumer-AI surface. No observation or pair in the frozen main corpus was removed after validation.
The Orthodontics observations were collected August 30, 2026, across Ahwatukee, Chandler, Gilbert, Mesa, and Queen Creek. The Med Spa observations were collected August 31, 2026, across 12 Arizona areas. Each vertical contained 40 general and 40 specialized observations, evenly divided between ChatGPT and Gemini.
Matched contexts
The general prompts asked for a really good local orthodontist or med spa. The specialized Orthodontics prompt introduced an adult considering clear aligners who prioritized experience with adult cases. The specialized Med Spa prompt introduced wrinkle-relaxer injections, natural-looking results, and experienced injectors.
The specialized prompts changed multiple elements. They do not isolate a single decision criterion.
Outcome definitions
- Recommendation-set inclusion
- A normalized business appeared in the visibly extracted recommendation list.
- Primary recommendation
- The first or clearest favored business under the frozen extraction rule, using explicit ordering, superlative language, and concluding selection logic where needed.
- Jaccard overlap
- The businesses shared by two normalized recommendation sets divided by the businesses appearing in either set.
- Primary turnover
- The two members of a matched pair had different validated primary recommendations.
Track M extraction correction
Validation found that a known-universe extraction step omitted visibly recommended businesses from part of Track M. An analytic copy was corrected against the preserved raw responses. Independent QA reviewed all 23 proposed primary corrections: 22 were confirmed; one was rejected and restored to the raw-response-supported primary. Original source files were never modified, and their hashes remained unchanged. Every public statistic in this article uses the independently reviewed analytic copy.
Provider provenance
Environment provenance was not perfectly symmetric across all sources. Authentication, Temporary Chat, memory, and personalization controls differed or were unknown in some provider records. Cross-provider results are therefore descriptive. Visible citations were retained where exposed, but they were not treated as proof of hidden retrieval behavior.
Read the broader Haenisch methodology.
Limitations
- Observational, nonrandom sample. The data came from commercial-validation work, not a randomized research panel.
- One main observation per context cell. The primary corpus cannot separate context effects from run-to-run variability.
- Multi-element prompt changes. General and specialized requests changed service need, persona, and decision criteria together.
- Substantial identical-prompt instability. Repeated trials rule out a simple causal reading of primary turnover.
- Dependent repeated comparisons. The 45 pairwise comparisons per cell reuse ten observations.
- Provider-provenance asymmetry. Environment controls were not identical across all source records.
- No universal mention coding. Inclusion means recommendation-list inclusion, not every brand mention or conditional reference.
- Local and vertical scope. Results cover sampled Arizona Orthodontics and Med Spa markets only.
- Time and model specificity. AI outputs can change with providers, products, evidence, and time.
- No accuracy ground truth. The study measures output behavior, not objective business quality.
- Identity normalization. Conservative business-name normalization can affect overlap and concentration.
- No causal inference. The study does not estimate how much turnover was caused by buyer context.
Data and provenance
Internal provenance records preserve exact prompts, timestamps, complete raw responses, raw-response hashes, source and publication extractions, correction rationales, pair keys, exclusions, and chart-source tables. The publication uses no verbatim AI-response excerpts and identifies no individual business example.
The frozen Track L and Track M source files remain unchanged. External spend for the study was $0, and no paid API call was used for this publication build.
About Haenisch Research
Haenisch Research studies how consumer AI systems form commercially important recommendations: which candidates appear, which options are favored, how outcomes vary across decisions and repeated observations, and what the visible evidence can and cannot establish.
Haenisch is an AI-native Recommendation Intelligence & Engineering company. The research is designed to remain useful independently of Haenisch's commercial positioning. Return to the Haenisch Research index.