How Many AI Recommendation Checks Are Enough? We Found No Universal Answer

Executive summary

One answer showed only part of the recommendation picture.

We repeated the same four buyer questions twenty times on both ChatGPT and Gemini, producing 160 valid consumer AI responses. The businesses shown continued to change as more responses accumulated.

A typical one-response sample captured about 41% of the businesses that appeared across twenty repetitions. Ten repetitions produced a useful directional view. Fifteen improved candidate coverage and inclusion estimates materially. Even then, some conditions continued to reveal new businesses.

There was no universal stopping point. Inclusion frequency became more stable sooner than complete candidate discovery in some conditions. Primary selection required separate interpretation because ChatGPT named one clearly favored business in 47 of 80 responses, while Gemini did so in only 3 of 80.

What we tested

The central question was how many repeated observations were needed before the measured recommendation picture became meaningfully more stable.

Five findings in plain English

More repetitions produced a clearer, still incomplete picture

Candidate coverage asks how much of the twenty-response candidate universe a smaller sample found. “Candidate universe” means every distinct business observed across the twenty responses in one exact prompt and surface condition.

Seven of 102 candidate and condition records appeared in all twenty responses. Eighteen appeared once. Five candidates first appeared during repetitions sixteen through twenty, all in the two injectables conditions. Other conditions stopped adding candidates earlier.

Inclusion estimates improved with depth

Inclusion frequency is the share of responses in which a business appeared in the extracted recommendation set. Mean absolute error describes how far a smaller sample's inclusion estimates were from the estimates based on all twenty responses.

Ten observations met a prespecified ten-point mean inclusion-error tolerance in all eight conditions. A stricter five-point tolerance was met at fifteen observations in four conditions and required twenty in the other four.

What the findings mean

This extends Study #1. The first study showed that one answer was not a stable representation across buyer contexts and repeated prompts. Study #2 measured the repeated-prompt problem directly. More observations produced a clearer picture, while no single depth stabilized every outcome.

What the findings do not mean

1. Every business needs exactly twenty AI checks.
2. Twenty observations reveal the true recommendation distribution.
3. AI recommendations are random or meaningless.
4. One provider is more accurate than the other.
5. A larger recommendation list is better.
6. The Gilbert results generalize to every market, vertical, date, prompt, or provider version.

Complete recommendation lists rarely repeated

Each condition produced 190 unordered pairs of responses. Across all eight conditions, only 94 of 1,520 pairs, 6.2%, contained exactly the same set of businesses.

How overlap was measured: Jaccard similarity divides the businesses shared by two sets by the businesses appearing in either set. A value of 1 means identical sets. A value of 0 means no shared business.

The general Med Spa ChatGPT condition was comparatively stable, with a median overlap of 0.714. It still produced 12 distinct sets in 20 responses. The injectables ChatGPT condition produced 19 distinct sets and a median overlap of 0.300. Not every condition was equally unstable.

Primary recommendation depended on response structure

A unique primary business was identifiable in 50 of 160 responses. The provider split was 47 of 80 for ChatGPT and 3 of 80 for Gemini.

Primary selection also varied within ChatGPT. One adult-aligner business was primary in 11 of 20 responses. In the injectables condition, the leading business was primary in 6 of 20, another appeared in 5 of 20, five responses had no unique primary, and four other businesses appeared once each.

Inclusion and primary selection told different stories

In every condition, the most frequently included business appeared more often than the most frequently selected primary business.

This is the empirical distinction between measuring whether a business surfaced and measuring whether it was distinctly favored for the decision. It does not rank businesses or compare provider accuracy.

Position was limited and often changed

Meaningful order was available in 33 of 160 responses: 7 general Orthodontics ChatGPT responses, 10 adult-aligner ChatGPT responses, 14 injectables ChatGPT responses, and 2 injectables Gemini responses. The other four conditions did not provide enough visibly ranked structure for position analysis.

Thirty candidate and condition records had at least two meaningful position observations. Twenty-five of those 30, 83.3%, appeared in more than one position. This finding is conditional and sparse. It does not support treating order in a single answer as a stable search rank.

ChatGPT and Gemini produced different visible structures

ChatGPT responses contained more businesses on average in all four prompts. Average list lengths ranged from 5.35 to 6.75 on ChatGPT and from 3.45 to 4.05 on Gemini.

Candidate-universe overlap between providers ranged from 0.217 to 0.385. Weighted inclusion overlap ranged from 0.066 to 0.168. These are visible output differences under the recorded consumer conditions. They are not provider-accuracy scores.

Study design

The study used four frozen prompts, two consumer surfaces, and twenty valid repetitions per prompt and surface. The resulting eight exact conditions contained 160 valid observations.

Collection conditions

Collection ran on September 2, 2026, from 15:05:04 through 19:59:41 UTC. ChatGPT used the signed-out consumer web product in a new root conversation for every observation. The interface did not expose an exact model label. Gemini used a new Temporary Chat for every observation through the same organization-managed Work account, with the visible mode set to Flash.

Twenty randomized blocks each contained all four prompts on both surfaces. Prompt order varied by a frozen seed. Provider order alternated by block. This balanced sequence reduced systematic time and order effects without claiming control over hidden provider conditions.

The attempt ledger contains 187 records. Twenty-seven were invalid technical attempts, and eleven planned observations required at least one retry. Invalid attempts remained preserved with a reason. No valid answer was replaced because of its recommendation outcome.

Extraction

The extraction recorded the visible recommendation set, a unique primary business when the response clearly favored one, and meaningful order when the response visibly ranked or sequenced businesses. `no_unique_primary` remained a valid outcome when a response presented coequal options.

Business identities received deterministic typography cleanup and conservative normalization. A business was included only when the visible response recommended it under the frozen rule.

Calculations

Each twenty-response condition contains 190 unordered response pairs. Exact-set repetition and Jaccard overlap were calculated for all 1,520 pairs. These pairwise comparisons reuse observations and are descriptive rather than independent inferential samples.

Every possible subset was evaluated at 1, 2, 3, 5, 10, 15, and 20 observations in each condition. This produced 217,115 subset evaluations per condition and 1,736,920 overall. Candidate coverage, inclusion-frequency error, and primary-distribution distance were compared with the complete twenty-response condition.

Technical definitions

Recommendation set

The normalized businesses visibly recommended in one response.

Candidate universe

Every distinct business observed across twenty responses in one exact condition.

Candidate coverage

The share of the twenty-response candidate universe found by a smaller subset.

Inclusion frequency

The share of responses in which a business appeared in the recommendation set.

Primary recommendation

One business explicitly or structurally favored under the frozen extraction rule.

No unique primary

A valid outcome when a response did not distinctly favor one business.

Jaccard overlap

The businesses shared by two sets divided by the businesses appearing in either set.

Mean absolute error

The average absolute difference between smaller-sample inclusion frequencies and the twenty-response reference.

Meaningful position

Visible order that supports a first, second, or later position. Coequal lists were not given artificial ranks.

Extraction quality assurance

An initial pass produced candidate sets, primary selections, and positions for all 160 responses. A separate high-recall structural review agreed directly on 110 and routed 50 to raw-response resolution. A final vocabulary audit found five records containing non-business descriptor text. One was already routed, so four more records received manual review.

Manual resolution removed eleven non-business strings, added three visibly recommended businesses to one response, and corrected twelve `no_unique_primary` cases in which the raw response explicitly favored one business. Conservative coding remained when the wording favored a subcategory or gave a coequal shortlist.

Validation found zero missing observations, zero duplicate IDs, zero empty recommendation sets, zero primaries outside their set, and zero raw-file hash mismatches. No raw source file was changed.

Limitations

1. Limited scope. One local market, two verticals, four prompts, two surfaces, and one collection day.
2. Twenty is not truth. It is the maximum observed depth and the internal reference for smaller samples.
3. Dependent diagnostics. Pairwise comparisons and exhaustive subsets reuse observations.
4. Different surface conditions. ChatGPT was signed out; Gemini used an organization-managed Work account in Temporary Chat.
5. Unobservable backend state. Provider routing, retrieval, indexing, model state, and backend changes were not controlled.
6. Different response structures. Gemini produced only three unique primaries, limiting winner-stability analysis.
7. Sparse order. Meaningful position was available in 33 of 160 responses.
8. Time window. Balanced blocking reduced but could not remove drift during the roughly five-hour collection.
9. Extraction judgment. Identity normalization and outcome extraction required review, even with independent QA.
10. No quality ground truth. The study does not determine which recommendations or businesses were objectively better.
11. No universal threshold. No single repetition count stabilized every measured outcome within the observed range.

Data and provenance

The internal package preserves exact prompts, the randomized manifest, timestamps, visible environment fields, complete raw responses, invalid attempts, extraction decisions, resolution logs, hashes, tables, and analysis outputs. Raw responses are not public because they can contain unverified claims about local businesses.

About Haenisch Research

Haenisch Research studies how consumer AI systems form commercially important recommendations: which candidates appear, which options are favored, how outcomes change across buyer decisions and repeated observations, and what visible evidence can and cannot establish.

Haenisch is an AI-native Recommendation Intelligence & Engineering company. The research is designed to stand on its evidence and remain useful independently of the company's commercial positioning.