Studies / Ask twice
Ask an AI the same question twice and you'll often get a different list. We asked identical questions thousands of times, on ChatGPT, Gemini, Claude, Perplexity and Google. Any single answer changes a lot, but over many asks a business's rate can be measured to within a few points. It's like a coin: one flip tells you little, a hundred flips tell you the odds.
What we found
One answer tells you little.
35–40%
Ask Gemini the same thing twice and only 35–40% of the businesses it names appear both times.
Finding 02 ↓But over many asks, the rate is precise.
±2–4 pts
Count how often a business is named across many asks, and that rate is accurate to within about 2–4 points.
Finding 01 ↓A silent AI update can move a business 30 points.
54→84%
When a new Gemini version became the default in July, one practice went from 54% to 84% on the same questions.
Finding 04 ↓Why it matters
If someone shows you a screenshot of ChatGPT recommending your business, or leaving you out, they've shown you one roll of the dice. Ask again and the list can change. Ask next month and the AI itself may have changed. Decisions made on one answer are decisions made on luck.
The useful question isn't "were we named?" It's "how often are we named, and has that really changed?"
What the study delivered
This is the groundwork under every number we report. By asking the same thing over and over, we know how much each platform moves on its own. So when your number changes between studies, we can tell you whether it really moved, or whether that's just the AI being the AI.
Rates, not snapshots
Every figure is how often, across many asks.
A known margin
How far each platform drifts on its own.
Real change spotted
A change bigger than both rounds' margins is real.
Along the way
Patterns from repeated questions across our studies. No business is named.
01
Measured properly, a business's rate on ChatGPT, Claude, Gemini or Perplexity is accurate to within 2–4 points. That's what makes a rate trustworthy, and a single answer not.
±2–4 pts
margin on a measured rate, current models
02
Ask Gemini the same question twice and, of all the businesses it names across the two answers, only about 35–40% appear in both. Claude is steadier at around 55–60%. Google's map is the steadiest, at around 80%.
03
On Google's map, the same business comes first in about 8 of every 10 repeats. On Gemini, the first name changes far more often, and a single question can produce half a dozen different "first" answers over a study.
04
In July, Google quietly switched the version of Gemini most people use. Same questions, same town, old version against new: one practice went from being named 54% of the time to 84%. Another fell from 75% to 46%. The normal wobble is a couple of points.
54→84%
one practice, after the update
75→46%
another, same update
05
On 8 October, in the middle of one of our studies, ChatGPT's answers changed format from one day to the next: longer, bolder, more like a list. Earlier version changes on the same questions halved answer lengths, or made an AI far more likely to say "it depends".
219 → 361
median words per ChatGPT answer, 7 vs 8 October
06
Perplexity's answers on the same day look more alike than its answers on different days, so it drifts over time. Gemini varies just as much within an hour as across days. Its wobble is in every single answer, not in the calendar. That's why we spread every study across several days and times.
Your own study gives you a rate you can trust, and a repeat tells you what really changed.
The wobble and overlap figures come from our two summer dental studies, on the AI versions being served from late July onward. The July update figures come from Peterborough. The October format change was seen across two separate studies on either side of it. Claude's figures were collected with an earlier setting that cut off most of its answers, since fixed.