Why Does ChatGPT Cite Different Sources Every Time You Ask?

Because ChatGPT searches the web again every time, and gets a different set of pages back. In a new study, researchers asked ChatGPT the same 80 questions 20 times each. Across those runs, 40.8% to 44.5% of the pages it pulled up appeared in only one run. Fewer than 1 in 10 came back in 14 runs or more.

This is the gap topgeo.ai works on. topgeo.ai audits how often, and how favorably, AI engines cite your brand versus your competitors, and tracks it continuously, so one answer on one day doesn't decide how visible you think you are. You can start with a free AI visibility snapshot.

What did the study test?

The paper, "Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations", was posted on arXiv on October 11, 2026 by Yunxuan Jiang and four co-authors. It's a preprint, so it hasn't been peer reviewed yet.

The team took 80 questions from Search Arena and ran them through ChatGPT with web search between March and July 2026. The questions covered four types: fact lookups, information synthesis, analysis and recommendations. Half were in English and half in Chinese. The main tests used GPT-5.3 Instant and GPT-5.5 Instant, with 20 runs per question and 1,600 runs per model.

How much did the sources change?

A lot:

• One-off pages: 40.8% (GPT-5.3) and 44.5% (GPT-5.5) of the pages ChatGPT retrieved showed up in only one of the 20 runs.
• A small steady core: only 9.4% and 7.7% showed up in 14 runs or more.
• Little overlap: two runs of the same question shared only about 30% of their pages, counted out of all the pages either run pulled up. Counted by website instead of exact page, it rose to about 40%.
• Many pages per question: over 20 runs, each question pulled up 96 to 132 different pages on average.

Which pages actually get cited?

In each run, ChatGPT pulled up about 19 to 23 pages and cited about 6 of them. Pages that came back in more runs were more likely to be cited.

Google rankings were a weak guide. Google's results matched only 9.4% (GPT-5.3) and 6.4% (GPT-5.5) of the pages ChatGPT cited. Even combined with OpenAI's own API, the two together matched only about a quarter.

What doesn't the study prove?

Quite a bit, and the authors say so:

• General questions: they came from Search Arena, not B2B buyer questions, and half were in Chinese.
• Observational: it shows what ChatGPT did, not why, and it doesn't test any GEO tactic.
• Model switches: two sessions quietly fell back to a different model after 14 and 16 runs, so the authors advise checking which model answered.

What does this mean for checking your AI visibility?

One check is a snapshot, not the full picture. If you ask ChatGPT a buyer question once and your brand isn't there, that tells you less than it seems. The same goes if it is there. A few habits help:

1. Ask more than once: run the same question several times, on different days, before drawing conclusions.
2. Count websites, not just pages: which sites keep coming back is steadier than which exact pages do.
3. Don't use Google rankings as a stand-in: in this study, they matched only a small share of ChatGPT's citations.
4. Compare with competitors: track whether you show up more or less often than the brands buyers compare you with.

Source: Yunxuan Jiang, Shiqi Zhang, Anqi Xing, Xiangyu Chang and Xiaokui Xiao, "Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations", arXiv preprint, October 11, 2026.