OpenAI’s API Only Partly Matched ChatGPT’s Citations, Paper Finds

Stand-Ins for ChatGPT's Citations Miss Most Cited Pages, So Page-Level AI Visibility Reports Deserve Doubt

Search Engine Journal reports on an arXiv preprint from researchers at the National University of Singapore and Xi'an Jiaotong University. They tested whether Google search results and OpenAI's Responses API can stand in for what ChatGPT's web app cites, using data gathered mostly during 2026. Together, the two stand-ins found 28.3% of the pages GPT-5.3 Instant cited and 26.2% for GPT-5.5 Instant. For cited sites the figures were 50.3% and 48.7%. Google alone found 9.4% and 6.4% of pages, and 26.1% and 22.8% of sites. The API's full source list found 22.9% and 22.5% of pages.

Can I use OpenAI's API to track what ChatGPT cites?

Only loosely. In the paper's tests the API's source list covered 22.9% and 22.5% of the pages ChatGPT cited, depending on the model version, so most cited pages never showed up.

The comparison also ran the other way. Of the pages the API cited in its own answers, GPT-5.5 Instant cited 23.3%, a higher share than GPT-5.3 Instant managed. That is a closer match, but still a minority.

Those are different measures, and most dashboards blur them. Coverage asks how much of ChatGPT's citation set a proxy captures. Agreement asks how much of the proxy's output ChatGPT also cites. A vendor quoting "accuracy" without saying which is telling you very little.

Is tracking by site more reliable than tracking pages?

Somewhat. Google results plus the API matched 48.7% to 50.3% of cited sites versus 26.2% to 28.3% of cited pages, so site-level tracking helps but still misses about half.

The authors suggest tracking by site, rerunning prompts and re-testing after model changes. Site-level data tells you whether a domain appears, not whether a particular article earned the citation. The paper treated subdomains as separate sites, so the site figures are, if anything, on the strict side.

Classic rank data fares worst. Google results alone covered 9.4% and 6.4% of cited pages. Our earlier note on a ChatGPT traffic drop that appears to be a tracking problem is a reminder that measurement is the fragile part of this field.

How far should I trust these numbers?

With caution. It is an arXiv preprint, which is not peer reviewed, and it tests a limited question set. How the authors define "cited" shapes every percentage.

A page counted as cited if ChatGPT cited it in at least one of the repeated runs per question, with percentages averaged across questions. The researchers captured traffic from their own signed-in sessions. The questions came from Search Arena, a public collection of real chatbot queries, in English and Chinese, spanning recommendation, fact lookup, synthesis and analysis. Nothing guarantees that mix resembles your commercial queries.

Match rates also shifted between versions, from 28.3% to 26.2% of pages. Some older versions were retired before collection ended and got fewer questions. We worked from SEJ's report, not the paper itself.

What should I ask my AI-visibility vendor?

Ask whether the tool reads ChatGPT's web app or the API, whether it reports pages or sites, how often it reruns prompts, and when it last re-validated after a model change.

Most vendors do not publish how they collect answers. If the answer is the API, this paper suggests expecting most cited pages to go missing. Ask for a match rate against captured web-app sessions, and for the denominator behind any "citation share" figure. The same questions apply to scoring products such as the KSMModel.app launch we covered earlier. A score is only as good as the sampling beneath it.

Frequently asked questions

Does the paper say API-based tracking is useless?

No. It says the stand-ins are partial, and closer on sites than on pages. The authors recommend site-level tracking, repeated prompts and re-testing after model changes.

Did Google rankings predict ChatGPT's citations?

Poorly. Google results alone accounted for 9.4% and 6.4% of cited pages and 26.1% and 22.8% of cited sites, for GPT-5.3 Instant and GPT-5.5 Instant respectively.

Which ChatGPT versions were tested?

Versions from GPT-5.2 Instant to GPT-5.5 Instant. The fullest runs were on GPT-5.3 Instant and GPT-5.5 Instant, which is why the headline figures cover those.

Sources: Search Engine Journal