AI Hallucinations: A Threat to Scientific Research

Scientific articles, the pillars of modern knowledge, are now threatened by a troubling phenomenon: artificial intelligence hallucinations. A recent study conducted by researchers from Cornell and UCLA has highlighted the extent of this issue, revealing that 146,900 false citations generated by AI have been identified in scientific articles across four major research databases.

Large language models, such as Gemini and ChatGPT, are at the root of this phenomenon. While they are capable of producing information that seems plausible, these models can generate entirely fictitious references if researchers do not verify the citations provided by these chatbots. This situation is particularly concerning as scientific articles, although often not accessible to the general public, have a significant impact on our daily lives, influencing innovations ranging from the Internet to lithium-ion batteries.

A Science in Peril

The study analyzed 111 million references from 2.5 million scientific articles. The researchers searched for citations with titles that did not correspond to any existing publication. While some errors were due to typos, many were hallucinations. Unscrupulous researchers had falsified citations long before the rise of chatbots, and the team examined the rates of unmatched citations in research published before 2023, when chatbots were not yet ubiquitous. This problem is not new, but it has intensified with the widespread adoption of large language models (LLMs) since 2023. The bad citations are scattered across numerous articles, indicating a widespread use of AI-generated references without adequate verification.

A Warning for the Scientific Community

Usha Haley, a management professor at Wichita State University, expressed her concerns about the proliferation of these false citations. According to her, this undermines trust in the academic record, which is crucial for peer review and the accumulation of knowledge. This skepticism, which is now emerging within the academic community itself, is particularly alarming for early-career researchers.

The databases arXiv, bioRxiv, SSRN, and PubMed Central, which play a vital role in disseminating scientific research, are the most affected by these false citations. These repositories allow authors to share their work before official publication, thereby increasing their visibility. The new article on AI hallucinations regarding citations is currently hosted on arXiv. In response to this crisis, arXiv has recently decided to ban submissions containing unverified AI-generated citations.

Steinn Sigurdsson, the scientific director of arXiv, told Katelyn Chedraoui from CNET in February that the corpus of science is being diluted by AI-generated content. This content is often incorrect or insignificant, complicating the task for researchers to discern true scientific advancements and potentially misleading those who rely on this information.