LLM: Research Results Become Obsolete in 15.7 Months

A study conducted at Stanford on 50 statements regarding language models shows that a significant portion of these "effects" cease to be true within one to two years. While some trends persist—such as political bias and sycophancy—the disparities between models of the same generation remain pronounced, complicating any generalizations.

Documenting Models and Publishing Prompts is Requested by Authors

Researchers recommend presenting empirical results as observations on specific models at a given time, rather than as general truths. They advocate for systematically mentioning the model, its version, and the date, and for making prompts and analysis code public to allow for future retests. Performance varies significantly from one model to another, even for those released during the same period. Between July and September 2026, the number of results that became obsolete ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Neither the release date, the provider, nor the size of the models reliably predicts these patterns. Andrej Karpathy is quoted regarding the discrepancies between perceived behavior of LLMs and their effects on trust and usage, and he recently stated that current models are merely a glimpse of future capabilities.

Rapid Obsolescence Measured on 50 Statements and 20 Models

Fifty statements were tested on twenty different models launched between July 2024 and September 2026 by eight developers. The authors note that 44% of the results are no longer valid and that, for 22 effects with an assessable expiration date, the average validity duration was only 15.7 months. The initial effect has largely disappeared for expired results. Additionally, 34% of effects have weakened, while 16% remain valid and 6% have strengthened. In a temporal reading, 16% of results had expired after one year, and 39% after two years.

The Limits of "What Models Cannot Do" Fade Quickly

Statements describing the impossibilities of models are those that expire the fastest. Five of the nine results related to reasoning are declared expired, and none have maintained or strengthened. Examples cited include counting letters in a simple word, calculations in unknown numerical systems, information loss amid long contexts, or abandoning a correct answer when challenged. Researchers suggest that these errors, which have identifiable correct responses, provide clear training targets. In contrast, sycophancy, deception, and strategic behavior have more results that persist or strengthen.

What Holds: Evaluations by LLMs, Sycophancy, and Political Bias

Among the 11 results that persist or strengthen, three concern the role of LLMs as evaluators: preference for their own responses, similar judgments for accepted or rejected articles, and influence of the language in medical summaries. Three others are related to sycophancy and the tendency to approve user requests. Other constants noted include a left-leaning political bias, nearly uniform responses to open-ended questions, and an uneven distribution when generating random numbers. Regarding social biases, gender stereotypes and implicit association biases are decreasing, and the trend regarding African American English speakers is reversing in the data.

Reference Hallucinations Declining in Some Models, Not All

One case study focuses on the fabrication of scientific citations. In 2023, Walters and Wilder reported 55% of references fabricated by GPT-3.5. Using the same prompts, GPT-6.1 Sol produces none, while Claude Sonnet 5.5 generates 1%. In contrast, Qwen3.7-Flash still reaches 69% in the same test. Researchers urge caution in interpreting these advancements. Regarding two-hop questions, Claude Sonnet 5.5 still fails in about 10% of cases where it correctly answers both base questions, while ChatGPT showed 43% in early tests. Other phenomena, such as the curse of inversion and inconsistent responses to paired temporal questions, seem less pronounced but persist.