Talkie: The Retro AI Envisioning 2026 Through 1930s Eyes

Talkie: A Language Model Rooted in the Past

Researchers led by Alec Radford have developed a language model named "Talkie," featuring 13 billion parameters. This model is distinguished by its exclusive training on texts published before 1931, which limits its knowledge to that of the early 20th century.

When asked about the future, Talkie responds with a perspective from before 1931. For instance, it considers a Second World War unlikely and imagines the year 2026 dominated by steamships and railway networks, reflecting the technological expectations of that era.

A Vision of the World Without a Second World War

When asked to describe the world in 2026, Talkie offers a vision inspired by Victorian futuristic novels: a Europe populated by a billion inhabitants, crisscrossed by iron railways, and steamships connecting London and New York in ten days. When questioned about the possibility of a second world war, Talkie responds negatively, asserting that nations have learned from the madness of 1914-1918 and are turning towards peaceful activities.

However, Talkie remains cautious and warns of latent tensions in Europe, particularly between China and Japan, or between Italy and Yugoslavia. It emphasizes that global peace depends on many factors, none of which can be overlooked.

The Predictive Limits of Talkie

The researchers also attempted to quantitatively measure Talkie's predictive limits. They submitted nearly 5,000 descriptions of historical events from the "On This Day" feature of the New York Times to the model and measured how surprising it found them. The pattern is clear: after the knowledge cutoff date of 1930, surprise values increase sharply, peaking in the 1950s and 1960s, before stabilizing.

Victorian Etiquette Guides Instead of Modern Chat Data

The team chose the end of 1930 as the cutoff date because that is when works enter the public domain in the United States. Each text had to be transcribed from physical sources, which created serious quality issues. In controlled experiments, standard OCR transcriptions achieved only 30% of the performance of a model trained on human transcriptions using the same computing power. A simple regex cleanup brought this figure up to 70%. A custom vintage OCR system is planned to reduce the remaining gap.

Another challenge is to prevent knowledge from later periods from seeping into the training data. A book from 1925 might receive an updated preface in a 1960 edition, library catalogs sometimes list the wrong publication date, and footnotes or comments may be added to a historical text long after it was written. Despite a classifier designed to detect this type of contamination, information about Roosevelt's presidency, the Second World War, and the United Nations still filtered through, according to the team. Better classifiers are planned for future versions.

A Vintage Model Capable of Basic Programming

The team also tested whether a model without knowledge of digital computers could learn modern programming languages. On the HumanEval benchmark for Python, vintage models perform significantly worse than their modern counterparts but gradually improve as they are extended.

Each correct solution is a simple line of code or a slight adjustment of an example program. Talkie, for instance, correctly implemented the decoding function of a rotation cipher by replacing an addition with a subtraction. The researchers claim this indicates a basic understanding of inverse functions.

Towards a GPT-3 Level Model from the Past

Talkie is available as a base model and chat version on Hugging Face, with the code on GitHub. You can also test it live on the project's website, where Claude Sonnet queries Talkie about its knowledge and skills 24/7.

However, the 13 billion parameter model is just the beginning. Developers plan to significantly scale up Talkie in the coming months, targeting a GPT-3 level model for summer 2026. Early estimates suggest that the corpus could grow to exceed a trillion tokens of historical texts, enough to train a model comparable to GPT-3.5. A multilingual expansion beyond English is also planned.

The broader question driving the project: can a vintage model anticipate discoveries and inventions that occurred after its cutoff date? Could a model trained only up to 1911 independently derive general relativity, as suggested by DeepMind CEO Demis Hassabis? Larger vintage models could help reveal these scaling trends.