Theoretical Physics: Astra Solves 49/50 for $116

A set of 50 verifiable problems in theoretical physics places GPT-6-Astra at the top with 98% correct answers and a total cost just under $116. These tests are deemed representative of "work to solve," and it is noted that web access changes scores little. However, they warn that designing good questions and choosing fruitful directions remain weaknesses of the agents, with implications for training and equal opportunity.

Training and Budgets Already Impacted

Effects on training are already being observed: some exercises that could have served as initial projects for beginners are no longer suitable unless explicitly aimed at a pedagogical goal. They remind us that theoretical physics has historically required less funding than experimental research, which has helped maintain relatively equal opportunities. According to them, the arrival of high-performing AI agents could alter this balance. The full execution of Astra on all problems cost about $116, but a significant increase in resource needs is anticipated for more complex problems. At this stage, the budget of a research group could become a more determining factor, raising questions of access and equal opportunity. Similar debates already exist in mathematics, particularly around the Leiden Declaration, and it is estimated that theoretical physics should engage with this. One school of thought has long encouraged "shut up and calculate"; if taste and vision were to gain more importance, "stop and think" might become necessary.

What Calculations Do Not Capture: Theory, Choices, and "Taste"

It is emphasized that a verifiable calculation comes at the end of the research process: one must first build a theory and ask the right questions. When a problem proves too difficult, human researchers know how to turn to a related, more accessible problem, something current AI agents do not do well. They believe that physicists share, albeit without unanimity, a sense of the interest of a problem, a "taste" that current models lack. When asked to design new problems, agents linger on details that a human would deem irrelevant. It is specified that these observations are not quantifiable and do not constitute a forecast about the future of AI, even if it is deemed plausible that agents could soon conduct high-level research autonomously, provided they move beyond mere problem-solving. The benchmark was designed to target "to solve" type tasks, slightly modifying known questions to prevent deducing the answer from previous articles, a common format for early work assigned to PhD students.

Scores: Astra Leads with 98%, Opus at 92%, Gemini at 68%

GPT-6-Astra, executed in OpenAI's Codex environment, achieved 49 correct answers out of 50, or 98%, at an average cost of less than $3 per problem and a total just under $116. Anthropic's Opus 5.5 reached 92%, Google’s Gemini 3.8 Flash 68%, and DeepSeek-V4.1-Flash and GLM 5.3 scored 54% and 42%, respectively. Without internet access, Astra and Opus each scored 92%, showing that web research had a limited effect on the results. With web access, Astra only failed on one problem that stumped all models: it involved finding an equation producing a given curve, whose expert solution relies on mathematical forms that only loosely correspond to the required differential equation, a case deemed realistic. On the black hole problem, Gemini and GLM reused valid formulas in five dimensions instead of the four required.

A Battery of 50 Problems, 11 Subdomains, and Doubled Runs

Thirty-five researchers from institutions including ICTS, Harvard, KU Leuven, and Google DeepMind designed TheoPhysEval, a set of 50 unpublished research problems covering eleven subdomains, from fluid flows to cosmology. Commercial agents were evaluated without human hints, each tested twice: once with web research, once without internet access. An expert in the subdomain typically solves a problem in a day or less, while an outside specialist would take much longer. An example in black hole research illustrates the complexity: one must locate an old article, adapt equations from five to four dimensions, and then perform the calculation with a specific program. Suvrat Raju, co-author of the original paper, took about a day to solve it, and before modern agents, even experienced experts would have needed several days.

Adjusted Grading Keys, Discussed Benchmarks, and Model Memory

Post-hoc checks with experts revealed small errors in some reference solutions; once corrected, they validated Astra's answers. The agents performed better on TheoPhysEval than on other benchmarks like CritPt, even though these problems are considered, on average, more difficult. A recent study showed that some benchmarks in theoretical physics are poorly rated and sometimes underestimate the capabilities of the models. Many exercises rely on formulas from articles that the models generally have not been exposed to, which is interpreted as a sign of extensive memorization of the literature by the best systems. They argue that some models may already surpass the best humans in concrete calculations, seeing this as a major step. They note that the impact is even more visible in mathematics, where, according to OpenAI, agents have already solved open problems like the Navier-Stokes problem, while no comparable breakthrough has yet occurred in theoretical physics.