Brief IA
›

MedChemBench Assesses AI on 99 Medicinal Chemistry Problems

Models & LLM·Tom Levy·

MedChemBench Assesses AI on 99 Medicinal Chemistry Problems

MedChemBench Assesses AI on 99 Medicinal Chemistry Problems
Key Takeaways
1MedChemBench evaluates reasoning in medicinal chemistry with 99 problems.
2GPT-6.2 Astra achieves 84.5% of the available credit at maximum tuning.
3Prior exposure of the models to the used searches cannot be ruled out.
Why it matters : the benchmark tests the correlation of molecular structures with experimental data, without demonstrating an ability to discover drugs.

MedChemBench evaluates AI models on 99 medicinal chemistry problems. GPT-6.2 Astra achieves 84.5% of the available credit at maximum reasoning tuning. However, these results do not demonstrate an ability to discover drugs.

PRMT5 confronts models with molecular trade-offs

Inhibiting a protein more effectively is not enough to make a compound preferable. An exercise focused on PRMT5, a protein studied in cancer research, requires predicting the behavior of three related compounds based on their structures and previous experimental data. The predictions concern inhibition in cells, crossing a cellular barrier, and the rate of degradation by liver enzymes during a laboratory test.

The results of subsequent experiments are not communicated to the models. They show that compound 23 inhibited PRMT5 at a lower concentration in cells, while compound 24 crossed the barrier better and was more resistant to enzymatic degradation. The exercise thus verifies the established link between a structure and multiple behaviors, rather than an isolated property.

These trade-offs are part of the work of medicinal chemists, who design, synthesize, and then test related molecules before choosing the next steps in their experiments. An improvement may come with a degradation elsewhere: a compound that is very active against a protein associated with a disease may, for example, be poorly absorbed or decompose too quickly in the body.

GPT-6 Astra succeeds in a demanding molecular assembly

The representation of the molecule constitutes another aspect of the evaluation. Chemical diagrams indicate the bonds between atoms, cycles, and stereochemistry. The reference checks that this information is preserved when a model reconstructs a compound from a drawing and chemical groups.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

One exercise thus requires assembling a central core and two groups while maintaining a cycle of 18 atoms. At the xhigh level, the maximum tuning common to the eleven models in this example, GPT-6 Astra correctly reconstructed the compound in all ten responses. GPT-5.6 Sol and GPT-5.6 Luna failed in each of the ten responses; these errors change the molecule that is actually represented.

GPT-6 Astra achieved the best results for molecular reconstruction and the explanation of experimental data. Choosing the compounds and anticipating the numerical values of their properties posed more difficulties for it.

Scores are based on already published research

The corpus includes 117 articles and patents, with 761 distinct molecular structures. For the majority of the exercises, the model receives information accessible at a stage of the scientific program: biological target, objectives pursued, molecules already examined, and some experimental results. In addition to reconstruction, the scoring focuses on scientific justification, ranking and selection of compounds, as well as the alignment of predictions with measured or calculated property categories or values.

At maximum reasoning tuning, GPT-6 Astra achieved 83.3% of the available credit, GPT-6.1 Sol 82.0%, and GPT-5.6 Sol 58.2%. Each response corresponds to a unique trial, without the use of tools, without corrective feedback, and without additional experience.

It remains possible that the models have already been exposed to the published sources used. The test evaluates reasoning based on historical data rather than drug discovery in real conditions: its results do not establish whether the models are capable of discovering new drugs or improving laboratory decisions.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.