LLMs: Masters of Code, but Struggling with Simple Tasks

Large language models (LLMs) demonstrate an impressive ability to solve complex problems in programming and mathematics, but struggle when it comes to simpler everyday questions. Andrej Karpathy, an influential figure in the field of artificial intelligence, explains why this situation is not as contradictory as it may seem.

Karpathy identifies two distinct groups in the perception of LLM progress. The first group consists of users who have tested free versions of ChatGPT or its voice mode, often disappointed by obvious errors and hallucinations. According to Karpathy, these models do not reflect the current state of technological advancements.

The second group, on the other hand, uses more recent and powerful models, such as GPT-5.4 Thinking from OpenAI or Claude Opus 4.6, in professional environments like Codex or Claude Code. These tools have enabled significant advancements this year, particularly in restructuring entire codebases or autonomously detecting security vulnerabilities. Karpathy emphasizes that these two groups are actually discussing fundamentally different subjects.

It is interesting to note that while OpenAI's free voice mode may fail on simple questions, advanced models like Codex can spend hours coherently restructuring a codebase or identifying security flaws in computer systems.

Verifiability: The Key to AI Progress

Karpathy highlights a crucial aspect: areas where it is possible to clearly verify the correctness of an answer, such as code or mathematics, benefit more from AI advancements. This is due to reinforcement learning, which relies on verifiable rewards to optimize performance. In contrast, more subjective areas, such as writing or consulting, lack clear metrics to optimize.

This situation raises a central question: can general intelligence emerge from language models, or are these models destined to excel only in specific domains? Karpathy has previously addressed this issue in an earlier essay, emphasizing that in the "Software 2.0" paradigm, the ability to verify a result is more important than the specification of the task itself.

Last summer, rumors circulated about a universal verifier from OpenAI, which could make reinforcement learning applicable across all domains. However, no concrete advancements have been made to date. Meanwhile, Jerry Tworek, a key figure in OpenAI's reinforcement learning strategy, recently left the company, stating that "deep learning research is over."