Can AI Surpass Textbook Authors?

AI Writing: Between Promises and Disappointing Realities

Writing a manual on artificial intelligence raises a crucial question: how long will it take before AI can perform this task better than a human can? AI-generated writing has sparked numerous criticisms, especially when it comes to creative or voice-driven texts, such as blogs. These critiques, including those from many experts, emphasize that true writing is characterized by a distinct voice, a personal perspective, and a profound expression of humanity. These elements are often absent from the outputs of large language models (LLMs), which, although becoming more sophisticated tools, seem to regress in their ability to produce truly inspiring writing.

In contrast, non-fiction writing, which includes explanatory and informative texts, initially appeared to be a domain where LLMs could excel. Sam Altman, in a recent podcast, mentioned the ability of LLMs to generate filler text and useful copy. However, shortcomings in this area are often attributed to a lack of general intelligence in the models or training issues. Despite improvements, these models struggle to achieve a high level of non-fiction writing quality.

The Challenges of Non-Fiction Writing for LLMs

The stagnation of models in producing long-form non-fiction writing is concerning, especially for those who hope these technologies can autonomously solve major scientific problems. Currently, models struggle to organize and present even the most established sciences convincingly. This poses a significant barrier to their ability to independently tackle complex and open-ended problems. As long as these challenges remain unaddressed, the progress of LLMs in the scientific domain will be limited to solving simple problems and making connections between different fields, rather than achieving groundbreaking discoveries.

This position may seem paradoxical to those who are optimistic about AI advancements, especially at a time when companies like Anthropic announce progress on famous mathematical problems. For example, Anthropic published a blog post about Claude, which made strides on the famous Riemann Hypothesis. However, the scope of scientific problems is vast, and it is unlikely that current AI models can cover them as broadly as some believe.

Why is Progress Slow?

Much more progress was expected from AI models in non-fiction writing, to the point that publishing a book of this kind could almost have seemed naïve. Some of the most renowned models in writing capability, such as OpenAI's GPT 4.5 and Moonshot's Kimi K2, are already considered outdated. Around these releases, models have transitioned from acceptable to human-level performance in other tasks like coding and mathematics. However, writing well seems to be an orthogonal skill to most of these. Writing is probably not simply being overlooked: it is difficult, and it lacks good training data for specific intervention.

There are certainly easy opportunities to improve AI models in writing, such as specialized harnesses like Claude Code, prompts, and training environments that make models spend significantly more inference tokens on output, but this will probably not have a multiplicative impact on their capability. Writing well is a very challenging task! It is unfortunate that the inference scale has not been unlocked for one of the great intellectual quests.

Current Limitations of AI Models

Today, models seem really poor at long-form technical writing. They can produce a correct sentence, but if you try to get them to write an entire chapter, it will be a mix of confusing formulations, poorly organized, and generally a bit off. They try to be too clever where it is unnecessary and, in the process, make random conceptual errors. Models will improve significantly on small mistakes in the near future, especially as they grow larger, allowing them to retain more knowledge, but their ability to utilize that knowledge is unlikely to transform.

For example, GPT models have been incredible at finding typos and minor issues for a long time. In one test, GPT 5.5 Pro was given the almost-final draft of a 200-300 page manuscript in PDF format and found surprising minor typos in it. In contrast, Claude models have been much more helpful as editors. They have much better taste, tend to understand the mental model of the task better, and offer more interesting suggestions to unblock various forms of writer's block.

These examples all share a consistent theme. The models know how to check each unit of content, usually a sentence, equation, or figure, or create a specific section where you are stuck. With these skills, they do not do a good job revisiting components and linking them together, as they pile many things on top of each other. This resembles a kind of irreducible errors that accumulate. These errors used to be common in mathematics and code, but RLVR has been a true magic solution for reducing them.

Extracting Value from Current Models as a Writer

In practice, a few phrases of technical explanation in a manual can come from an AI model, well under 1% of the text, when they are genuinely good. Including a few AI tokens in a book is not cheating if a true expert judges that the phrase is what the reader needs. It is especially in the editing process, which demands a very careful eye, that this path proves extremely valuable. For instance, when an editor's questions are interwoven in a LaTeX file with a specific delimiter like \editor{}, one approach is to let Claude Code navigate to each comment, print the context before and after, and indicate whether it is a simple typo correction or something more nuanced. What remains is to write a response, that is, the text to insert, or to ask Claude for suggestions before making corrections. Intellectually, this is a very focused editing process and a fun way to enhance a book. Sometimes, phrases from Claude's suggestions find their way into the book.

It is definitely a slippery slope: accepting a few AI suggestions becomes tempting during the second complete revision of a manuscript, when the project feels emotionally finished but there is still work to do. It is much more enjoyable to write in a way that is solely you—voice-driven, so highly valued for the process—but writing a standard manual is not really an activity known for being fun. It is easy to understand why people turn AI tools into crutches when most of their writing is just a product to fill space, rather than a means to achieve a goal. Writing abundantly remains a way to learn, feel, and express.

Similar balances arise in scientific work. AI models are excellent for repetitive parts of the paper, such as drafting a related works or context section that one knows by heart, but using them for the abstract, introduction, experiments, or conclusion is a shame. That is where the story and soul of the work are communicated—where you learn what your research truly means.

Having AI models generate and verify non-fiction writing creates much more net value. They make writing equations trivial, can help refactor the repository, translate between languages, and much more.

AI becomes crucial when simultaneous Markdown and LaTeX versions of a book have to be maintained in two places, while readers provide feedback on the web version and an editorial team reviews a derived copy. Without AI agents, synchronizing between the two would easily take five times longer, for a task that already runs to dozens of hours.

Another observation, tied to the use of agents: your pace of understanding will not accelerate by using agents. This understanding, in the form of intuition, taste, instinct, etc., is what will be valuable in the future. Using AI for non-fiction writing takes away from that progression. Moreover, if you were not already an expert, you would not be able to grasp its flaws.

When the urgency is to pour knowledge onto the page, using AI models can be a valuable tool. A manual makes full sense when it offers a unique reference for important post-training methods like rejection sampling or character training, where there are very few resources.