LLMs Are Not Miracle Solutions: Rethink Their Use

The Limits of LLMs in Solving Complex Problems

As part of an ambitious project, I set out to transform a complex set of 100 compliance PDFs into structured JSON rules. The initial idea was simple: entrust this task to an agent capable of processing the source text, understanding the mission, and generating the expected rules. This approach, while straightforward, quickly revealed its limitations.

At first glance, the results seemed promising. The produced JSON was valid and met the initial expectations. However, a careful review uncovered significant flaws. Some rules were too general, others lacked precision, and some did not respect the subtleties of the original text. I attempted to use another agent to correct these errors, but the task proved too vast to ensure thorough verification.

This experience highlighted the fragility of this method. The errors were often subtle and difficult to detect, making this approach unviable at scale. While I cannot share the technical details of the implementation, the lessons learned from this experience could be valuable for those looking to develop robust and scalable AI systems.

Reducing Complexity to Improve Outcomes

After several attempts, I realized that improvement did not lie in a better prompt or a more sophisticated tool, but in redefining the problem itself. Instead of trying to make the agent smarter, I reduced the complexity of its task.

The first adjustment was to prepare the data in advance. Rather than letting the agent navigate a database to retrieve the necessary information, I opted for a more controlled approach by temporarily storing the relevant data locally. While this method is not always applicable, it reduces the uncertainty associated with data retrieval.

I also simplified the agent's task by eliminating metadata and other superfluous information before submitting the raw content to it. This reduction of irrelevant context helped decrease distractions and clarify the reasoning task.

The most significant change was to process the documents iteratively, one by one, rather than handling everything at once. This method made each task smaller, easier to verify, and correct. I set up five sub-agents to process the documents in parallel, each recording its progress individually. If a document failed, I could retry only that document. Moreover, if an output had formatting issues, I could correct that specific case without restarting the entire batch. The pipeline was designed to resume from the last successful checkpoint, thanks to progress caching.

Better Management of Responsibilities

The separation of responsibilities became essential. The agent was tasked with the semantic work: understanding the content, identifying relevant parts, and generating the JSON output. The surrounding code, on the other hand, managed the mechanical aspects: parallelizing tasks, enforcing the schema, generating identifiers, writing files, and validating references.

An orchestrator monitored the entire process, ensuring smooth and efficient task management. This approach made the system more reliable, not by perfecting the agent, but by structuring the workflow to facilitate traceability, validation, and output recovery.

Improving the Auditability of Results

To make the output more easily verifiable, I added reference IDs to each generated rule. Each output item could thus be linked to a specific source, facilitating audits. Instead of wondering whether a generated rule was correct, it was possible to check if the cited source text actually existed.

This method also allowed for another agent to conduct selective audits on more complex documents, ensuring that important nuances were preserved. I also implemented a lightweight version of evaluations by testing a small batch of raw documents to manually verify the coverage and accuracy of the results.

A Lesson on Using LLMs

The main takeaway from this experience is that LLMs should not be viewed as complete solutions. The system became more reliable not because the agent became perfect, but because the workflow was structured to make its outputs easier to trace and validate.

This reflection was reinforced during my participation in the inaugural AI Engineer Singapore conference, held from May 15 to 17, 2026. At this event, JJ Geewax, Director of Applied AI at Google DeepMind, emphasized the need to stop using LLMs as universal problem solvers.

For reliable production systems, a hybrid approach is often best. Let the agent handle tasks requiring semantic judgment, while the code takes care of structural, validation, and control aspects.

I will soon share more thoughts on this conference and the workshops I attended. I hope this article has been helpful, and I look forward to new discoveries soon.