Operational AI: Why It Fails Despite Advanced Models

The Persistent Challenges of Operational AI

In the field of artificial intelligence, it is common for a large language model (LLM) to produce results that seem flawless at first glance. However, when these results are integrated into an operational system, problems arise. The fields are present, the structure is correct, and the values seem plausible, but something is not working. This could be an incorrect data type, a missing field, or a value that, while technically correct, is contextually inappropriate. These issues are not due to flaws in the models themselves, but rather to gaps in the architecture of the systems. As long as these questions are not addressed as architectural problems, they will persist.

Most discussions around these issues are conducted by engineers for other engineers, focusing on technical solutions such as fine-tuning models, optimizing prompts, and deployment infrastructure. While these tools are real and useful, they are not always accessible to those who need them, particularly project managers and operations leaders. The latter, although they do not directly build AI products, seek to integrate AI into their existing workflows to enhance efficiency.

For these professionals, the challenges appear from a different perspective. The modes of failure they encounter are not those found in technical manuals on LLMs. The architecture that works in practice often does not resemble that described in standard tutorials.

Understanding Operational AI Errors

The fundamental problem has never been a lack of intelligence in the systems, but rather a misunderstanding of the nature of operational tasks. With the increasing accessibility of LLMs, the idea was that these systems, capable of understanding language and reasoning, could solve operational problems. However, these problems are not related to reasoning, but to repeatability. Operations require that the same input consistently produces the same output. When AI begins to reason creatively about tasks such as processing refunds or updating records, trust in the results is compromised.

LLMs are inherently non-deterministic, meaning that the same question asked twice can generate two different answers, both correct but never identical. This variability is acceptable for a conversational assistant, but problematic for systems requiring reliable and repeatable execution. In an environment where results must be accurate and conform to precise standards, this variability becomes a structural incompatibility.

Most demonstrations show how to build a tool that works in isolation, taking an input and producing an output that appears correct on screen. However, they do not show what happens when this output needs to be integrated into another system, a database, or a downstream process that expects a specific structure. At this point, the output is no longer evaluated based on its correct appearance, but on its exact compliance with the expectations of the receiving system.

The Limits of Pure Automation

In the face of the unpredictability of LLMs, an apparent solution would be to revert to systems based on explicit rules, with predictable outputs. These systems work well as long as the world remains unchanged. However, the world is constantly evolving, and rule-based systems quickly become obsolete. Input formats, field names, and data structures are designed for one version of the world that changes as soon as automation is implemented. When these changes occur, the system does not adapt and ultimately breaks down.

Another issue is the cost of maintenance. Each exception to the rules requires human intervention, followed by rule updates, testing, and deployment. This process quickly becomes a considerable workload, turning process management into a full-time task. What is lacking in this process is human judgment, the ability to handle unexpected situations with discernment, which neither pure automation nor LLMs can replace.

Towards a Hybrid Architecture

The solution does not lie in improving LLMs, but in a better distribution of responsibilities among different systems. LLMs and deterministic systems fail for opposite reasons, and the key is to understand which part of the problem each tool is best suited to solve. LLMs excel at converting ambiguity into structure, transforming disordered inputs into normalized outputs. This process is crucial but does not constitute the complete solution.

Deterministic systems, on the other hand, are perfect for execution. With structured input, they can perform the same operation repeatedly and reliably. This predictability is their strength and what makes them reliable at scale. The hybrid model thus proposes to place each system where it is most effective: LLMs for structuring data and deterministic systems for repeatable execution.

In conclusion, for operational AI to succeed, it is essential to rethink the architecture of systems by integrating the strengths of LLMs and deterministic systems. This hybrid approach could very well be the key to overcoming the current failures of AI in operations.