Enterprise LLMs Must Clarify Analytical Choices

In enterprise data systems, the correct answer does not solely depend on accurate calculations. Recent benchmarks and studies highlight the limitations of similarity rankings and very long contexts. The challenge is to clarify analytical choices before any execution.

Benchmarks and Studies Highlight Clear Limitations

Spider 2.0, an enterprise text-to-SQL benchmark covering 632 real-world cases and published at the end of 2024, initially recorded a resolution rate of about 17% for its best-performing system, an agent based on o1-preview, while the top systems on the original Spider reached nearly 91%. The hierarchy later changed, with a Snow track seeing a provider submit a score of 96.7% for an agent. These benchmarks primarily measure success on defined tasks and provide little information on the ability to recognize an ambiguous question before answering it. Meanwhile, research conducted by Chroma on the decline of contextual relevance revealed that model effectiveness decreased as input size increased, even for basic tasks, and that the presence of distractors close to relevant data exacerbated this phenomenon. Adding more documents may enrich the amount of available information without necessarily improving the ability to spot relevant evidence, while the real question is to identify which data is crucial, for what reasons, and how it relates to the request.

The Root Problem: A Question Hides an Analytical Program

A seemingly simple business query, such as "Which brand improved confidence this quarter?", requires several implicit decisions. It is necessary to define the measure of confidence in the survey, specify the survey waves constituting the quarter, delineate the population of respondents, decide on the application of weighting, and agree on what counts as improvement. In other words, the question encapsulates an analytical program whose parameters have not yet been set. Without these choices, two analysts may produce different answers without any calculation error.

Correct Calculation, Wrong Answer: Ambiguity Creeps In Before Execution

In response to a query, an enterprise LLM can identify a table titled "Brand Confidence," extract the latest values, and compare two periods. The calculation and explanation may be flawless, while responding to a different interpretation than expected. The table may indeed reflect the share of maximum ratings while the expectation is for an overall KPI. The available period may only cover part of the necessary waves for a quarterly comparison. A result that seems relevant by similarity is not necessarily valid for the intended task.

At Scale, the Multiplicity of Candidates Complicates Discovery and Resolution

In an enterprise environment, multiple tables may carry the same metric, and several waves may resemble the requested period, while cross-tabulations exist by market, segment, question variant, or weighting scheme. The correct answer is out there, but buried among alternatives. The system then faces two distinct challenges: locating where the answer might be and deciding which candidate truly aligns with the user's intent.

The "Context Genius" Targets Upstream Analytical Choices

Anthropic presents the context genius as an advanced form of prompt engineering, focused on managing an agent's context and considering this context as a limited resource whose effectiveness decreases as it is utilized. This approach looks at the elements included in the context window as well as methods to keep it relevant and focused. The problem addressed here occurs upstream: clarifying the meaning of the question and admissible evidence. Neither the retrieval of similar content nor a semantic layer necessarily determines the definition intended by the user or the appropriate table among close candidates. A structured path is needed, from intent to evidence. Specifically, this involves establishing definitions, valid reporting periods, included populations, potential application of weightings, and authoritative sources, then seeking clarifications in case of ambiguity. The goal is to clarify the intended analytical operation, not just to ensure understanding of the words. The example of "hours per day" illustrates that a correct calculation can rest on a faulty denominator; conventional checks validate the arithmetic but not always the analytical decisions, which require verifying population, filters, denominator, and relationships between sources.