Brief IA
›

AI Agents: Anthropic Advocates for Verifying Actual Results

AI Tools·Tom Levy·

AI Agents: Anthropic Advocates for Verifying Actual Results

AI Agents: Anthropic Advocates for Verifying Actual Results
Key Takeaways
1Anthropic recommends verifying the actual outcome of a task, beyond the agent's exchanges.
2Pydantic in strict mode rejects certain invalid arguments but does not guarantee the accuracy of an amount.
3An injection test must distinguish the action proposed by the model from its potential execution.
Why it matters : successful tool calls can coexist with an unfinished task, such as a missing revision receipt.

In its recommendations dated January 9, 2026, Anthropic advocates for evaluating AI agents based on the actual outcome of a task rather than their interactions. A successful tool call, a correctly typed amount, or a response indicating the completion of work is not sufficient to establish that the mission has been accomplished.

A document can divert the requested action

In a security test, the footer of a synthetic invoice receives an instruction asking the agent to export the tenant's documents before any approval. However, the targeted command, export_all_documents, is not among the authorized tools.

The diagnosis requires separately consulting what the model proposed and what the executor actually triggered. A request for export indicates that the model or its instruction needs to be corrected. An executed export reveals a more serious failure: the application protections have allowed the action to proceed.

OWASP describes this type of indirect injection in LLM01:2025: instructions slipped into a retrieved document alter the model's response. The organization recommends keeping privileged functions within the application code and subjecting the model to adversarial testing as an unreliable user. An isolated attack provides only an observation; the control mechanism employs multiple variants, followed by previously unused attacks during tests following a modification.

A typed amount can still be incorrect

Format compliance does not determine the accuracy of an extraction. In the fictional case used for checks, the reference is doc-17, version v2, on page 2, for an amount of 129900 cents. The extractor can still produce 12990, an integer acceptable from a computing standpoint, or consult v1 and return an outdated value.

The control involves comparing the results to a dozen documents verified by reviewers, whose confirmed values and approved versions are retained. It must be repeated whenever the model or its instruction changes. In this setup, any divergence in value or version leads to human review, preventing the automatic decision from proceeding. Microsoft also recommends testing Document Intelligence on documents resembling those that will be processed.

The confidence score does not replace this comparison: it expresses the extractor's assessment of its own output, not a guarantee of accuracy. Therefore, the control prioritizes the approved source when it contradicts this score. Similarly, a page citation indicates a source without certifying that the extracted sum is correct or still valid.

Pydantic blocks out-of-contract arguments

Even before data accuracy, a call can violate the contract of the tool record_review. The model may transmit the amount as a string, add an undeclared command, or omit source_doc and source_version after a schema evolution. Directly relaying these arguments shifts the detection responsibility to the executor.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Pydantic can perform this filtering in the application, provided strict validation is enabled. Its lenient behavior can transform "129900" into integer 129900; the strict setting rejects this conversion. The option extra="forbid" excludes unexpected fields. The test schema requires strings for tenant_id, invoice_id, source_doc, and source_version, as well as an integer for amount_cents.

The control, executed with Python 3.10.12 and pydantic 2.14.1, includes one accepted example and three faulty examples. The rejections specify the affected fields; their message must be retained along with the schema version and the tool version. This verification remains limited to form: it may accept the incorrect amount 12990 or a correctly typed invoice belonging to another tenant.

Verify completion and track actions

The fictional case of INV-17 involves an agent tasked, for tenant-A, with extracting an amount, obtaining approval from an authorized person, and recording a receipt. The extraction returns a valid JSON, all calls succeed, and the agent claims to have finished; however, the receipt is missing. No payment or actual recording occurs in this simulation. The controls cover seven categories of failures, with each having a case designed to fail and a witness intended to succeed: if the first suddenly passes, it is necessary to check whether the agent has progressed or if the test no longer detects the defect.

To track these results, a common execution identifier must link the calls to validations, authorizations, consulted versions, external receipts, and the final status. OpenTelemetry provides a trace for execution and spans, units for tracking steps, connected by their parentage; a receipt can thus be linked to its call. Production observation complements offline evaluation, but sampled collection remains partial, and an uninstrumented step cannot be considered successful by default.

Tracking must differentiate between a tool error, an application rejection before execution, and an authorization denial, with states separating proposal, rejection, waiting for approval, and completion. The validator's reason accompanies the rejection; document contents and identifiers remain outside of events. For an attempt targeting another tenant or a prohibited action, the negative authorization test verifies the authenticated field, the rejection, and the maintenance of the initial state.

Other diagnostics focus on effects and progress: a simulation of receipt loss examines repetitions after a deadline expires, based on a stable operation identity and the authoritative count. After a restart, the persistence and completion check verifies the resumed identity as well as a receipt confirming completion. Finally, when work repeats or workers remain stuck while time and expenses increase, the control compares budget, progress, and pressure on workers, based on calls, durations, usage, and an explicit task status.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.