Mastering Token Costs in Agentic AIs

Identifying Hidden Token Costs in Agentic Loops

In the world of agentic artificial intelligence systems, the costs associated with tokens can quickly become a major issue if the necessary precautions are not taken. These costs often accumulate insidiously, especially when workflows involve multiple steps. This article examines the reasons for this accumulation and proposes architectural models to better control them before they reach critical levels.

Topics Covered

The article addresses several key points: why token costs increase non-linearly in multi-step agentic workflows, and how the distinction between state and context is essential for managing them. It identifies five distinct failure modes, ranging from O(N²) context accumulation to static system prompt duplication, which explain most of the excessive token spending in production deployments. Practical solutions are proposed for each pitfall, including context compression, circuit breakers, payload filtering, dynamic model routing, and real-time prompt injection.

The Central Problem

Creating a single-turn LLM wrapper may seem simple, but preventing an autonomous agent from compromising your infrastructure over an extended period is a far more complex challenge. The fundamental problem lies in the fact that each time an LLM processes text, it charges in tokens—those small units of text that are essential for models to read and write. Tokens are comparable to the units measured on your cloud bill: the more tokens you send per API call, the higher the bill. This remains relatively straightforward for a chatbot. However, in an agentic loop, where an AI autonomously calls tools, reads results, and plans its next actions through numerous steps, token costs do not grow linearly. They accumulate. A naive setup that integrates every tool output into a constantly growing message array can turn a $0.05 automation task into an infinite loop costing $5.00 without triggering a single error.

The solution begins with a clear mental distinction between state, which is the minimum set of facts needed to advance the task, and context, which is the complete and detailed transcription of everything that has happened so far. Most agentic frameworks conflate these two notions by default, and assessing which frameworks are worth your time before architecting around them is crucial. The five cost traps below illustrate what this state/context confusion looks like in production.

1. O(N²) Context Accumulation Tax

The Concept: In an agentic loop, transmitting the complete conversation history with each model call means you are paying for the same historical tokens multiple times, not just once.

How It Works: Most orchestration frameworks by default add each user, assistant, and tool message to a single growing array. By the 20th step of a 20-step workflow, the model is re-reading everything from steps 1 to 19. The solution is context compression: reducing previous turns into a dense summary or using KV-cache prompt caching to freeze the prefix state and only pay for the delta—a direct consequence of how attention mechanisms evolve with sequence length.

Note: Compressing too aggressively can lead to "contextual amnesia." The agent loses a critical parameter it retrieved at step 2, hallucinates a replacement at step 8, and cascades into a chain of failed tool calls.

When to Use It: Apply context compression to any multi-step workflow expected to exceed five turns or interact with heavy data and high-latency external APIs.

2. Unlimited Retry Loops on Stale State

Context bloat is not just an accumulation problem. It actively worsens when things go wrong.

The Concept: When a tool call fails, the agent tries to correct itself but drags the bloated context of the failure into each retry, increasing costs with every attempt.

How It Works: A standard ReAct (Reasoning and Action) loop catches an exception—e.g., a 400 Bad Request error—and adds the error trace to the context before asking the model to correct it. If the agent gets stuck, each retry also sends all previous failures. The solution is a circuit breaker at the orchestrator level: removing failed trajectories from the state before presenting the error to the model, or completely halting execution after a threshold.

Note: Completely removing the history of failures means the agent will likely repeat exactly the same invalid tool call. You need to extract and inject a deterministic "failure heuristic" (e.g., "Tool X failed because parameter Y was missing") rather than the raw stack trace.

When to Use It: Apply circuit breakers and trajectory sizing on all non-deterministic external API calls where the model dynamically generates the payload.

3. Unfiltered Tool Payload Bloat

With retry loops under control, the next point to examine is what is introduced into the context in the first place—specifically, the raw output from your tools.

The Concept: Feeding raw, unparsed API responses directly into the agent's context wastes tokens on structural elements and fields that the agent will never use.

How It Works: An agent queries a database or third-party API and receives a massive JSON payload. Instead of dumping this raw JSON into the prompt, pass it through a deterministic extraction layer (jq, a regex filter, or a dedicated parser) that removes metadata, null fields, and boilerplate. What enters the context should only be the key-value pairs validated by the schema that the agent actually needs to proceed.

Note: If the extraction layer silently removes a field that the agent needs downstream, it will silently hallucinate a plausible value to fill the gap—and that value goes directly into your database writes.

When to Use It: Deploy a payload filtering middleware whenever an agent integrates with legacy systems, verbose REST APIs, or unstructured web scraping tools.

4. Monolithic Model Routing

Once your context is cleaned up and your payloads are filtered, there remains a cost lever that most engineers overlook: which model is doing the work.

The Concept: By default, using your most performant (and expensive) model for every step of a workflow—including trivial tasks like formatting a JSON object or classifying an intent.

How It Works: An agentic workflow is actually a directed graph of heterogeneous tasks. Complex semantic reasoning and planning require a heavy model. But for nodes dealing with intent classification, JSON formatting, or schema validation, the orchestrator can dynamically route to a smaller, less costly model (e.g., Llama 3 8B or GPT-4o-mini) at a fraction of the token cost.

Note: Routing adds orchestration overhead. If your system has to load a different model into VRAM or open a new provider connection at each step, the latency time can negate the savings achieved.

When to Use It: Dynamic model routing is beneficial in high-throughput multi-agent systems where the workflow graph contains clearly isolated nodes for deterministic data transformation.

5. Static Context Duplication

The final pitfall occurs right at the beginning of each API call, in the system prompt itself.

The Concept: Injecting a massive system prompt covering every tool definition and every edge case into every API call, even when most of them are irrelevant to the current step.

How It Works: Instead of loading a 5,000-token system prompt defining 20 tools, build your prompts dynamically using the techniques discussed here. The orchestrator maintains a lightweight vector index or rule engine of available tools and constraints. At runtime, it injects only the tool definitions and behavioral directives that the current step actually needs—nothing more.

Note: Dynamic context injection opens a prompt injection vulnerability if the search request is influenced by unreliable user input. A malicious request could cause the orchestrator to retrieve and execute a tampered tool definition.

When to Use It: Switch to dynamic prompt building when the number of tools in your agent exceeds a dozen, or when you are running multi-tenant systems with distinct role-based access controls.

Managing Token Costs in Production

These five pitfalls share a common root cause: treating context as unlimited. Once you start managing it deliberately—by compacting history, trimming failures, filtering payloads, routing by task complexity, and injecting only what each step needs—the cost profile of your agentic system changes significantly.

But reducing your token consumption at runtime is only the first problem. By day 100 in production, you will face rising infrastructure costs related to state management. Storing massive uncompressed agent trajectories for observability or crash recovery will bloat your storage and quickly degrade query latency. Implement aggressive TTLs on session states and cold storage archiving for long-term audit logs, so your operational database contains only active and prioritized states.

Tokens are the computational currency of agentic systems. Treating them as a free resource is a surefire way to fail in production. Do not expect model providers to lower their API prices. Architect your orchestration layer to treat context as a constrained and volatile resource from day one.