Multi-Agent AI: Effectively Reducing Token Costs

The Issue of Token Consumption in Multi-Agent AI Systems

In the field of artificial intelligence, the use of multiple agents to collaborate on complex tasks has become common. However, this approach can quickly lead to a significant increase in the number of tokens used. Tokens, which are units of text or information, accumulate through various elements such as memory logs, tool specifications, and system instructions. This accumulation can not only slow down processes but also exhaust allocated computing budgets.

For developers and AI practitioners, effectively managing token usage has become crucial. Fortunately, there are solutions to optimize multi-agent architectures without necessarily increasing costs. This article explores four strategies that allow for a reduction in token usage while maintaining system efficiency.

Strategies to Optimize Token Usage

Here are four commonly adopted approaches to improve the efficiency of multi-agent AI solutions in terms of token management.

1. Caching Static Instructions

The first strategy relies on caching static instructions, also known as Prefix-Match Caching. Large Language Models (LLMs), which form the core of modern AI agents, often spend a lot of time re-reading the same instructions during each interaction. By storing these instructions as key-value pairs, prefix caching allows for long and static directives to be retained as references. Thus, instead of re-reading the entire instruction manual for each request, the model can simply access a summarized state and focus on the new task. This method significantly reduces preparation latency and decreases token costs.

2. Semantic Caching: Intent-Based Recall

Semantic caching is another effective strategy. It is based on the idea that if an AI agent has already solved a similar problem, there is no need to generate a new response from scratch. This approach uses embeddings, which are numerical representations of text, to quickly identify similar intents from the past. For example, queries like "How to reset my router?" and "What steps to restart my wifi box?" can be recognized as having the same intent through semantic caching. This sometimes allows for completely avoiding the use of LLMs while still providing an appropriate response.

3. On-Demand Tool Usage

The third strategy, known as lazy loading, aims to avoid preloading vast reference manuals for every available API and tool. Rather than overwhelming agents with unnecessary information, it is more efficient to provide a concise directory of their capabilities. The agent retrieves detailed instructions and necessary parameters only when a specific task requires them. This approach reduces token consumption by loading only the information needed at the right moment.

4. Task Escalation: Cost-Effective Model Routing

Finally, task escalation is a method that involves analyzing each user request to determine the most appropriate model to use. Not all tasks require the involvement of massive and costly models. Efficient multi-agent AI architectures function like sorting centers, directing simple tasks to lightweight and free models capable of handling them locally. More powerful models are reserved for complex tasks requiring deep reasoning or multi-step orchestration.

Example of Practical Implementation

To illustrate the application of these strategies, let's examine an implementation example combining semantic caching and model routing. The following code uses a sentence transformation model to convert text into the embeddings necessary for semantic caching. Calls to LLMs are simulated but can be replaced with actual free models available on platforms like Groq.

import numpy as np
from sentence_transformers import SentenceTransformer

# Loading a free local model to convert text into embeddings
embedder = SentenceTransformer('all-MiniLM-L6-v2')

# In-memory semantic cache and similarity threshold (0.90 = 90% similar)
semantic_cache = {}
SIMILARITY_THRESHOLD = 0.90

def cosine_similarity(vec1, vec2):
    """Calculates how closely related two queries are."""
    return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))

def route_and_respond(user_query):
    # 1. Convert the current query into an embedding vector
    query_vector = embedder.encode(user_query)

    # 2. Semantic caching: Check if a similar problem has been solved recently
    for cached_vector, past_response in semantic_cache.values():
        if cosine_similarity(query_vector, cached_vector) >= SIMILARITY_THRESHOLD:
            return f"[Served from cache] {past_response}"

    # 3. Model routing: Triage the task based on its complexity
    # Simple tasks are routed to a free, locally hosted model (e.g., Llama 3 via Ollama)
    # This routing logic is illustrative only; do not use in production
    if "summarize" in user_query.lower() or len(user_query) < 100:
        response = call_free_local_agent(user_query)
    else:
        # Complex multi-step reasoning is escalated to a larger orchestration agent
        response = call_heavy_reasoning_agent(user_query)

    # 4. Store the new vector and response in our cache for future users
    semantic_cache[user_query] = (query_vector, response)

# --- Simulated agent functions for illustration: no real LLM invoked here ---
def call_free_local_agent(prompt):
    return "Action completed by a local model, at no cost."

def call_heavy_reasoning_agent(prompt):
    return "Action completed by a complex orchestration agent."

# Example usage: simulating the alternative use of different agents/models
# Comment/uncomment to try both examples and test your own
print(route_and_respond("Summarize today's server logs"))
# print(route_and_respond("Draft an optimal month-long itinerary for my upcoming trip to Japan. Consider all documents, public transport schedules, and other provided documents, as well as real-time API information."))

The complexity of the task requested in the prompt passed to `route_and_respond()` will determine which type of model is used.

The output from executing this code will be one of two return messages from these functions:

```python
def call_free_local_agent(prompt):
    return "Action completed by a local model, at no cost."

def call_heavy_reasoning_agent(prompt):
    return "Action completed by a complex orchestration agent."

This article has outlined four key strategies to consider when implementing multi-agent AI applications and architectures, focusing on optimizing token usage and reducing costs and latency. Through a practical example based on simulations, we reinforced our understanding of applying two of them in combination: semantic caching and model routing.

Iván Palomares Carrascosa is a leader, writer, speaker, and advisor in AI, machine learning, deep learning, and LLMs. He trains and guides others in leveraging AI in the real world.