Prompt Compression: Optimizing LLMs Without Sacrificing Essentials

Prompt Compression: Optimizing LLMs Without Sacrificing Essentials

What is Prompt Compression?

Prompt compression is the process of shortening a prompt without removing the necessary information to accomplish the task. It eliminates repeated, irrelevant, or low-value content while retaining the main instructions, important facts, and useful context.

The goal is not to create the shortest prompt possible, but to reduce the number of tokens without diminishing the quality of the responses. A well-compressed prompt should still help the model understand the task, follow the instructions, and produce an accurate response.

Types of Prompt Compression Techniques

Prompt compression can be achieved in several ways. Some methods remove entire phrases or documents, while others shorten phrases, eliminate low-value tokens, or rewrite the complete context in summary form.

The main techniques include:

1. Manual Prompt Rewriting
   Manual prompt rewriting is the simplest compression technique. It removes repeated instructions, filler words, and unnecessary explanations.
   Compressed version:
   Respond using only the provided context.
   This method works well for system prompts, reusable models, evaluation prompts, and tool instructions. However, it requires manual effort and may not adapt well to large or changing inputs.

2. Structural Compression
   Structural compression transforms long text into a compact format. It uses bullet points, tables, key-value pairs, JSON, or YAML.
   Compressed version:
   plan: Gold
   duration_months: 18
   monthly_expense: 240
   support_contacts_30d: 5
   This method is effective for client files, claims, product details, and tool outputs.

3. Sentence-Level Filtering
   Sentence-level filtering removes complete sentences that are not relevant to the task. It retains only the sentences that help answer the user's question.
   This technique is useful in RAG systems, reports, policies, and long conversations.

4. Phrase-Level Compression
   Phrase-level compression removes unnecessary words from a sentence while preserving its main meaning.
   Compressed version:
   The request was submitted after the deadline and cannot be processed.
   This method keeps the text readable and reduces token usage.

5. Token-Level Filtering
   Token-level filtering removes individual words or tokens that have little value.
   Compressed version:
   Client, Gold membership, five support contacts last month.
   This method can reduce prompts more aggressively than sentence-level filtering.

6. Extractive Compression
   Extractive compression selects the most important parts of the original prompt.
   This method is useful for RAG systems, document analysis, and policy-based applications.

7. Abstractive Compression
   Abstractive compression rewrites long content into a shorter summary.
   This method is useful for discussion histories, reports, meeting notes, and long documents.

8. Query-Aware Compression
   Query-aware compression retains information based on the user's current question.
   This technique is very useful in RAG systems and for answering questions about documents.

9. Coarse-to-Fine Compression
   Coarse-to-fine compression reduces a prompt in multiple steps.
   This method offers better control over the compression.

10. Soft Prompt Compression
    Soft prompt compression converts long text into a small set of learned vectors.
    This method can significantly reduce the size of the context.

Prompt Compression in RAG Systems

Prompt compression is highly beneficial in retrieval-augmented generation (RAG). A RAG system retrieves documents and adds them to the prompt before generating a response.

A compressed RAG pipeline works as follows:
Compression can remove irrelevant documents, select important paragraphs, and retain only the sentences that support the response.

Prompt Compression for AI Agents

AI agents often build long prompts over time. Prompt compression helps agents retain only the necessary information for the next step. Old tool outputs can be summarized, complete actions can be stored as short status updates, and repeated instructions can be eliminated.