RAG and Context Engineering: Revolutionizing LLMs

The Challenges of RAG Systems

Retrieval-Augmented Generation systems, known as RAG, face significant difficulties when the context of a conversation extends beyond a few exchanges. The real issue does not lie in retrieving relevant information, but in managing what is actually included in the context window. A context engine is essential for controlling memory, compression, readjustment, and token limits.

The Breaking Point of RAG Systems

I personally built a RAG system that worked flawlessly until the conversation history was integrated. At that point, the system began to degrade. Relevant documents were discarded, the prompt overflowed, and the model started forgetting elements mentioned only two exchanges prior. This problem was not due to faulty retrieval or poorly formulated prompts, but rather a lack of control over the information integrated into the context window.

Such situations are not rare. For example, when the retrieved context reaches 6,000 characters but the remaining budget is 1,800, or when three of the five retrieved documents are near-duplicates, discarding the only useful document. What happens when the first exchange of a twenty-exchange conversation remains in the prompt, taking up space, long after it has ceased to be relevant?

What’s Missing in Tutorials

The answer lies in a layer that most tutorials completely overlook. Between raw retrieval and prompt construction, there is a deliberate architectural step: deciding what the model actually sees, how much of it, and in what order. In 2025, Andrej Karpathy named this: context engineering. I had built this for months without calling it that.

What Context Engineering Really Is

It is crucial to understand the terms well, as they can be confusing.

• Prompt Engineering: the art of what you say to the model — your system prompt, your few-shot examples, your output format instructions. This shapes how the model reasons.

• RAG: a technique for retrieving relevant external documents and including them before generation. This anchors the model in facts it has not been trained on.

• Context Engineering: the intermediate layer — the architectural decisions regarding the information that enters the context window, how much of it, and in what form. It answers the question: given everything that could go into this prompt, what should we actually include?

These three elements are complementary. In a well-designed system, each has a distinct role.

Complete Pipeline Architecture

A complete context engineering pipeline for RAG systems combines retrieval, memory management, compression, and token budget control to build efficient and scalable LLM applications.

Component 1: The Retriever

Most RAG implementations choose a retrieval method and stop there. The problem is that no single method dominates all types of queries.

• Keyword Matching: fast and accurate for exact terms.
• TF-IDF: manages term weighting.
• Dense Vector Embeddings: capture semantic relationships that keywords completely miss.

The retriever supports three modes: keywords, tfidf, and hybrid. The hybrid mode executes both methods and mixes their scores with an adjustable weight.

Component 2: The Re-Ranker

Retrieval gives you candidates. The re-ranker decides the final order. It applies a weighted sum to two factors, blending the retrieval score with an importance value based on tags.

Component 3: Memory with Exponential Decay

This is the component that most tutorials completely overlook, and where naive systems collapse the fastest. Conversational memory has two modes of failure: forgetting too quickly or forgetting too slowly. The solution is exponential decay, where exchanges continuously fade based on three factors.

Token Budget Under Pressure

How the token budget is distributed across exchanges is crucial for maintaining the system's efficiency.