AI Support: Hybrid Architecture Surpasses Chatbots

Rethinking AI Support with a Hybrid Architecture

In a context where companies are looking to optimize their customer service, a client recently expressed the need for a chatbot that is not only secure and accurate but also capable of providing near real-time responses. This type of request is becoming increasingly common as businesses recognize that data security, latency, and response quality have a direct impact on their financial outcomes. AI systems that fail to meet these requirements risk losing user trust and incurring significant costs. According to an IBM report, the global average cost of a data breach could reach $4.44 million by 2025.

However, generic chatbots and large language models (LLMs) available on the market often fail to meet the expectations of businesses. Although powerful, these models are limited by constraints such as token limits, context usage, and hallucinations. These limitations become particularly evident when specific domain knowledge and strict response formats are required. The question then arises: how can we build an AI that can respond like an expert, relying on real data, while remaining fast, secure, and controllable?

My experience in model development has taught me that the solution does not lie in a single model or technique. A broader architectural approach is needed that separates what the model knows from how it responds, while combining learning and retrieval.

Main Challenges

Effective Context Limits

Modern language models boast context windows of up to 128,000 tokens. However, in practice, their attention capacity degrades well before reaching these limits. When a large volume of text is used as context, models tend to underutilize information located in the middle of the prompt, a phenomenon known as the primacy-recency bias.

Increasing context size does not guarantee better responses. In enterprise environments, where knowledge bases can extend over millions of tokens, this approach is not viable.

Weak Utilization of Long-Tail Information

LLMs can overlook relevant information, misinterpret it, or overemphasize irrelevant sections, even when the correct content is present in the prompt. Studies like "Lost in the Middle" support this theory, showing that long-context inputs often lead to incomplete reasoning if not carefully controlled. This makes naive strategies of "dumping everything into the prompt" unreliable for complex, domain-specific support systems.

Trade-offs Between Accuracy and Performance in Retrieval

Retrieval introduces real latency and computational costs. If you retrieve too much, the increased context raises response time and dilutes the model's attention. If you retrieve too little, you increase the risk of hallucinations. The real challenge is not retrieval itself, but precise retrieval — ensuring the minimum sufficient context required for accuracy without overwhelming the system or the model.

Hallucinations in the Absence of Context

LLMs rarely refrain from generating responses even when relevant information is missing. Instead, they respond confidently with generic or fabricated answers. In a support framework, this behavior is unacceptable. It directly impacts trust, accuracy, and compliance.

A thorough examination of these constraints clearly showed that adding more context was not the solution. We needed a smarter architecture.

The Answer: A Hybrid Architecture

Our work consistently pointed toward a hybrid approach combining retrieval-augmented generation (RAG) with finely-tuned language models. The key idea was that fine-tuning and retrieval solve different problems. Fine-tuning teaches the model how to respond, while retrieval provides what to respond with. We found that forcing one method to do both leads to inefficiency, instability, or high costs. Therefore, we designed a system that allows these two components to operate from their respective strengths.

Using RAG for Accuracy Through Retrieval

We deliberately avoided overwhelming the model with large volumes of raw documents. Instead, we built a searchable and curated knowledge base from our internal question-answer pairs, product manuals, technical documentation, and reference materials on policies and configurations. At inference time, the retriever selects only the most relevant pieces of content and inserts them into the prompt. This ensures that responses are grounded in verified real data.

We found that this approach significantly reduced hallucination rates, improved factual accuracy, and increased response speed by keeping context windows small and specific to queries. However, relying solely on RAG proved insufficient. Even when retrieval accuracy was very high, outputs exhibited significant variability in tone, structure, formatting, and level of procedural detail included. These outputs indicated that, while factually accurate, the responses were neither consistently structured nor reliably formatted.

In a chatbot use case specifically designed to enhance request convergence, evaluation showed that even though the small language model had access to nearly 100% correct context, the accuracy of outputs was only about 70%. The model struggled to extract meaning from long-context inputs and failed to maintain the conversational tone necessary to guide users toward deeper technical discussions or follow-up meetings.

This revealed a fundamental limitation of retrieval: RAG provides information but cannot instruct a model on how to reason or communicate in a specific domain.

Fine-Tuning Qwen: Teaching the Model How to Respond

To improve consistency, tone, and reasoning, we fine-tuned the Qwen model on approximately 1,000 carefully selected expert question-answer pairs aligned with the model's target domain. The goal was not to teach the model facts. We wanted it to learn how to apply domain-specific language, maintain the company's voice and communication style, follow a consistent response format, reason procedurally, and handle edge cases that arise in support workflows.

Fine-tuning adjusts a model's behavior, not what it knows — and this distinction is important. Fine-tuning the entire model can lead to catastrophic forgetting and excessive computational costs. To mitigate these risks, we used low-rank adapters (LoRA). LoRA adapters allow for fine-tuning a small set of adapter matrices while preserving the majority of the base model's general knowledge. They also reduce the GPU memory required for fine-tuning and produce performance nearly equivalent to that of a full model fine-tuning.

The results were clear. The model became substantially more coherent and nuanced. For stable and procedural questions, it produced correct responses many times without requiring retrieval. However, as expected, the model struggled with questions about new features, updated policies, and long-tail factual queries.

In the same chatbot example, fine-tuning improved tone alignment to about 90%, but decreased factual accuracy to around 50%. The lesson was once again reinforced: fine-tuning does not replace retrieval.

Why Neither RAG Nor Fine-Tuning Alone Were Sufficient

These experiments provided clarity on the trade-offs involved:

• RAG-only systems offer better factual grounding and timeliness but suffer from poor tone consistency and higher latency.

• Fine-tuned-only systems provide superior voice and structure consistency but fail when knowledge changes or long-tail facts are needed.

Choosing a single approach meant accepting the weaknesses of the other. Combining a fine-tuned model with RAG produced better results than either method alone. Tone accuracy improved to about 75% — better than RAG, which had no reliable tone control, though slightly below the 90% of the fine-tuned model. Factual accuracy improved to about 73%, surpassing both the fine-tuned model alone (around 50%) and RAG alone (around 70%). The fine-tuned model's understanding of domain context and expected output format allowed it to better utilize and apply the retrieved context than a base model could.