Refining an LLM: 98% on a 7B, LoRA Hits a Wall at 1.4 TB

Making a LLM reliable on complex forms is expensive in terms of context and human corrections. A project reports that a fine-tuned Mistral 7B achieves about 98% accuracy on medical reports, compared to 35% with prompts and RAG, while eliminating estimated API costs of $320,000. However, fully fine-tuning a giant model requires up to 1.4 TB of VRAM and can degrade acquired skills, highlighting the practical interest in LoRA/QLoRA.

Full fine-tuning requires up to 1.4 TB of VRAM and can degrade the model

Full fine-tuning of a large model proves to be prohibitive in practice. For a model with 70 billion parameters in FP16, it is necessary to store in VRAM the weights as well as two additional moments per parameter related to the Adam optimizer. The requirements then rise to between 1.2 and 1.4 TB of VRAM, a threshold largely inaccessible for most teams. Beyond the hardware cost, full fine-tuning can also destroy existing knowledge due to the updating of all parameters and the risk of overwriting general language and reasoning capabilities. It is precisely this double challenge – resources and stability – that motivates the use of LoRA, an approach described by Hu et al. to effectively specialize a foundational model on downstream tasks.

On a breast cancer synoptic report, a fine-tuned 7B achieves about 98%

A team reports having fine-tuned a 7 billion parameter model to fill out synoptic breast cancer reports, a task requiring a strict order of fields, highly branched logic, and the identification of a histological type among 40 subtypes without hallucination, under penalty of invalidating the entire output. With aggressive system prompts and some RAG on Claude Opus 4.6, accuracy peaked at about 35%, at the cost of a context of around 30,000 tokens and outputs containing omissions, unnecessary subsections, and hallucinations requiring a complete review. After fine-tuning a Mistral 7B via QLoRA, the team reports an accuracy of about 98%, which is 63 percentage points better, while eliminating API costs for this task. Operating expenses at scale with a cutting-edge model had been estimated at around $320,000. The team clarifies that this "free" result does not include the cost of fine-tuning, large-scale local execution, or energy measurement per call.

When to prioritize fine-tuning: rigid formats, context costs, and overlapping rules

Fine-tuning is essential when the output format allows for little error: enterprise models anchored with multiple conditional branches, legal documents specific to each jurisdiction not seen in pre-training, or redundant medical forms that must be filled out perfectly. At scale, system prompts of several thousand or tens of thousands of tokens per call increase costs and latency, while a fine-tuned model can incorporate these patterns and do without this ballast. System prompts also show their limits when rules overlap and the decision tree explodes, as illustrated by a case where it became unreasonable to encode everything in a table. The quality of context declines with length; a prompt accumulating 50 rules risks omitting some and invalidating the output. Finally, to impose a brand voice in customer support, fine-tuning often works better than repeated instructions: maintaining a particular tone requires a dedicated prompt for each exchange, and a template of 2,000 tokens quickly drives up the bill.

What RAG does better and the option of a mix

An empirical rule among practitioners is that RAG primarily serves to augment the model's knowledge when the base changes frequently, when access to evolving documents and policies is crucial, or when only a few hundred training examples are available. RAG can also suffice if a small system prompt already reliably modifies behavior. On the other hand, fine-tuning mainly affects output behavior, while it can, in some cases, inject new knowledge. The boundaries are not clear-cut and depend on judgment. In practice, effective solutions combine both: the team that enhanced its 7B did not completely eliminate system prompts or RAG, but significantly reduced its reliance on these levers.

SFT objective and LoRA mechanics: low rank, A/B matrices, and gamma

Fine-tuning extends the pre-training objective, which generally consists of predicting the next token. In supervision (SFT), the same causal objective applies to prompt-completion pairs, with many configurations calculating the loss only on response tokens. Mathematically, this involves cross-entropy on the tokens; the difference mainly lies in the dataset, massive and raw in pre-training, targeted in fine-tuning. It is also possible to add knowledge through fine-tuning without resorting to prompt-completion pairs, by introducing new data to the same objective. Mechanically, LoRA freezes the original weights and adds a low-rank update: instead of updating a complete matrix (for example, 4096×4096 in a transformer projection), it learns small dimension matrices A and B that compose the correction. The forward pass is expressed as h = W0 x + (alpha/r) B A x. In libraries like Hugging Face or PEFT, the term alpha/r is replaced by an equivalent hyperparameter gamma. Understanding these elements – particularly the choice of rank (8 instead of 32, for example) and the effect of alpha – avoids "blind fine-tuning" and makes debugging more methodical.