Local LLM Lending: Key Memory, Metrics, and Adjustments

Not all slowdowns are equal on a locally installed LLM. Between the wait for the first token and the generation speed, the causes differ and require different remedies. Tools like Ollama and llama-server help to separate prefill and decode, while certain configurations—especially a model split between GPU and CPU—impose limits that simple adjustments cannot overcome.

Generation speed primarily depends on memory and has its limits

When token generation is slow, the cause is most often related to bandwidth or memory, rather than pure computation. It is possible to estimate a performance ceiling based on memory bandwidth and model size, and then deduce whether to switch to a smaller model or to faster memory or bus. Some techniques, such as speculative decoding and MTP, aim to circumvent these limits, but they are only useful under specific loads. They can also backfire by slowing down or consuming more VRAM or context.

Measure prefill and decode separately with Ollama or llama-server

Two key indicators are essential for understanding performance: prefill, which corresponds to the evaluation of the prompt, and decode, which is the generation of tokens. The detailed outputs from Ollama display these metrics, allowing for the identification of the problematic part. The times reported by llama-server also provide these measurements, facilitating the diagnosis between latency at the first token and generation throughput.

Avoid GPU–CPU distribution and apply the right settings

An initial checkpoint is to ensure that the model is not split between GPU and CPU, as such a configuration forces both to block for each token and cannot be corrected by other adjustments. On the parameter side, comparing cache_n to prompt_n helps confirm that caching is effective. Adjusting batch sizes, including via -ub and -ubatch, can improve prefill when appropriate. Distinguishing from the outset between the wait for the first word and the generation speed helps avoid looking for a solution in the wrong place.