Ollama: Local Optimization of Language Models Revolutionized

Ollama: A New Era for Local Language Models

Language models continue to transform how practitioners and developers in machine learning build applications. The advent of compact and efficient language models adds an interesting dimension. By bypassing third-party APIs, running models locally ensures complete data privacy, eliminates token costs associated with APIs, and allows for offline operation. Among the tools driving this revolution, Ollama has established itself as one of the standards for local inference thanks to its lightweight Go-based engine, simple command-line interface, and robust model management system akin to Docker.

However, it is rare that simply downloading a model and running it with default settings is optimal. Default configurations are tuned for a broad general audience, often at the expense of performance, deterministic reasoning, or the specific needs of systems. If you are building a coding assistant, an automated ETL pipeline, or a multi-agent system, the default configurations will likely lead to high latency, context window limitations, or random and unpredictable results.

To enhance your local AI applications, you need to understand how to adjust both model-level hyperparameters and server-level runtime environments. In this article, we will explore in depth Ollama's configuration engine, examining how to fine-tune the parameters of local language models using Ollama's Modelfile, optimize hardware performance with server environment variables, and format precise prompt streams using Go template syntax.

1. Ollama's Modelfile: Your Local Model Blueprint

Just as a Dockerfile defines how a container is built, an Ollama Modelfile is a declarative configuration file that defines how a local language model should behave. It allows you to customize system instructions, adjust model parameters, and bundle these configurations into a new reusable model variant that you can run with a single command.

A basic Modelfile consists of a base model reference (using the FROM directive), system-level directives (using SYSTEM), and parameter modifications (using the PARAMETER directive):

// Example: A Custom Developer Modelfile
# Use Llama 3.1 8B as the base model
FROM llama3.1:8b
# Set model-level parameters
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
PARAMETER min_p 0.05
# Define system persona and behavioral directives
SYSTEM """You are an elite software engineer, highly precise.
Provide concise, modular, and optimized code solutions.
Do not include conversational filler unless explicitly prompted."""

To compile and run your custom model, you use the ollama create command in your terminal:

// Create the model named 'dev-[llama](/dossier/meta-ia)' from the Modelfile
ollama create dev-llama -f ./Modelfile
// Run the newly created model
ollama run dev-llama

By encapsulating these parameters directly in the model definition, you ensure that every application or API call querying dev-llama inherits these default optimizations without needing to pass raw JSON parameter payloads in each API request.

2. Fine-Tuning Sampling Parameters

When a model generates text, it does not "know" which words to use; it calculates a probability distribution over its vocabulary for the next most probable token. Sampling parameters dictate how the engine chooses the next token from this distribution. Adjusting these parameters is the most effective way to align the model's creativity and accuracy with your specific use case.

Temperature: The Randomness Regulator

The temperature parameter controls the scale of the probability distribution of tokens. Mathematically, it divides the raw logits (pre-softmax scores) generated by the model before they are converted into probabilities:

• Low temperature (e.g., 0.1 to 0.2): Flattens low-probability options and amplifies high-probability ones. This results in very deterministic, coherent, and logical completions. Ideal for code generation, mathematical reasoning, structured data extraction (JSON/YAML), and factual synthesis.

• High temperature (e.g., 0.8 to 1.2): Flattens the differences between token probabilities, making less probable tokens more competitive. This introduces diversity, randomness, and "creativity" into the responses. Ideal for creative writing and brainstorming.

Configuring for Highly Deterministic and Structured Tasks

PARAMETER temperature 0.1

Top-K, Top-P, and Min-P: Narrowing the Token Pool

Without control, even at low temperatures, models can sometimes select very inappropriate tokens from the tail of the probability distribution. To prevent this, model engines filter the active token pool before selecting the final token.

• Top-K (e.g., 40): Restricts the pool to the next K most probable tokens. Any token ranked below 40 is immediately rejected, regardless of its actual probability. This is a rudimentary but effective method for pruning very erratic tokens.

• Top-P / Nucleus Sampling (e.g., 0.90): Restricts the pool to a dynamic set of tokens whose cumulative probability exceeds the P threshold. For example, at 0.90, Ollama sorts all tokens from highest to lowest probability and retains only the top group that constitutes 90% of the distribution. If the model is very confident, the pool may shrink to just 2 or 3 tokens; if it is confused, the pool expands.

• Min-P (e.g., 0.05 to 0.10): A modern alternative, far superior to Top-P. Instead of taking a static cumulative slice, min_p filters tokens whose probability is below a dynamic threshold relative to the probability of the main token. For instance, if the main token has a probability of 0.80 and min_p is set to 0.05, the minimum threshold for another token to be considered is 0.80 * 0.05 = 0.04. If the main token is very certain (e.g., 0.99), all other tokens are aggressively pruned. If the main token is uncertain (e.g., 0.15), the threshold drops to 0.0075, keeping a wide range of creative choices open.

Establishing Robust Sampling Limits in the Modelfile

PARAMETER top_k 40
PARAMETER top_p 0.90
PARAMETER min_p 0.05

⚠️ When using min_p, you should generally leave top_p at its default value (1.0) or set it very high (0.95+) so that it does not interfere with the superior dynamic scaling behavior of min_p.

3. Stopping Loops and Repetitive Outputs

One of the most frustrating failures in deploying local models is the repetition loop, where a model starts generating the same phrase, block of code, or sentence indefinitely. This is typically triggered by a combination of a small model size (e.g., 1.5B or 3B parameters) and a lack of penalty limits.

Ollama provides three key parameters to prevent and interrupt these looping states.

Repetition and Presence Penalties

• Repetition penalty (repeat_penalty): Multiplies the raw logits of tokens that have already been generated, making them less likely to appear again. A value of 1.1 to 1.2 is generally sufficient to discourage loops without causing the model to avoid necessary grammatical words (like "the" or "and").

• Presence penalty (presence_penalty): Applies a unique penalty to any token that has appeared at least once in the generated text, encouraging the model to introduce completely new topics or vocabulary.

• Frequency penalty (frequency_penalty): Applies a penalty proportional to the number of times a token has appeared, gradually discouraging the excessive use of specific terms.

Discouraging Loops and Encouraging Vocabulary Variety

PARAMETER repeat_penalty 1.15
PARAMETER presence_penalty 0.05
PARAMETER frequency_penalty 0.05

Stopping Generation with Stop Sequences

Sometimes, the model does not loop internally, but it does not realize when it has finished its turn, continuing to hallucinate false responses from the user. You can avoid this by defining explicit stop sequences (stop tokens). When the model generates a stop sequence, the engine immediately halts inference and returns the response.

Common stop tokens include chat markers like <|im_end|>, markdown section headers, or custom delimiters:

# Stop generating when ChatML tags or User lines are generated
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
PARAMETER stop "User:"

4. Managing Context Windows and Memory

Local hardware resources—especially video memory (VRAM) on your GPU—are very limited. Understanding how to size your model's memory structures is essential for building robust local applications.

Context Length (num_ctx)

The context length (num_ctx) defines the size of the attention window (in tokens) that the model can process at once. This includes both the input prompt (and system history) and the newly generated output tokens.

By default, Ollama initializes many models with a conservative context window of 2048 or 4096 tokens to avoid memory overflow on low-end hardware. However, modern models like Llama 3.1 or Mistral support native context windows of up to 128,000 tokens. If you are building a retrieval-augmented generation (RAG) system or importing large code files, 2048 tokens will lead to silent truncation of the prompt, resulting in loss of context and highly inaccurate completions.

You can explicitly increase this parameter in your Modelfile:

# Extend the context window to 16,384 tokens
PARAMETER num_ctx 16384

⚠️ The attention calculation grows quadratically ($O(N^2)$) with context length. Doubling your num_ctx will significantly increase the VRAM required to store the model's active state during generation. Ensure your hardware can handle the increased allocation.

KV Cache Quantization (OLLAMA_KV_CACHE_TYPE)

To track relationships between tokens during a long conversation, the model maintains an active key-value (KV) cache in VRAM. At large context lengths (like 32k or 128k), the size of the KV cache could exceed the size of the model weights themselves, causing crashes due to lack of memory.

To combat this, Ollama supports KV cache quantization. Just as model weights can be compressed from 16-bit floats to 4-bit integers, the KV cache can be quantized to lower precisions with minimal degradation in text quality.