Local SLM: Practical Guide and Settings with Ollama

Running a language model on your own machine is no longer a technical curiosity. With benefits like privacy gains, predictable costs, and reduced latency, local LLMs are becoming essential for practical uses—provided you choose, test, and configure them correctly. Here’s how to validate a model, integrate it into effective architectures, and run it with Ollama while adhering to hardware and configuration constraints.

Validate First: 20 to 50 Examples and Focus on Failures

Before any integration, the priority is to verify that a candidate model meets the requirements of the intended task. Early on, create a small evaluation set consisting of 20 to 50 representative examples with expected outputs to make a decision. It doesn't need to be formal: running each model on this set and conducting a manual review against your specific criteria provides better insights than generic rankings. The focus should first be on failure modes rather than flattering averages. A model that succeeds in 90% of cases but fails severely in 10% may be less suitable than another that is slightly less performant but has more controlled errors. Identifying where a model breaks before building on it avoids dead ends. Tests should also reflect actual context lengths: a model comfortable with short queries may degrade significantly when the input length increases.

Integrate Locally: RAG, Code Assistants, Agents, and Pipelines

Several architectures particularly benefit from a local LLM. Answering questions about personal documents is an effective starting point: by combining a local model with a retrieval system, a RAG-type pipeline synthesizes answers from files, PDFs, notes, or internal documentation without these data leaving the machine. Local coding assistants also benefit from low latency: models fine-tuned for code run quickly on modest hardware, integrate as editor plugins or command-line tools, and can access the file system and the complete project context without calling an API. For more advanced uses, agent-based flows where an agent decides which tools to call and in what order work well with compact, often specialized models that keep everything fast and modular. Locally, these agent chains operate offline and without external dependencies. Finally, structured extraction, classification, and transformation pipelines fall within the scope of a 7 billion parameter model, with the advantage of handling sensitive data without exposing it.

Run Locally with Ollama on macOS, Linux, and Windows

On the tooling side, Ollama stands out as the entry point for running models locally. The tool manages model downloads, GPU acceleration, and exposes a local API, all through a command-line interface. It works on macOS, Linux, and Windows, and once installed, allows you to pull and launch a model with a single command. By default, Ollama serves the model via a local REST API on port 11434, enabling application dialogue with the same schemas as a standard HTTP API. Libraries like LangChain and LlamaIndex have native integrations, making it easier to insert the model into an existing architecture. The provided model library is well-curated, but it is not limiting: it is possible to import GGUF-format models from Hugging Face and run them via Ollama, opening access to community adjustments and specialized variants—useful when a generalist model does not exactly fit the need.

Select the Model: 1–3B, 7B, 13B, and GGUF Quantization

Choosing a model depends as much on the task as on the available hardware. Generally, models with 1 to 3 billion parameters run on almost any modern machine with 8 GB of RAM; 7 billion models form the most popular category on consumer hardware; those with 13 billion approach the limit of what a single high-end GPU or a machine with ample RAM can handle effectively. Specialization matters: for code, models like Code Llama or Qwen-Coder are fine-tuned; others are better suited for following instructions, summarization, or chat. Rather than aiming for the largest executable model, it is better to match the model to the task and then validate it on real cases. Benchmarks remain useful, but the decisive measure is the quality of outputs for your use case. The quantization format impacts size and speed. Most local models are released in GGUF, which allows for 4 or 8-bit quantizations, reducing memory with a modest quality trade-off. For many applications, Q4 or Q5 is a relevant default setting. Among families, Llama 3, Mistral, Gemma 2, Phi-3, and Qwen 2.5 deserve evaluation, each having its own strengths.

Measure Local Gains: Privacy, Costs, Latency, and Limits

Running locally changes the game on three fronts: privacy, costs, and latency. Queries and responses do not leave the environment, a key advantage for sensitive documents, proprietary, or personal data; going through a cloud API introduces a third party into the inference chain. On the cost side, the API is billed per use while local operation relies on a fixed hardware cost without per-token billing, freeing up prototyping and scaling of internal tools. Regarding latency, eliminating network travel improves perceived responsiveness, even on modest hardware. These gains come with constraints: generally smaller context windows, lower raw capacity on complex tasks, and the need for initial setup and suitable hardware. As a rough order of magnitude, a 7 billion parameter model requires about 8 GB of VRAM or RAM for speeds considered reasonable in 4 bits. In this landscape where the logic of "bigger = better" is no longer the rule, clarifying these trade-offs upfront guides the choice; to broaden the perspective, five resources dedicated to LLMs provide a useful overview of the ecosystem.

Optimize: Modelfiles, 4K–32K Context, Temperature, and Prompts

Beyond execution, configuration makes a difference. Ollama relies on Modelfiles—similar in spirit to a Dockerfile—to define behavior: system prompt, context length, temperature, and sampling parameters, stops, or formatting constraints can be specified. Many models offer context windows ranging from 4K to 32K tokens; the larger the context, the more memory it consumes. Reducing the window speeds up exchanges and lightens the footprint when the application handles short queries; for a document assistant or summarization tool, it is necessary to balance context size with available resources. Temperature adjusts the randomness of outputs: low values, around 0.1 to 0.3, favor determinism for code generation or data extraction; higher values, from 0.7 to 1.0, diversify responses for creative tasks. Finally, a large part of the quality comes from the system prompt itself: clearly defining the role, expected format, and constraints brings a generalist model closer to a specialized tool. It is better to refine this prompt before considering a larger model.