Local Small Models: Three Effective Uses

Small language models are not oracles, but they excel when knowledge comes from external sources. A simple rule helps in choosing: if the task requires facts derived from the model's weights, aim for a large distant model; if it provides its own material, a local model may suffice. Three use cases then emerge — latency, volume, confined data — provided that extended reasoning, recall, and effective context are circumvented.

Choosing Local or Distant: The Decisive Rule

Compliance, cost, and latency needs describe three distinct families of use cases. Their commonality is that knowledge remains outside the model: a schema provides structure, a set of labels bounds the answers, and the screen supplies the reference text. Each of these elements is verified against the output, and the model is only called upon to execute the task with what is provided to it. This selection rule guides the choice of tool. If a task demands facts from the model's own weights, a small local model is not the right candidate, and a large distant model is more suitable, even if it remains uncertain. Conversely, when the task brings its own material, a small model running on the company's hardware can do the job.

Latency: Micro-tasks Read on Screen, Continuously

Latency is a significant issue in its own right. A model that responds in a few hundred milliseconds from local memory is not merely an accelerated version of a service that takes two seconds over a network: it is a different type of tool, one that is accepted to be invoked very frequently. The suitable tasks are short, repeated dozens of times per hour, and so elementary that an additional capacity makes no perceptible difference. The input is brief and already visible, the user reads the output in real-time, and errors appear immediately, without reliance on recall. This is precisely the area where a small model performs best, and where response time impacts perceived cost. Pitfalls arise when feeding the model an entire file rather than a selection — a field, a line, a paragraph — which reactivates context limits. A summary that completely ignores the context of a document may go unnoticed. Conversely, tasks where the output is displayed prominently lend themselves well to this local format.

Volume: A Two-Tier Sorting and Targeted Escalations

When arithmetic dominates, the cost per item of a cutting-edge model may be negligible for forty documents but become prohibitive at four hundred thousand. Bulk classification and labeling tasks — sorting support tickets by subject, organizing archives by type, routing incoming mail — consist of simple decision-making atoms: choosing a label from a short list. If they lag, it is rarely due to intrinsic difficulty, but rather to an inability to justify the cost at scale. Local inference reshuffles this calculation, with a clear limit: a small model sorts easy cases very well and judges difficult cases poorly. The effective configuration is then two-tiered: the local model handles the vast majority, and uncertain cases escalate to a larger model when permitted, or to a person. This escalation path is part of the design: a system that processes the bulk of the volume at lower cost and only escalates as needed fulfills its mission. The mistake would be to entrust the entire queue to the small model with unconditional confidence.

Confined Data: Structuring Under Schema Constraints

Once information cannot leave the organization — operational secrets, critical data, clinical notes, HR files under investigation, client documents under privilege or regulation — capacity comparison no longer applies: the best model is not an option anyway, and the real alternative is a small model or nothing. The most robust task within this perimeter is document structuring: converting unstructured text into queryable records (table, spreadsheet, baseline). It involves reading a report and emitting fields, with output reduced to a line rather than an essay. The key is to shift the difficulty away from the model with schema-constrained decoding: at each step, tokens that would violate the schema are masked, forcing outputs that can be decoded; a malformed output becomes mechanically impossible. The model then focuses on reading and assigning segments to fields, a skill that a small model masters well. Schema complexity remains the enemy: a flat schema of twelve string fields is easy to adhere to, but compliance degrades with the nesting of objects and keys, a pitfall that disproportionately affects small models. In practice, it is better to flatten and make several passes: one to extract, a second to explore nesting, then adjust the number of passes to the desired depth. When the computation is local, these additional passes incur little penalty.

What SLMs Do Wrong: Reasoning, Recall, Context

Three limits structure the perimeter of a small model. First, extended reasoning: the gap with cutting-edge models is clear in complex mathematics and code generation, and it is unlikely to close on the small model side. A system that falters at step seven of a twelve-step plan will not be corrected by a more skillful prompt. Next, it is important to remember that parametric knowledge remains frozen at the cutoff date, does not benefit from selective updates, and its quality varies by domain. A small model, having a more limited stock of starting knowledge, tends to produce more hallucinations when training data is scarce, particularly for specialized APIs, region-specific regulations, or any information that has evolved over the past year. Finally, effective context: it is generally well below the displayed window, with potential reliability losses starting at forty percent, sometimes abruptly. Information in the middle of a long input is the most disadvantaged, and accuracy drops between about thirty and seventy percent of the text, while the beginning and end hold up better. This counterintuitive dynamic easily misleads the user.

Perceived Capacity, Hardware Constraints, and "Eight Billion"

Three circumstances motivate local execution: data that cannot exit, volume already funded, or latency that makes the product. On non-specialized hardware that one already owns, the practical ceiling is around eight billion parameters or less; within this framework, ready-to-use and competent models handle the described tasks correctly without fine-tuning. Discussions about the "amount of knowledge" held by a small model distract from the main point: the widespread notion that it "lacks knowledge" is misleading. No model available on the market can be considered a reliable factual source, and the general capacity measured does not allow for anticipating this level of reliability, as these two aspects are not related. It is true that a small model has a lower volume of knowledge than a large one, but this factor proves to be of little relevance in guiding a precise use case choice.