Brief IA
›

Ollama and llama.cpp Welcome Decision Models

Code & Dev·Tom Levy·

Ollama and llama.cpp Welcome Decision Models

Ollama and llama.cpp Welcome Decision Models
Key Takeaways
1Ollama and llama.cpp accept decision models via /v1/systemone.
2Decider offers up to 255 choices per question.
3TypeSafe has not released the weights, training method, or complete architecture of Jev.
Why it matters : these tools enable experimentation with AI decision-making locally without relying on a paid API.

Ollama and llama.cpp now support decision-making models via the /v1/systemone endpoint, compatible with TypeSafe. These systems choose from allowed responses and associate probabilities with them, without generating text. Their local execution offers an alternative to paid APIs, but the reliability of the scores must be verified before automating decisions.

Running a Decision-Making Model on Your Computer

Ollama supports Nimble and Tev1, while llama.cpp supports Laya, Kev, Julia-1, and OpenJev. A local installation can reduce API costs while keeping sensitive information on the hardware used. It also allows for experimentation without resorting to a paid API.

With a recent version of llama.cpp, the command llama serve -hf ggml-org/Laya-GGUF launches a server hosting Laya. A curl request can then target http://localhost:8080/v1/systemone, with content declared as application/json. In a customer support example, it submits a complaint for double billing, specifies the type as choice, and asks which team should intervene.

The options billing, shipping, and technical correspond to payments or refunds, delivery issues, and bugs or account difficulties, respectively. The expected result designates a service and associates a probability with each possibility. The focus on billing here remains a hypothetical example, not the result of a test on real requests.

Five Open Projects Against the Proprietary Jev Model

At TypeSafe, Jev remains proprietary and accessible via API. The company has provided insights into its approach without revealing the entirety of its architecture, weights, or training method. Open solutions can be examined, adapted, and hosted locally. Some of these projects claim better performance on specific evaluations, faster execution, and potentially lower operating costs.

For local execution, SemIf supports NVIDIA graphics processors, Apple Silicon chips, and central processors. An interactive demo accessible from a browser accompanies it, along with reproducible measurements of speed and accuracy. Capable of running on modest hardware, Laya is a lightweight model supporting over 100 languages, with fast inference and automatic selection of the appropriate model.

Originating from Qwen, the Kev family includes models ranging from 0.8B to 27B parameters; it supports long documents, custom adaptation, and an API compatible with Jev. Simple Jev allows for classification or scoring with existing Hugging Face models without additional training. Its compatibility with visual models extends this use to images.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Decider aims to make multiple decisions in a single computation pass, with a maximum of 255 possible responses per question. Its experimental demos include Tetris and Pong. Depending on needs, these various solutions allow for prioritizing small models for speed, specialized tuning, or local hosting to limit API expenses.

How Models Produce Their Probabilities

AI decision support includes ranking a request among commercial, technical, or billing services, assessing the urgency of a complaint, and binary identification of spam. The system receives a context, a question, and the accepted responses. Its structured outputs are directly usable by an application, without needing to analyze written paragraphs. This functionality is particularly suitable for processing thousands of similar requests.

A large language model can be adapted for this use by leveraging its internal predictions. It calculates logits, which are raw scores for the next possible tokens. The scores corresponding to the selected responses can be extracted and then converted into probabilities using the softmax mathematical function.

In a hypothetical example of ranking reviews, logits of 2.0 for positive and 1.0 for negative yield approximately 73% and 27% after this transformation. These probabilities depend on the selected options.

Another adaptation involves substituting a small prediction head for the text production layer and then fine-tuning the model on decision-making tasks. Several options can then be evaluated, with the possibility of handling multiple questions in the same computation pass. The absence of successive token generation speeds up repetitive decisions and improves their efficiency.

Testing Reliability Before Automating

Before automating customer support, the accuracy of responses and the associated probabilities should be tested on real messages. A displayed confidence of 95% raises questions about its reliability. Training, probability calibration, and human oversight are identified means to prepare for real-world usage. The model can also be queried about the necessity of human intervention.

The association with language models could make certain applications faster and less costly. In a programming assistance scenario, a small decision-making model would choose a tool, decide to consult the Web, or determine if a task requires a more powerful model. The use of costly reasoning would then be reserved for tasks that require it; this is a contemplated use, not a measured result.

In gaming, experiments are already using these models to select a move in a matter of milliseconds. Extensions are being considered for robotics, fraud detection, or autonomous agents.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.