Smaug from Abacus.AI: Where the Gains Are Clearest

Abacus.AI releases numerical results for three open models optimized for agents. Score increases vary significantly depending on the baseline and the task, with a claimed focus on stability during long executions and inference compatibility. The cited benchmarks show marked progress for Smaug Flash and targeted advantages for Smaug Mini, while Smaug Agentic highlights more consistent behavior rather than spectacular deltas.

Stability in Long Execution Highlighted for Smaug Agentic

Smaug Agentic shows a total score variation of +0.3 compared to its baseline, but executed 113 agentic coding tasks over more than seven hours, with a median of 78 agent steps per task. Abacus.AI reports that no infrastructure errors or delays were observed during these sequences. The emphasis is on behavioral change, notably the elimination of erratic reasoning while maintaining normal deliberation. Consistency during long loops is prioritized over mere score increases.

Where the Baselines Remain Ahead: Two Benchmarks Where the Adjusted Falls Short

On Terminal-Bench 2.1, the baseline of Smaug Agentic surpasses the adjusted version with 88.3 versus 86.5. On MMMU-Pro, the gap is slight but still favors the baseline with 81.6 versus 81.0. These cases are explicitly mentioned as situations where optimization does not yield score gains.

Smaug Flash Bridges Gaps on Targeted Tasks

Smaug Flash is an adjustment of DeepSeek V4 Flash, described as fast and cost-effective but prone to confusion during prolonged use of tools. The fine-tuning targeted this mode of failure. Reported results indicate a gain of 14.3 points in agentic coding on LiveBench, moving from 46.8 to 61.1. On AutomationBench, the announced progress is 13.7 points, from 25.1 to 38.83. On NL2Repo-Bench, the increase reaches 19 points, from 54.2 to 73.3. These gains are presented as corrections on precisely targeted workflows.

Smaug Mini Stands Out on IFBench, JobBench, and NL2Repo-Bench

Smaug Mini, an advanced iteration of Qwen3.8-27B with 27 billion parameters, is particularly aimed at multimodal tasks and tool usage. Abacus.AI indicates that it has surpassed several closed models on certain benchmarks, ranking ahead of Claude Sonnet 5, Claude Opus 4.6, and GPT-5.6 Luna on IFBench, AutomationBench, and JobBench. The provided figures include a gain of 13.5 points on NL2Repo-Bench compared to the baseline, from 42.3 to 55.8, and a gain of 17.1 points on JobBench, from 33.4 to 50.5. The largest increases are in Mathematics (+3.5) and Reasoning (+2.8). Meanwhile, the score for agentic coding slightly decreases compared to the baseline (-0.6), illustrating trade-offs based on the evaluated categories.

Why Abacus.AI Created Smaug: Agent Limitations and Approach Choices

The development of agents faces practical challenges: an agent can produce an output without completing the task and must chain understanding, tool selection, output analysis, and subsequent actions. Over repeated cycles, it can drift from the goal, incurring additional costs and risks. Abacus.AI reports tasks lasting several hours, utilizing a large context and tool calls to documents, APIs, and databases, with risks of burnout or failure. The company asserts that transforming a state-of-the-art model into a good agent is more about training than prompting, and that general LLMs are not always suited for prolonged use, particularly with a less effective prompt cache and an expanding context. Rather than prompt adjustments or local corrections, the choice was made to enhance the models themselves, leading to the Smaug family. Gains appear where the baselines had the most room for improvement, and performance heavily depends on the model and task type.

Common Recipe and Fine-Tuning Variants, with Compatibility Preserved

Three open-weight models make up the Smaug range, each based on a different architecture and all publicly accessible on Hugging Face. A shared approach combines human sequences from real agentic tasks with synthetic data focused on complex situations, then applies filtering to exclude cases that maximize the score without actually completing the task. The stated goal is to prioritize problem-solving over format adherence. For Smaug Flash, only the attention factor matrices are modified via three LoRA adapters, two trained in a supervised manner and one by preference, while the rest remains identical to DeepSeek V4 Flash, maintaining inference compatibility and a context window of 1 million tokens. For Smaug Agentic, supervised training masks reasoning tokens in the loss to preserve the native thought process while improving tool usage, without altering architectural parameters, allowing for a direct replacement of Kimi K3. Smaug Mini applies the same logic to a dense base of 27B, with gains where there is room for improvement and trade-offs elsewhere. Abacus.AI claims that many fine-tunings would sacrifice reasoning for format, and asserts that it dedicates the training budget to data that effectively enhances the model rather than exploiting benchmarks.