GRPO on 350M: 29.7% Valid Outputs After 100 Steps

A GRPO adjustment of LFM2.5-350M conducted with TRL and around 500 samples advances compliance with structured outputs on IFStruct from 22.6% to 29.7%. The setup runs on a free-tier GPU and relies on a local evaluation stack via llama.cpp. Detailed JSON and YAML scores, as well as training configuration and rewards, are published. The pipeline claims to demonstrate feasibility rather than reproduce the official IFStruct score.

The adjusted 350M reaches 29.7% on IFStruct, JSON at 31.9%

After adjustment, the model achieves 29.7% success on 2000 samples, with 319/1000 in JSON (31.9%) and 275/1000 in YAML (27.5%). The average latency measured during this run is 1518 ms. In the base configuration served locally, the overall score was 22.6% (452/2000), with 18.0% in JSON (180/1000) and 27.2% in YAML (272/1000), for an average latency of 1453 ms. The overall gap replicated locally thus goes from 22.6% to 29.7%. The IFStruct blog reports 21.1% for LFM2.5-350M; the local benchmark measure at 22.6% via llama.cpp in BF16 is considered close and serves as a comparison baseline on the same stack.

Targeted rewards and 100 training steps

Three reward functions, all bounded between [0,1], evaluate the structure of outputs: requested format and analyzability, first-level field count, and schema validation. They are aggregated with weights [1.0, 0.5, 2.0]. The GRPO training runs for 100 steps, with 8 generations per prompt group, and is designed for a 16 GB GPU. The configuration includes a learning rate of 5e-5, 10 warmup steps, a batch size per device of 4, gradient accumulation over 8, 2 steps per generation, a maximum completion length of 1024, disabling masking of truncated completions, a temperature of 1.1, and logging at each step. During execution, the three reward components progress, the KL divergence from the reference model increases after warming up, and the share of truncated outputs remains close to zero.

Nemotron data and two prompt augmentations

The dataset used for training is nvidia/Nemotron-RL-instruction_following-structured_outputs, which links each prompt to a reference JSON schema and specifies the expected number of fields, totaling around 500 examples. To align this distribution with that of the IFStruct evaluation, two modifications are made: in 40% of cases, an instruction requests presenting the output in a closed code block to reinforce format compliance, and in 20% of another subset, prompts are reformulated into higher-level table exercises, which requires generating raw lists with a defined number of items.

Targeted LoRA adaptation then merging into a standalone checkpoint

The LFM2.5-350M model is adjusted with a LoRA adapter, targeting specific modules in its hybrid attention/convolution architecture: q_proj, k_proj, v_proj, out_proj, in_proj, w1, w2, and w3. This adaptation adds approximately 6 million parameters, or about 1.66% of the total. At the end of training, the adapter is merged into the base weights to create a standalone checkpoint, and then the model and tokenizer are saved in a directory suffixed with -merged, ready for conversion to GGUF.

Local evaluation stack: llama.cpp, GGUF, and 2000 samples

The evaluation relies on uv for Python tools and on llama.cpp to serve the model, installable via Homebrew, with the command llama-server --version allowing for verification. The service exposes an OpenAI-compatible API, used locally here, notably on a MacBook Pro equipped with an Apple M5 Max and 36 GB of unified memory. The base model is served in BF16 GGUF, with an alias, host 127.0.0.1, and runtime parameters, then evaluated on 2000 samples via http://localhost:8080/v1 with a max-tokens of 2048. After adjustment, the merged checkpoint is converted to BF16 GGUF using the llama.cpp converter (repository cloned at depth 1, gguf-py installed, conversion script executed), served with a new alias, and evaluated via http://localhost:8081/v1, with results recorded in a separate file.

Declared scope: demonstrator, not reproduction of the IFStruct pipeline

The authors clarify that the presented pipeline is not that of the RL model mentioned by IFStruct and that the goal is not to reproduce the official score. It is a feasibility demonstrator showing that task-specific adjustment can enhance a small model's output compliance, potentially rivaling larger models. The code is available on GitHub. Training is conducted on a free-tier GPU, and evaluation can be executed locally via llama.cpp, both components forming a complete and lightweight pathway.