GRPO: Training Small Models with Verifiable Scores

Comparing several attempts of the same model rather than training a dedicated critic: this is the GRPO approach, born in DeepSeekMath. It relies on verifiable rewards and memory-efficient techniques to enable local training. Under one central condition: that the scoring rule correctly defines success.

Defining the reward shapes learning, both good and bad

Rewarding the mere presence of a string, like 42, can validate incorrect answers that mention it without providing the correct result. The illustration thus imposes a single block of final response, with no text afterward, and extracts an integer via a regular expression limited to 12 digits with an optional sign. The parsing function returns None when the format is invalid, distinguishing a valid zero from a parsing failure, then a scoring function returns 1.0 if the integer matches the expected value, otherwise 0.0. The parser accepts spaces, signs, and leading zeros but rejects duplicate blocks, trailing text, decimals, and floating-point numbers, a choice suitable for a constructed integer task but too narrow for general mathematical verification. These decisions explain why binary scoring can exclude 42.0 even though it is equivalent, and why certain tasks require units or fractional equivalences. In practice, these choices determine the favored outputs: a model may primarily learn the required tags rather than arithmetic, hence the necessity to separately track malformed responses. It is preferable to finalize these criteria before configuring the training; otherwise, an increase in score may only reflect better adaptation to the verifier rather than enhanced problem-solving.

What the outcome verifies does not indicate the intermediate reasoning

A model may claim it verifies or reconsiders a calculation and yet be wrong: for accuracy assessment, the final number counts. Verification by execution, illustrated by the Python assertion confirming that 6×8−6 equals 42, allows the use of question-answer pairs without detailed solutions. In this scheme, the question is provided to the model while the expected value remains with the verifier; the score assigned to the output then guides the adjustment. This is the advantage of verifiable rewards: an objective feedback derived from the result. However, verifying only the final integer does not guarantee the consistency of the reasoning followed to achieve it.

GRPO replaces the PPO critic with a group of outputs

Introduced in DeepSeekMath, GRPO is an alternative to the PPO actor-critic, which typically trains a value estimator as a basis for judgment. Here, the basis comes from a group of responses produced for the same prompt: we compare their actual success. In an example with four attempts scored 1, 0, 1, 0, the average is 0.5, and the advantage is obtained by normalizing by the standard deviation, which can yield values such as [1.0, −1.0, 1.0, −1.0], with the exact implementation varying. The optimizer increases the probability of high-reward outputs compared to others; the signal remains global to the response and does not localize the decisive step. Mechanisms like clipping and, depending on configurations, a drift penalty relative to a reference policy, limit the updates. When all responses in a group have the same reward, the advantage becomes null, and the task then provides no useful differential. Conversely, if successes are too rare, the algorithm lacks guidance; if they are almost systematic, the margin for improvement decreases. Learning assumes frequent informative differences within groups.

Less memory thanks to GRPO, LoRA, and QLoRA, but not without cost

Removing the learned critic reduces memory footprint, and local training becomes more affordable when limiting the parameters updated and storing weights more efficiently. LoRA encodes updates as low-rank matrices, which reduces the number of parameters for which gradients need to be calculated and the optimizer's state stored. QLoRA combines adapters with a quantized base in four bits and employs other methods to save memory. However, these approaches do not eliminate the cost of generating and processing responses during training. Tools like Unsloth aggregate these ideas into practical workflows that include reinforcement learning based on GRPO, with documentation covering integration. All these levers contribute to making small-scale reasoning experiences more accessible.

What DeepSeek models say about unsupervised training

DeepSeek's R1-Zero highlighted reinforcement training without supervised pre-training, with reported behaviors of reevaluating approaches and increased token usage on difficult problems. DeepSeek-R1, on the other hand, utilized a broader pipeline, including cold-start data, and the two processes are not interchangeable. In this context, GRPO stands out for the modesty of the feedback required: the model produces multiple responses to the same question, each is scored, and the differences guide the adjustment. Verification can be limited to the outcome, without a reference solution to imitate, which refocuses the difficulty on defining scores and their role in updating the model.