Alibaba: An AI Algorithm Redefining Reasoning

A Revolutionary New Algorithm from Alibaba's Qwen Team

Alibaba's Qwen team has recently unveiled an innovative training algorithm that could transform the way artificial intelligence models perform complex reasoning. This new algorithm assigns different weights to each token based on the influence of each step on the subsequent reasoning chain, rather than treating all tokens equally. This method has enabled the creation of significantly longer reasoning chains, where the model learns to independently verify its intermediate results and explore alternative solutions. This behavior emerges naturally through a weighted reward signal.

So far, the algorithm has only been tested on mathematical tasks, and its application to other domains remains to be explored. The team plans to make this training system open source, which could pave the way for new applications.

Overcoming the Limitations of Current Reasoning Models

Reinforcement learning has limitations with reasoning models, as each token receives the same reward. The new algorithm from the Qwen team addresses this issue by weighting each step according to its impact on the subsequent reasoning, effectively doubling the length of thought processes. When a large language model learns to reason through reinforcement learning, it typically receives a simple judgment of success or failure at the end of each generated response. This reward is then distributed evenly across each token in the sequence, whether it is a logical turning point or just a comma.

The Qwen team claims that this blunt credit assignment is one of the major reasons why reasoning models hit a ceiling with common training methods like GRPO (Group Relative Policy Optimization). Reasoning chains reach a certain length and then stagnate. With Future-KL Influenced Policy Optimization (FIPO), the team aims to break this bottleneck. Instead of evaluating each token individually, the algorithm anticipates: how does the model's behavior change downstream after generating this particular token?

FIPO calculates the cumulative probability change across all subsequent tokens and uses this signal to assign rewards more accurately. Tokens that trigger a productive reasoning chain receive a larger share. Those that lead the model into a dead end receive less.

FIPO Outperforms PPO-Based Methods Without an Auxiliary Model

Previous attempts to solve the flat reward problem primarily relied on PPO (Proximal Policy Optimization) methods that use a separate value model to estimate a benefit score for each token. This auxiliary model typically requires pre-training on long thought chain data, meaning that external knowledge can seep in. Researchers claim this makes it difficult to determine whether performance gains come from the algorithm itself or are simply inherited from the pre-trained assistant. FIPO completely avoids the auxiliary model while providing comparable results.

FIPO outperforms the benchmark as well as Deepseek-R1-Zero and o1-mini on the AIME-2024 benchmark during training.

Doubled Thought Processes with Increased Accuracy

The team tested FIPO on Qwen2.5-32B-Base, a model with no prior exposure to synthetic long thought chain data. They trained it exclusively on the public dataset of DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization), a popular open-source GRPO training variant, to ensure a fair comparison.

The results are clear. While the average length of thought chains from DAPO stagnates around 4,000 tokens, FIPO exceeds 10,000 tokens. On the AIME 2024 mathematical benchmark, accuracy increases from 50 to 56 percent, peaking at 58 percent. This places FIPO ahead of Deepseek-R1-Zero-Math-32B at around 47 percent and OpenAI's o1-mini at about 56 percent. On the more challenging AIME 2025, scores rise from 38 to 43 percent.

Researchers note that this is not just a few outliers stretching longer. The entire distribution of response lengths shifts upward, from the shortest to the longest responses. This suggests a fundamental change in how the model approaches problems.

The Model Begins to Verify Its Own Results

The article describes four phases that the model goes through during training. Initially, it produces superficial planning patterns—essentially sketches without real mathematics that end with a hallucinatory answer. In the second phase, where DAPO-trained models remain for the rest of the training, the model executes a clean linear reasoning chain and stops at the first answer it finds.

In the third phase, the model begins to spontaneously verify its own intermediate results. It reaches an answer but then pivots to another approach, for example, shifting from algebraic manipulation to geometric interpretation for verification. In phase four, the model performs systematic multi-pass verification, recalculating large square numbers step by step and working through the full derivation multiple times.

The article notes that this behavior closely resembles inference time scaling strategies in OpenAI's o series and Deepseek-R1, but FIPO achieves this solely through reinforcement learning, without synthetic long thought chain data.

More Testing to Be Done

FIPO has been evaluated solely on mathematical problems, trained on a single dataset, and tested only on base models without pre-training on long thought chains. Longer sequences also increase computational costs. Therefore, there is still much testing to be done, according to the team.

Additionally, it remains uncertain whether these gains translate to other domains such as code or symbolic logic. There is also a performance gap compared to distillation from larger teacher models. Pure reinforcement learning teaches a model less than direct instruction from a stronger model.

The team plans to make the training system open source along with all configurations.