DeepSeek Revolutionizes AI with Unprecedented Autonomous Learning

The "Aha Moment": A Revelation in AI Learning

Imagine yourself facing a complex puzzle. You move in one direction, realize a contradiction arises, and backtrack to explore a new path until you find the solution. This moment of sudden understanding, often referred to as the "Aha Moment," was observed by researchers at DeepSeek during the training of their experimental model, DeepSeek-R1-Zero. This model, having never been exposed to examples written by humans, demonstrated an astonishing ability to solve complex logical puzzles.

During this pure reinforcement learning phase, the model was confronted with tasks such as: "Solve a complex logical puzzle." The model's thought process was fascinating: it began by assuming that "X = 5" according to one rule, then detected a contradiction with another rule, prompting it to reevaluate its solution. Ultimately, it concluded that "X must be 2." What is remarkable is that the model learned to stop, identify its mistakes, and adjust its approach, thereby developing genuine problem-solving strategies without being explicitly programmed to do so.

A New Teaching Approach: Grading on a Curve

To understand how DeepSeek achieved this feat, it is helpful to compare two distinct teaching methods. The traditional method, known as PPO, involves the use of a critical model that monitors each step of the learner and predicts an absolute score. This approach requires considerable computational power as it runs two large AI models simultaneously.

In contrast, DeepSeek's innovative method, called GRPO, takes a different approach. Instead of relying on a critical model, it submits the same problem to a group of eight students. Once the answers are provided, they are compared to determine which ones are correct. For example, if student C outperforms the average, their reasoning method is rewarded. This intra-group comparison allows the AI to learn what works best without requiring additional resources for an assistant model.

Technical Exploration: Under the Hood of DeepSeek

For developers and machine learning enthusiasts, it is crucial to understand how DeepSeek's architecture and training pipeline function. The DeepSeek-R1-Zero model demonstrated that reasoning capabilities could emerge through trial and error, although its initial results were chaotic.

• DeepSeek-R1-Zero: This model relies on pure reinforcement learning, without human examples. While it has a strong reasoning capacity, its outputs were often messy, frequently mixing languages and producing difficult-to-read formatting.

• DeepSeek-R1: To refine these capabilities, DeepSeek adopted a four-phase hybrid approach, combining human examples with reinforcement learning. The result is a model whose reasoning is not only powerful but also structured, with clear and coherent steps.

The Four-Phase Training Pipeline

To improve formatting while preserving the raw intelligence of reasoning, DeepSeek established a four-step training pipeline:

• Cold Start: The base model is refined with a small set of very clear reasoning examples to teach it to produce readable outputs.

• RL Reasoning: The GRPO algorithm is applied to tasks in logic, mathematics, and programming, where automatic verification of answers is possible.

• Rejection Sampling: The model solves thousands of problems, and only the best answers are retained for further training.

• Diverse RL: The model is adjusted to align with human preferences regarding safety, tone, and general conversation.

GRPO: Calculating Success Through Group Statistics

In traditional AI training, a critical model evaluates the quality of responses. GRPO eliminates this additional model by using a simpler approach: Group Statistics. Instead of predicting absolute scores, GRPO assesses the performance of a response relative to others in the group.

1. Calculating Group Advantage: GRPO measures how much a response stands out compared to others.

2. Training Loss Rule: To update the AI without compromising its capabilities, GRPO balances three essential aspects...