The High Cost of Thinking
Modern Large Language Models are powerful reasoners, but their detailed "chains of thought" lead to enormous token usage. This increases computational costs, resource requirements, and response times. The common assumption has been that longer, more complex reasoning is necessary for accuracy. This report challenges that idea.
A Counter-Intuitive Finding: Brevity is Better
The paper reveals a strong, overlooked correlation: for reasoning models, concise responses are more likely to be correct. Explore the data from Table 1 of the paper below. On average, incorrect answers use significantly more tokens than correct ones across various models and benchmarks.
The "Why": RL Training Dynamics
The paper's core technical contribution is explaining *why* models become verbose. It's not a reasoning strategy, but a side effect of how Reinforcement Learning (RL) algorithms like PPO work.
PPO: The Push for Length (Fig. 1 Concept)
When a model using PPO gets an answer wrong (negative reward), the loss function is minimized by generating longer responses. This interactive chart simulates that dynamic: as response length increases, the penalty (policy loss) for being wrong decreases.
The Two-Phase Solution
Knowing these dynamics, the paper proposes a novel, two-phase RL strategy to build reasoning ability first, then enforce conciseness.
Phase 1: Build Raw Power
The model is trained on a wide range of challenging problems. During this phase, the model's core problem-solving capacity is enhanced. As it often answers incorrectly at first, the RL algorithm's dynamics naturally encourage longer, more exploratory responses, which is an expected outcome.
Phase 2: Enforce Conciseness
Training continues on a small set of problems the model can occasionally solve correctly. Now receiving positive rewards, the RL algorithm flips its incentive: it starts to favor shorter, more efficient paths to the correct answer. This phase prunes verbosity without sacrificing the accuracy gained in Phase 1.
The Evidence: Interactive Results
This approach works. The following tables show the results of the two-phase training. The first table (based on Table 2) shows the overall impact, while the subsequent tables (from Section 7.3) dive into performance and robustness.
Overall Performance Comparison (Table 2)
| Benchmarks | R1 1.5B | R1 7B | ||||||
|---|---|---|---|---|---|---|---|---|
| Accuracy (%) | Length (tokens) | Accuracy (%) | Length (tokens) | |||||
| Baseline | Ours | Baseline | Ours | Baseline | Ours | Baseline | Ours | |
Increased Performance and Robustness
The training not only reduces length but also improves the model's core performance. The post-trained models are more robust and less sensitive to changes in sampling temperature. They also show substantial accuracy gains on models that haven't previously undergone RL training, even with a tiny training set.
Robustness to Temperature Changes (Table 3)
| Temperature | Base Model | Ours (Post-Trained) | ||
|---|---|---|---|---|
| MATH500 | AIME24 | MATH500 | AIME24 | |
| temp=0, n=1 | 70% | 13.3% | 81% | 23.3% |
| temp=0.6, n=4 | 84.3% | 32.5% | 81% | 30% |
| Relative Degrade | 16.9% | 59% | 0% | 22.3% |
Accuracy Gains on Non-RL-Trained Models (Table 4)
| Model | MATH500 | AIME24 | AMC23 | |||
|---|---|---|---|---|---|---|
| Baseline | Ours | Baseline | Ours | Baseline | Ours | |
| Qwen2.5-Math-7B | 47.45% | 67.05% | 15.83% | 23.33% | 43.75% | 57.50% |
| Qwen2.5-Math-1.5B | 33.45% | 63.05% | 6.67% | 9.17% | 30.00% | 48.12% |
| Qwen2.5-7B | 54.30% | 70.20% | 5.00% | 10.83% | 40.00% | 54.38% |
Conciseness in Action ✨
See the difference for yourself. Enter a reasoning problem below and ask Gemini to solve it using a standard, verbose approach versus a concise one inspired by this paper's findings. (API features disabled for external hosting).
Standard Response
Concise Response
Key Contributions
This paper presents several key findings that challenge common assumptions and offer a new path toward more efficient LLM reasoning.
-
1
Correlation of Conciseness and Accuracy
Shows that for both reasoning and non-reasoning models, concise reasoning strongly correlates with higher accuracy, questioning the "longer is better" assumption.
-
2
Analysis of PPO Loss Dynamics
Provides a mathematical analysis proving that incorrect answers inherently drive longer responses under PPO, while correct answers encourage brevity.
-
3
Analysis of GRPO Loss Dynamics
Demonstrates that while GRPO has some positive properties, it suffers from "collapse modes" that make it unreliable for consistently enforcing conciseness.
-
4
Effectiveness with Limited Data
Proves that the conciseness-enforcing phase of RL training is effective even with a remarkably small dataset, offering a viable path for resource-constrained scenarios.