---
title: 'VibeThinker-3B: Compact Verifiable Reasoning'
url: https://www.emergentmind.com/topics/vibethinker-3b
type: topic
---

# VibeThinker-3B: Compact Verifiable Reasoning

VibeThinker-3B is a compact, 3 billion-parameter dense Transformer-based language model developed to push the limits of verifiable reasoning within the constraints of small-model architectures. Uniquely, its approach centers on the Spectrum-to-Signal post-training paradigm, systematically combining supervised curriculum fine-tuning, multi-domain reinforcement learning, offline self-distillation, and instruct-following optimization. Empirical evidence shows that VibeThinker-3B attains frontier-level accuracy on rigorous evaluative tasks, matching or exceeding models of much larger scale while maintaining tight instruction controllability and strong out-of-distribution generalization. The design and outcomes of VibeThinker-3B motivate the Parametric Compression–Coverage Hypothesis, distinguishing the minimal reasoning capacity needed for high-verifiability domains from the broader parametric scope required for open-domain knowledge and long-tail generalist competence [2606.16140].

## 1. Architecture and Model Specification

VibeThinker-3B is initialized from Qwen2.5-Coder-3B, a standard 3B dense Transformer model featuring 24 Transformer layers, each employing 16-headed self-attention (with $d_\mathrm{head}=64$) and feed-forward MLPs of inner dimension approximately 4096. Rotary positional embeddings are employed and extended to a context window of 64,000 tokens, enabling the model to capture entire trajectories for long-context reinforcement learning. The architecture includes no custom attention or gating modules beyond those in its Qwen base; all major innovations arise in the post-training regime, not at the level of core layer design.

## 2. Spectrum-to-Signal Post-Training Protocol

The core of VibeThinker-3B's training is the Spectrum-to-Signal Principle (SSP), a bifurcated framework comprising:

- **Spectrum Phase (Supervised Fine-Tuning, SFT):** Aimed at expanding the diversity of valid reasoning paths by maximizing the log-likelihood over a curated, multi-solution dataset. The loss is:
  $$
  \mathcal{L}_{\mathrm{SFT}}(\theta)
  = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}}
  \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.
  $$

- **Signal Phase (Reinforcement Learning, RL):** Drives the model to select and amplify accurate reasoning traces using MaxEnt-Guided Policy Optimization (MGPO), a policy-gradient method that clips policy ratios, dynamically weights prompts based on empirical accuracy proximity to $p(q)=0.5$, and employs domain-specific verifiers. The PPO-style objective is:
  $$
  \mathcal{J}_\mathrm{MGPO}(\theta)
  = \mathbb{E}\Bigl[\frac{1}{G}\sum_{i,t}
  \min (\rho_{i,t} w(q)A_i,\;\mathrm{clip}(\rho_{i,t},1\!-\!\varepsilon,1\!+\!\varepsilon)w(q)A_i)\Bigr]
  $$

### 2.1 Curriculum-Based Supervised Fine-Tuning

The SFT stage is organized as a two-phase curriculum:

- **Stage I (Coverage):** Full mixed-domain fine-tuning on math, code, STEM, dialogue, and instruction datasets (batch size 128, initial LR $5 \times 10^{-5}$ cosine decayed to $8 \times 10^{-8}$, for 5 epochs with 5% warmup), utilizing sequence packing for chains-of-thought up to ~10,000 tokens.

- **Stage II (Hard Reasoning):** Continuation from Stage I using only data with chain length $\geq$5,000 tokens and high VibeThinker-1.5B error rate (≥ 0.75 over 8 rollouts), further training for 2 epochs. This focuses parameter updates on difficult, long-horizon logical derivations.

Intermediate checkpoints are assessed for Pass@K diversity across domains, and specialist parameters are averaged to yield a unified SFT model retaining the full solution spectrum.

### 2.2 Multi-Domain Reinforcement Learning

MGPO samples $G$ trajectories per prompt, scoring sampled outputs via domain-verifiers and weighting training by information value:
$$
p(q) = \frac{1}{G}\sum_{i=1}^G \mathbb{I}(\text{correct}(y_i)),
$$
$$
w(q) = \exp(-\gamma D_\mathrm{KL}(p(q)\| 0.5)).
$$
Single-stage RL uses the full 64k input window. A subsequent Long2Short Math RL pass redistributes reward among correct trajectories based on brevity, enforcing a zero-sum reward shift to favor shorter valid solutions without affecting group mean reward.

### 2.3 Offline Self-Distillation

Correct trajectories from RL are filtered using a learning-potential score,
$$
S_\mathrm{LP}(q,y) = - \frac{1}{|y|} \sum_{t=1}^{|y|} \log \pi_{\theta_\text{stu}}(y_t|q,y_{<t}),
$$
then bucketed by length and mid-to-high potential traces are distilled into a unified student via final SFT.

### 2.4 Instruct RL

Final on-policy RL over mixed instruction data uses a combination of rule-based and learned rubrics, targeting adherence to format and instruction-following without loss of reasoning capabilities.

## 3. Empirical Results and Evaluation

VibeThinker-3B demonstrates high performance on verifiable reasoning and code benchmarks after stringent post-training:

- **AIME26 (Math):** 94.3% Pass@1, rising to 97.1% with Claim-Level Reliability (CLR) scaling.
- **LiveCodeBench v6:** 80.2% Pass@1.
- **Recent LeetCode contests (Apr 25–May 31, 2026):** 123/128 first-attempt Python submissions passed (96.1%).
- **Instruction Following (IFEval):** 93.4%.
- **GPQA-Diamond (open knowledge):** 70.2%.

Claim-Level Reliability (CLR) applies self-verification at test time: for each answer trajectory, $K=32$ rollouts are decomposed into $M=5$ logical claims, and a claim-verifier scoring is aggregated to select answers with maximized composite reliability. This enhances single-shot math reasoning accuracy by ≈3 points.

VibeThinker-3B consistently matches or slightly exceeds DeepSeek V3.2 (671B), GLM-5 (744B), and Gemini 3 Pro (≥100B) on highly structured reasoning, outperforming smaller and some equivalently sized models under identical protocols.

## 4. The Parametric Compression–Coverage Hypothesis

The model's empirical profile motivates a formal two-component capacity model:
$$
P_{total} = P_{reasoning} + P_{knowledge}
$$
where:

- $P_{reasoning}$: Parameter count sufficient for reasoning heuristics, constraint solving, and compositional logic. Empirically, $P_{reasoning} \ll 3$B.
- $P_{knowledge}$: Parameter count scaling with factual, conceptual, and long-tail knowledge coverage.

For highly structured, verifiable tasks, a "parameter-dense core" suffices. For open-domain QA and generalist competence, $C_K(S)\gg C_R$ is required for most knowledge domains. VibeThinker-3B is situated above the minimal reasoning threshold ($C_R$), but beneath full generalist coverage ($C_K$) for broad knowledge.

A plausible implication is that compact models may function as verifiable reasoning specialists, while larger models serve as generalist memory stores.

## 5. Comparative Performance and Deployment Efficiency

| Model             | Parameters | AIME26 | LCB v6 | GPQA-D | AIME26 (+CLR) |
|-------------------|-----------|--------|--------|--------|---------------|
| VibeThinker-3B    | 3B        | 94.3   | 80.2   | 70.2   | 97.1          |
| DeepSeek V3.2     | 671B      | 94.2   | 80.8   | 82.4   | –             |
| GLM-5             | 744B      | 95.8   | 85.5   | 86.0   | –             |
| Gemini 3 Pro      | ≥100B     | 91.7   | 87.4   | 91.9   | –             |

Despite using only 0.4%–0.5% of the parameters of flagship models, VibeThinker-3B matches DeepSeek V3.2 on AIME26 and trails by only several points on coding and STEM tasks. Inference memory and throughput efficiency are improved by factors of 10–20× in deployment settings due to the reduced parameter count. This situates VibeThinker-3B as a resource-efficient solution for tasks requiring intensive logical and programmatic reasoning under fixed compute budgets.

## 6. Significance and Research Trajectory

VibeThinker-3B substantiates that carefully engineered post-training pipelines—curriculum SFT, multi-domain RL, brevity-aware optimization, self-distillation, and precision instruction RL—enable a 3B-parameter network to condense a "reasoning core" capable of first-tier performance. Simultaneously, residual deficits on open-domain generalist tasks accentuate the differential role of parameter coverage for broad knowledge acquisition.

The evidence supports the dual-path notion proposed by the Parametric Compression–Coverage Hypothesis: compact, small-form models as reasoning specialists, and large models as general-purpose knowledge engines. This framework suggests a division of labor in future AI systems, where model size is optimized not merely for deployment, but for capability specialization [2606.16140].

Source: https://www.emergentmind.com/topics/vibethinker-3b