VibeThinker-3B: Compact Verifiable Reasoning

• VibeThinker-3B is a compact 3 billion-parameter transformer model that leverages a Spectrum-to-Signal post-training paradigm combining supervised curriculum fine-tuning, multi-domain reinforcement learning, and self-distillation.
• The model achieves frontier-level accuracy on rigorous evaluative tasks such as AIME26 and code benchmarks, matching or slightly exceeding performance of models with hundreds of billions of parameters.
• Its design supports the Parametric Compression–Coverage Hypothesis, suggesting that a compact, parameter-dense reasoning core suffices for high verifiability while broader knowledge requires larger models.

VibeThinker-3B is a compact, 3 billion-parameter dense Transformer-based LLM developed to push the limits of verifiable reasoning within the constraints of small-model architectures. Uniquely, its approach centers on the Spectrum-to-Signal post-training paradigm, systematically combining supervised curriculum fine-tuning, multi-domain reinforcement learning, offline self-distillation, and instruct-following optimization. Empirical evidence shows that VibeThinker-3B attains frontier-level accuracy on rigorous evaluative tasks, matching or exceeding models of much larger scale while maintaining tight instruction controllability and strong out-of-distribution generalization. The design and outcomes of VibeThinker-3B motivate the Parametric Compression–Coverage Hypothesis, distinguishing the minimal reasoning capacity needed for high-verifiability domains from the broader parametric scope required for open-domain knowledge and long-tail generalist competence (Xu et al., 15 Jun 2026).

1. Architecture and Model Specification

VibeThinker-3B is initialized from Qwen2.5-Coder-3B, a standard 3B dense Transformer model featuring 24 Transformer layers, each employing 16-headed self-attention (with $d_\mathrm{head}=64$) and feed-forward MLPs of inner dimension approximately 4096. Rotary positional embeddings are employed and extended to a context window of 64,000 tokens, enabling the model to capture entire trajectories for long-context reinforcement learning. The architecture includes no custom attention or gating modules beyond those in its Qwen base; all major innovations arise in the post-training regime, not at the level of core layer design.

2. Spectrum-to-Signal Post-Training Protocol

The core of VibeThinker-3B's training is the Spectrum-to-Signal Principle (SSP), a bifurcated framework comprising:

• Spectrum Phase (Supervised Fine-Tuning, SFT): Aimed at expanding the diversity of valid reasoning paths by maximizing the log-likelihood over a curated, multi-solution dataset. The loss is:

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$

• Signal Phase (Reinforcement Learning, RL): Drives the model to select and amplify accurate reasoning traces using MaxEnt-Guided Policy Optimization (MGPO), a policy-gradient method that clips policy ratios, dynamically weights prompts based on empirical accuracy proximity to $p(q)=0.5$, and employs domain-specific verifiers. The PPO-style objective is:

$$\mathcal{J}_\mathrm{MGPO}(\theta) = \mathbb{E}\Bigl[\frac{1}{G}\sum_{i,t} \min (\rho_{i,t} w(q)A_i,\;\mathrm{clip}(\rho_{i,t},1\!-\!\varepsilon,1\!+\!\varepsilon)w(q)A_i)\Bigr]$$

2.1 Curriculum-Based Supervised Fine-Tuning

The SFT stage is organized as a two-phase curriculum:

• Stage I (Coverage): Full mixed-domain fine-tuning on math, code, STEM, dialogue, and instruction datasets (batch size 128, initial LR $5 \times 10^{-5}$ cosine decayed to $8 \times 10^{-8}$, for 5 epochs with 5% warmup), utilizing sequence packing for chains-of-thought up to ~10,000 tokens.
• Stage II (Hard Reasoning): Continuation from Stage I using only data with chain length $\geq$5,000 tokens and high VibeThinker-1.5B error rate (≥ 0.75 over 8 rollouts), further training for 2 epochs. This focuses parameter updates on difficult, long-horizon logical derivations.

Intermediate checkpoints are assessed for Pass@K diversity across domains, and specialist parameters are averaged to yield a unified SFT model retaining the full solution spectrum.

2.2 Multi-Domain Reinforcement Learning

MGPO samples $G$ trajectories per prompt, scoring sampled outputs via domain-verifiers and weighting training by information value:

$$p(q) = \frac{1}{G}\sum_{i=1}^G \mathbb{I}(\text{correct}(y_i)),$$

$w(q) = \exp(-\gamma D_\mathrm{KL}(p(q)\| 0.5)).$

Single-stage RL uses the full 64k input window. A subsequent Long2Short Math RL pass redistributes reward among correct trajectories based on brevity, enforcing a zero-sum reward shift to favor shorter valid solutions without affecting group mean reward.

2.3 Offline Self-Distillation

Correct trajectories from RL are filtered using a learning-potential score,

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$0

then bucketed by length and mid-to-high potential traces are distilled into a unified student via final SFT.

2.4 Instruct RL

Final on-policy RL over mixed instruction data uses a combination of rule-based and learned rubrics, targeting adherence to format and instruction-following without loss of reasoning capabilities.

3. Empirical Results and Evaluation

VibeThinker-3B demonstrates high performance on verifiable reasoning and code benchmarks after stringent post-training:

• AIME26 (Math): 94.3% Pass@1, rising to 97.1% with Claim-Level Reliability (CLR) scaling.
• LiveCodeBench v6: 80.2% Pass@1.
• Recent LeetCode contests (Apr 25–May 31, 2026): 123/128 first-attempt Python submissions passed (96.1%).
• Instruction Following (IFEval): 93.4%.
• GPQA-Diamond (open knowledge): 70.2%.

Claim-Level Reliability (CLR) applies self-verification at test time: for each answer trajectory, $$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$1 rollouts are decomposed into $$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$2 logical claims, and a claim-verifier scoring is aggregated to select answers with maximized composite reliability. This enhances single-shot math reasoning accuracy by ≈3 points.

VibeThinker-3B consistently matches or slightly exceeds DeepSeek V3.2 (671B), GLM-5 (744B), and Gemini 3 Pro (≥100B) on highly structured reasoning, outperforming smaller and some equivalently sized models under identical protocols.

4. The Parametric Compression–Coverage Hypothesis

The model's empirical profile motivates a formal two-component capacity model:

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$3

where:

• $$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$4: Parameter count sufficient for reasoning heuristics, constraint solving, and compositional logic. Empirically, $$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$5B.
• $$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$6: Parameter count scaling with factual, conceptual, and long-tail knowledge coverage.

For highly structured, verifiable tasks, a "parameter-dense core" suffices. For open-domain QA and generalist competence, $$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$7 is required for most knowledge domains. VibeThinker-3B is situated above the minimal reasoning threshold ($$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$8), but beneath full generalist coverage ($$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{SFT}}} \sum_{t=1}^{|y|}\log \pi_\theta(y_t \mid q, y_{<t})\,.$$9) for broad knowledge.

A plausible implication is that compact models may function as verifiable reasoning specialists, while larger models serve as generalist memory stores.

5. Comparative Performance and Deployment Efficiency

Model	Parameters	AIME26	LCB v6	GPQA-D	AIME26 (+CLR)
VibeThinker-3B	3B	94.3	80.2	70.2	97.1
DeepSeek V3.2	671B	94.2	80.8	82.4	–
GLM-5	744B	95.8	85.5	86.0	–
Gemini 3 Pro	≥100B	91.7	87.4	91.9	–

Despite using only 0.4%–0.5% of the parameters of flagship models, VibeThinker-3B matches DeepSeek V3.2 on AIME26 and trails by only several points on coding and STEM tasks. Inference memory and throughput efficiency are improved by factors of 10–20× in deployment settings due to the reduced parameter count. This situates VibeThinker-3B as a resource-efficient solution for tasks requiring intensive logical and programmatic reasoning under fixed compute budgets.

6. Significance and Research Trajectory

VibeThinker-3B substantiates that carefully engineered post-training pipelines—curriculum SFT, multi-domain RL, brevity-aware optimization, self-distillation, and precision instruction RL—enable a 3B-parameter network to condense a "reasoning core" capable of first-tier performance. Simultaneously, residual deficits on open-domain generalist tasks accentuate the differential role of parameter coverage for broad knowledge acquisition.

The evidence supports the dual-path notion proposed by the Parametric Compression–Coverage Hypothesis: compact, small-form models as reasoning specialists, and large models as general-purpose knowledge engines. This framework suggests a division of labor in future AI systems, where model size is optimized not merely for deployment, but for capability specialization (Xu et al., 15 Jun 2026).