Step3-VL-10B: Compact Vision-Language Model

• Step3-VL-10B is an open-source compact vision-language model featuring unified pre-training on 1.2T tokens and extensive reinforcement learning for robust multimodal reasoning.
• It integrates a perception encoder, a projector bridge, and a Qwen3-8B decoder to ensure efficient cross-modal fusion and scalable inference modes.
• Empirical benchmarks demonstrate that its PaCoRe inference strategy significantly boosts performance on complex tasks like MathVision and MMMU.

Step3-VL-10B is an open-source, compact vision-language foundation model designed to realize efficient, high-performance multimodal intelligence at the 10 billion parameter scale. By adopting unified pre-training across a large, diverse corpus and scaling post-training with extensive reinforcement learning and multi-stream reasoning during inference, Step3-VL-10B delivers competitive results on both perception and complex reasoning benchmarks, sometimes surpassing proprietary models 10–20 times larger in parameter count (Huang et al., 14 Jan 2026).

1. Model Architecture and Components

Step3-VL-10B integrates three primary subsystems forming an intrinsically vision-language-aware framework:

• Perception Encoder (PE-lang, 1.8B parameters): A vision backbone pre-aligned to language features. It employs two stacked stride-2 convolutional layers for 16× downsampling and handles inputs as both a 728×728 global view and multiple 504×504 local crops, processed jointly in a batch-parallel regime. The spatial structure is preserved via “newline” token insertions at patch row ends and 1D rotary positional embeddings (RoPE).
• Projector Bridge (≈0.2B parameters): Two stride-2 layers compress feature maps for compatibility with the decoder’s tokenized input.
• Qwen3-8B Decoder (8B parameters): A 32-layer transformer (hidden size ≈4096, 32 heads), capable of autoregressive text generation. Adaptation for multimodal tasks occurs through cross-modal connections originating from the projector bridge, with both encoder and decoder fully unfrozen and updated during pre-training.

The end-to-end model totals approximately 10B parameters, with all modules co-trained for vision-language alignment (Huang et al., 14 Jan 2026).

2. Unified Pre-training Protocol

Step3-VL-10B employs a unified pre-training loss: $$\mathcal{L}_{\mathrm{LM}} = -\sum_{t=1}^{T} \log p(w_t \mid w_{<t}, V)$$ where $w_t$ are output tokens and $V$ are visual tokens from the perception encoder. Standard captioning losses are subsumed by this objective when image embeddings precede text tokens.

The model is trained on 1.2T multimodal tokens using a two-phase LR schedule:

• Phase I (900B tokens): LR annealed from $5\times10^{-5}$ to $1\times10^{-5}$
• Phase II (300B tokens): Training focused on higher-quality data with LR lowered to $6\times10^{-6}$

The corpus is diverse, comprising web image-text (Common Crawl, StepCrawl), major open datasets (LAION, COYO), educational material (≈15M samples), OCR (40M samples), document-to-code, grounding/counting (400M samples), VQA (≈30M), and GUI-grounded multimodal trajectories (23M). Training is performed at batch size 8192, sequence length 4096, utilizing AdamW with ($\beta_1=0.9$, $\beta_2=0.95$), and weight decay 0.01.

3. Reinforcement Learning and Post-training Pipeline

The post-training pipeline comprises several stages:

• Supervised Finetuning (SFT): Two stages—first on text-dominant data (9:1 text:multimodal, seq-len 128k, batch 32), then with balanced data. A cosine LR scheduler with 200-step warm-up is adopted ($1\times10^{-5}$ to $1\times10^{-4}$).
• Reinforcement Learning (PPO + GAE):
  • Policy $\pi_\theta(a_t|s_t)$ predicts output tokens given state $s_t=(I,q,w_{<t})$.
  • Rewards for verifiable tasks rely on intersection-over-union (IoU), Euclidean distance decays, or model-based metrics (semantic equivalence using GPT-OSS-120B). Non-verifiable tasks use generative reward models, pairwise preference training, and language calibration.
  • Schedules: RL with Verifiable Rewards (600 iterations, 512×16 rollouts, max length 24k), RL from Human Feedback (300 iterations), and PaCoRe-RL (500 on-policy iters, 64×16 rollouts, max length 64k).
• Generalized Advantage Estimation (GAE) and PPO surrogate objectives are used for stable advantage estimation and policy optimization.

4. Parallel Coordinated Reasoning (PaCoRe) at Inference

Step3-VL-10B implements Parallel Coordinated Reasoning ("PaCoRe", Editor's term) to scale perceptual reasoning at test time:

• Launches $M=16$ independent inferences (rollouts), each consuming a proportional fraction ($C/M$) of the FLOP budget.
• Contexts for each rollout are concatenated, and a template-based synthesis procedure produces the final output.
• The approach externalizes the generation and verification of diverse visual hypotheses, increasing robustness and completeness.

Empirical gains from PaCoRe are substantial:

• MMMU benchmark: SeRe (single sequence) 78.11% → PaCoRe 80.11% (+2.0)
• MathVision: 70.81% → 75.95% (+5.14)
• CountQA and All-Angles-Bench show similar or larger boosts (Huang et al., 14 Jan 2026).

5. Empirical Performance and Benchmarking

Step3-VL-10B demonstrates best-in-class performance within the 10B parameter range, with results extracted from comparative benchmarking against both similarly sized and much larger models:

Benchmark	Step3-VL-10B SeRe	Step3-VL-10B PaCoRe	GLM-106B	Qwen3-235B	Gemini-2.5-Pro	Seed-1.5-VL
MMMU	78.11	80.11	75.20	78.70	83.89	79.11
MathVision	70.81	75.95	63.50	72.10	73.30	68.70
AIME2025	87.66	94.43	71.88	83.59	83.96	64.06
MMBench (avg)	91.80	92.17	92.75	92.70	93.19	92.11

In compact model comparisons (7–10B parameters), Step3-VL-10B (92.05% on MMBench-EN, 78.11% on MMMU, 70.81% MathVision) outperforms GLM-4.6V-Flash and Qwen3-VL-8B, with 94.43% on AIME2025 exceeding previous state-of-the-art in comparable footprint models (Huang et al., 14 Jan 2026).

6. Training, Inference Modes, and Practical Considerations

Training: Pre-training covers 370k iterations on 1.2T tokens over 256–512 NVIDIA A100-80GB GPUs for roughly four weeks. Post-training (SFT + RL) involves about 2,000 PPO iterations using 128 GPUs for ~2 weeks.

Inference: A single 80GB GPU suffices for SeRe mode (≤65,536 tokens context). PaCoRe requires multi-GPU or pipeline parallelism for 131,072 token context. Throughput is approximately 8 tokens/s/GPU under default settings (temperature 1.0, top-p 1.0).

Inference Modes:

• SeRe (Sequential Reasoning): Chain-of-thought is wrapped in `` tags, max length 65,536 tokens.
• PaCoRe: 16 SeRe rollouts synthesized via a unified template, max context doubled.

7. Significance and Integration Context

Step3-VL-10B’s design—fully unfrozen, unified pre-training, saturation-free RL, and scalable test-time synthesis—produces strong intrinsic vision-language fusion and robust multimodal reasoning. The test-time scaling via PaCoRe represents a distinctive shift: rather than increasing model size, inference-time resources are allocated to hypothesis diversity and verification for perceptual reasoning, yielding ∆ performance gains on analytical multimodal tasks. This suggests architectural and algorithmic efficiency can offset the historical reliance on massively scaling parameter counts for performance increases.

The foundation is suitable for embedding high-accuracy VLP pipelines—such as Cam-shift + UKF-based trackers and geometry solvers—within modular, robust, and resource-efficient multimodal agents. Accurate calibration, GPU acceleration for visual primitives, and modular ROS interfaces are recommended for deployment in applications requiring both real-time tracking and advanced visual reasoning (Huang et al., 2021, Huang et al., 14 Jan 2026).