Qwen3.5-4B Transformer Model

• Qwen3.5-4B Model is a dense transformer-based language model featuring 4B parameters and specialized procedural demonstrations.
• Supervised fine-tuning with LoRA adapters and a chain-of-thought protocol significantly improves its step-by-step reasoning performance.
• It leverages extensive multilingual tokenization and a vast pretraining corpus to excel in various downstream tasks.

Qwen3.5-4B is a dense transformer-based LLM in the Qwen3.5 series, featuring approximately four billion parameters. It is designed to address procedural reasoning and skill transfer through supervised fine-tuning (SFT) with explicit procedural demonstrations. Qwen3.5-4B shares its core design and tokenization mechanisms with other Qwen3.5 models but distinguishes itself by its scale and documented behavior under procedural skill SFT, particularly in the context of step-by-step reasoning benchmarks. The architecture is derived from the Qwen3 lineage, which demonstrates state-of-the-art open-source performance in multiple downstream tasks and is evaluated against competitive proprietary and academic models (Strozzi, 12 May 2026, Bai et al., 2023).

1. Architectural Design

Qwen3.5-4B is implemented as a dense transformer with the following essential design elements, consistent with the Qwen3 series:

• Core Components: Standard decoder-only transformer with untied input embeddings and output projection, rotary positional embeddings (RoPE) with FP32 inverse-frequency, feed-forward blocks utilizing SwiGLU activation, and RMSNorm in place of Pre-LayerNorm.
• Attention and Tokenization: The attention mechanism includes QKV bias. Tokenization uses BPE (tiktoken’s cl100k), with explicit augmentation for multilingual coverage and Chinese characters; vocabulary size is approximately 152k.
• Parameterization: The 4B parameter variant extrapolates from published 1.8B and 7B specifications (Bai et al., 2023). A typical 4B configuration uses around 24 transformer layers and a model dimension near 3584, with a feed-forward width of $d_{ff} = (8/3) \, d_{model}$. Attention heads are set to 28. All Qwen3.5 models employ the same tokenizer and attention construction, varying only in layer depth and hidden dimension.
• LoRA Adapters: For SFT, LoRA adapters of rank 32 are injected into all linear modules and trained with 8-bit Adam.

2. Pretraining Data and Optimization

Qwen3.5-4B is pretrained on a composite corpus aggregating hundreds of public sources:

• Corpus Scale and Composition: The aggregate pretraining set for the Qwen3 family is up to $3 \times 10^{12}$ tokens, comprising web data, multilingual encyclopedias, books, open-source code (The Stack, CodeParrot), and instruction samples. Preprocessing includes deduplication (exact and fuzzy), language-ID filtering, perplexity-based quality scoring, toxicity filtering, and domain upsampling.
• Objective and Optimization: Pretraining objective is standard autoregressive maximum-likelihood estimation. Optimization uses AdamW ($\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$), with mixed BFloat16 arithmetic, weight decay (0.1), and dropout (0.1). Training employs FlashAttention for scalable sequence processing and gradient clipping at norm 1.0; cosine learning rate annealing is applied, starting at a peak suited for the model’s scale.
• Compute Requirements: Qwen-7B, the nearest published larger counterpart, is trained on $\sim$2.4T tokens with $\mathcal{O}(10^{23})$ FLOPs over several weeks on hundreds of A100/V100 GPUs; the 4B variant interpolates to similar per-token scaling (Bai et al., 2023).

3. Supervised Fine-Tuning Protocol

Qwen3.5-4B’s procedural skill transfer is primarily realized through chain-of-thought (CoT) SFT:

• SFT Corpus: The curated SFT set comprises 353 demonstration episodes, each with a system prompt (SOLVER plus a 5-step procedural-skill block including explicit criteria, constraints, and worked examples), a user prompt with a holdout task, and an Opus-4.7–synthesized assistant reply containing full CoT and a terminal ANSWER line.
• Baselines: The baseline condition omits the procedural-skill block, isolating its contribution.
• Fine-Tuning Implementation: LoRA adapters (rank 32) are used for parameter-efficient SFT, optimized with the bits-and-bytes 8-bit Adam configuration.
• Decoding Regimes: Inference uses greedy decoding with a repetition penalty of 1.05, max_new_tokens set to 2048 pre-SFT and 1024 post-SFT.
• Evaluation and Scoring: Matched-path LLM-only judging is performed by Opus-4.7, which bypasses previously used deterministic format-compliance rules. The main metric is $pass\_rate$ per condition (tasks passed / 200), with skill transfer effectiveness quantified by $$\Delta = pass\_rate(curated) - pass\_rate(baseline)$$ and the procedure-induced lift $\Delta_{lift}$ calculated as $\Delta_{post-SFT}$ minus $\Delta_{pre-SFT}$ (Strozzi, 12 May 2026).

4. Procedural Skill Transfer and Regime-Asymmetric Mechanism

Empirical results from (Strozzi, 12 May 2026) establish that procedural SFT yields a regime-asymmetric effect across Qwen3.5 scales. For the Qwen3.5-4B:

• Pre-SFT Performance: Under LLM-only judgments, $3 \times 10^{12}$0, $3 \times 10^{12}$1, so $3 \times 10^{12}$2.
• Post-SFT Performance (4B+LoRA r=32): $3 \times 10^{12}$3, $3 \times 10^{12}$4, so $3 \times 10^{12}$5 and $3 \times 10^{12}$6.
• Statistical Robustness: McNemar $3 \times 10^{12}$7 (exact), bootstrap 95% CI on $3 \times 10^{12}$8 is [0.030, 0.105].
• Capacity Trajectory: Across the 0.8B, 2B, and 4B scales, a W-shaped trajectory is observed: pre-SFT $3 \times 10^{12}$9 is negative for 0.8B (-0.075) and 4B (-0.010), positive for 2B (+0.060), and positive for the Haiku-4.5 reference (+0.030).
• Interpretation: SFT is most effective in regimes where the base model initially fails to leverage procedural instructions (i.e., where procedural guidance is seen as overhead). In such settings (e.g., the 4B variant), SFT flips the effect from negative to positive. Where the base already utilizes procedural information effectively, SFT offers smaller marginal gains. This regime-asymmetric phenomenon produces a falsifiable prediction: if future larger models inherently favor the 5-step procedural guidance, SFT-induced $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$0 should diminish toward the pattern seen in 2B models.

5. Evaluation Benchmarks and Cross-Family Validation

• Task Construction: Evaluation uses a holdout of 200 synthetic tasks spanning 40 procedural skills with controlled balance by answer type (YES_NO 61%, FREE_FORM 16.5%, SINGLE_WORD 11.5%, RANKING 11%), with each skill disjoint from the SFT corpus.
• Validation: All main findings are validated by cross-family LLM judges. GPT-5.4 (OpenRouter) agrees with Opus-4.7 on all per-model $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$1 directions for 2800 paired episodes, with a Cohen’s $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$2 and $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$3 episode-level agreement; maximum headline $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$4 shift is $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$5 percentage points.
• Format-Compliance Bias: A deterministic regex-based ANSWER-line extractor used in earlier benchmarks was found to systematically undercount valid free-form responses, disproportionately biasing against procedural SFT. LLM-only re-judging increased pass rates and corrected this artifact, raising confidence in measured procedural gains.

6. Methodological Constraints and Limitations

Several potential threats to the generality and interpretability of findings have been identified:

• Single-Seed Evaluations: All reported results are single-run, lacking multi-seed variance estimates.
• Corpus and Judge Overlap: Both the SFT corpus and the LLM judges are derived from Opus-4.7, raising the possibility of model-specific alignment biases. Cross-family validation partially addresses but does not eliminate this concern.
• Task Resolution: The test set, comprising 200 tasks, limits the $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$6 metric resolution to approximately $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$7 percentage points.
• Absence of Generic Instruction Ablation: The specificity of learned procedural skills versus broader instruction tuning was not isolated; effect claims pertain only to the procedural SFT configuration.
• Generalizability: Results strictly pertain to the Qwen3.5 family; extension to architectures outside this family remains unproven.
• Inference Regimes: Only greedy decoding was evaluated; robustness to different generation methods and out-of-distribution (OOD) tasks was assessed qualitatively.

7. Downstream Impact and Future Directions

The Qwen3.5-4B procedural SFT study provides an empirical basis for understanding how explicit step-by-step demonstration and LoRA-based SFT affect medium-scale transformer models’ ability to apply complex reasoning procedures. The key regime-asymmetric finding—that SFT gains are maximized precisely when the base model is least amenable to procedure injection—has falsifiable implications for scaling to 8B and 14B models. If larger models natively exploit procedural guidance, the procedural-SFT $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$8 may converge to that observed for 2B models; persistent negative or uniform $\beta_1=0.9,\ \beta_2=0.95,\ \epsilon=10^{-8}$9 would challenge this account (Strozzi, 12 May 2026). A plausible implication is that SFT strategies may need to be tailored to model scale and inherent reasoning capacity. The reliance on cross-family validation with strong agreement metrics consolidates the validity of these conclusions. Broadly, Qwen3.5-4B advances the methodological toolkit for evaluating and enhancing procedural skill generalization in LLMs, with immediate relevance for subsequent model scaling and architecture diversification within and beyond the Qwen ecosystem (Bai et al., 2023, Strozzi, 12 May 2026).