Qwen3-Coder-480B-A35B-Instruct Model

• The model’s fine-tuning on 35B code-oriented instructions significantly enhances invariant synthesis and program verification performance.
• Qwen3-Coder-480B-A35B-Instruct is defined by its transformer architecture, leveraging 480B parameters, extensive code datasets, and precise token-level optimization using LoRA adapters.
• Rigorous verifier-based evaluations reveal that while the model accelerates program verification on easier benchmarks, challenges remain for more complex, hard instances.

Qwen3-Coder-480B-A35B-Instruct is a large-scale autoregressive transformer-based LLM designed for code-centric tasks, particularly targeting program verification with invariant synthesis. As a member of the Qwen3 family, it leverages 480 billion parameters and an extensive mixture of code and natural language data for pretraining. The A35B-Instruct variant is further adapted for instruction-following behavior via extensive supervised finetuning on code-oriented prompts and optimized for invariant generation in loop verification pipelines. Its performance has been rigorously evaluated within a formal, verifier-based benchmarking framework, revealing both the potential and current limitations of LLM-augmented program verification workflows (Wei et al., 25 Sep 2025).

1. Model Architecture and Instruction Tuning

Qwen3-Coder-480B is a 480B-parameter, decoder-only transformer with 48 layers, a hidden size of 7,168, 64 attention heads, and a feed-forward dimension of 28,672. The model employs absolute positional encodings and is pretrained on approximately 2 trillion tokens from code and natural language sources. A35B-Instruct represents an instruction-tuned variant, further supervised on 35 billion instruction-following examples targeting code completion, debugging, and docstring generation. The model's instruction-following proficiency is enhanced with ranking losses inspired by reinforcement learning from human feedback (RLHF), improving compliance with "Write an invariant" prompts. LoRA adapters are used in select layers for parameter-efficient adaptation and instruction tuning.

2. Supervised Finetuning for Invariant Synthesis

For the invariant synthesis domain, Qwen3-Coder-480B-A35B-Instruct is fine-tuned on 3,589 synthetic C programs generated via GPT-4o seeding, with invariants extracted from UAutomizer logs. No overlap exists between the training set and the 226 evaluation instances. The fine-tuning objective is token-level cross-entropy on concatenated prompt-invariant sequences. Optimization utilizes LoRA with a rank of 32 for parameter efficiency across three epochs, using a batch size of 16 per GPU, a learning rate of $10^{-4}$ with linear decay, a 500-step warmup, weight decay of 0.01, and the AdamW optimizer ($\beta_1=0.9,\ \beta_2=0.999$). The model accommodates inputs up to 8k tokens, enabling full-program and invariant context.

3. Verifier-Based Decision Framework and Soundness

Invariant quality is assessed via a formal verifier-based decision procedure. Each candidate invariant $q=\langle\psi,\ell\rangle$ is evaluated for self-invariance and its ability to imply a target property $p^* = \langle\varphi^*,\ell^*\rangle$ by issuing two queries:

• $d_a := V(P,\emptyset,q)$ (verifies $q$ is an invariant for program $P$)
• $d_b := V(P,\{q\},p^*)$ (checks if $p^*$ holds assuming $q$)

A judgment $P \Rightarrow \langle p^*,q\rangle \Downarrow d$ where $d\in\{\checkmark,\,\times,\,?\}$ is derived by:

• (DEC-FALSE): If $V(P,\{q\},p^*) = \times$, then $$P \Rightarrow \langle p^*,q\rangle \Downarrow \times$$
• (DEC-PROP): If $V(P,\emptyset,q) = \checkmark$ and $V(P,\{q\},p^*) = d$ ($d\in\{\checkmark,\times\}$), then $P \Rightarrow \langle p^*,q\rangle \Downarrow d$
• (DEC-?): If $V(P,\emptyset,q)\ne\checkmark$ and $V(P,\{q\},p^*)\ne\times$, then $P \Rightarrow \langle p^*,q\rangle \Downarrow ?$

A formal soundness result guarantees:

• If $$P \Rightarrow \langle p^*,q\rangle \Downarrow \checkmark$$, then $P \models p^*$
• If $$P \Rightarrow \langle p^*,q\rangle \Downarrow \times$$, then $P \not\models p^*$

For loop invariants, assessment is rooted in the Hoare logic inference rule:

$$\frac{ P \;\Rightarrow\; I \quad \{\,I\wedge B\}\,S\,\{I\} \quad I\wedge\lnot B \;\Rightarrow\; Q }{ \{P\}\,\text{while }B\,\text{do }S\;\{Q\} }$$

Thus, an acceptable candidate $I$ must satisfy initiation, preservation, and post-condition properties.

4. Empirical Results: Invariant Quality and Solver Acceleration

Qwen3-Coder-480B-A35B-Instruct was evaluated on InvBench-Easy and InvBench-Hard splits, each with 113 program instances. Speedup is measured relative to UAutomizer's baseline solve time. Key quantitative results are summarized:

Table 1. InvBench-Easy (113 instances):

Model	% Correct	% Speedup	Speedup₍>1₎	Speedup₍all₎
Qwen3-Coder-480B (base)	14.2%	8.0%	1.09×	1.01×
Qwen3-Coder-480B (ft)	40.7%	29.2%	1.29×	1.08×

Table 2. InvBench-Hard (113 instances):

Model	% Correct	% Speedup	Speedup₍>1₎	Speedup₍all₎
Qwen3-Coder-480B (base)	15.9%	0%	1.00×	1.00×
Qwen3-Coder-480B (ft) (Bo16)	27.4%	2.7%	1.35×	1.02×

Best-of-16 sampling was required to elicit nontrivial speedups on the hard split. While fine-tuning led to a 3.6× relative improvement on easy programs (8% to 29.2% speedup cases), the gains plateau on more complex benchmarks.

5. Comparison with Baseline Solvers and LLM-Based Verifiers

UAutomizer, a non-LLM program verifier, solves all easy (≤30 s/instance) and hard (≤600 s/instance) splits. Among LLM-based verifiers, Qwen3-Coder-480B-A35B-Instruct in its fine-tuned state achieves parity or near-parity with state-of-the-art models:

• On InvBench-Easy:
  • gpt-5: 37.2% correct, 26.5% speedup
  • o3: 39.8% correct, 28.3% speedup
  • Qwen3-Coder-480B (ft): 40.7% correct, 29.2% speedup
• On InvBench-Hard, other LLM-based verifiers (LaM4Inv, Loopy, LEMUR) lag behind, with UAutomizer remaining the only solver to resolve all instances.

Unique solves (instances verified beyond UAutomizer's capabilities) were attained by LLM-based verifiers: LaM4Inv (13), Loopy (40), LEMUR (19) on the full set.

6. Key Insights and Practical Implications

Empirical analysis highlights several conclusions:

• Model Capacity: Larger models and robust instruction tuning (A35B-Instruct) are necessary for significant performance; base Qwen3-Coder-480B lagged state-of-the-art until supervised fine-tuning.
• Invariant Quality: High rates of correct invariant synthesis do not always translate into solver speedup; only invariants that sufficiently prune the verifier’s search reduce verification time.
• Hard Benchmark Plateau: Fine-tuning and best-of-N generation strategies yield diminishing returns on hard instances, indicating current architectures and data alone are insufficient for universally strong invariant synthesis.
• LLM–Solver Hybridization: LLM-based verification approaches (such as Qwen3-Coder-480B-A35B-Instruct) solve a subset of problems that traditional solvers do not, suggesting value in hybrid workflows where LLMs propose invariants for subsequent mechanized verification.
• Inference Overhead: Token-generation latency is included in overall evaluation, highlighting a trade-off between LLM inference cost and potential solver acceleration.

A plausible implication is that progress in tightening LLM-verifier integration and adopting more adaptive sampling or feedback strategies may be necessary to overcome current limitations on hard verification tasks.

7. Future Directions and Open Challenges

Qwen3-Coder-480B-A35B-Instruct’s performance evidences both the advances and constraints of current LLM-based invariant synthesis. Although state-of-the-art on nontrivial subsets, it does not yet eclipse the breadth or consistency of domain-specific tools such as UAutomizer. Continued research is suggested in:

• Tighter LLM–verifier interaction loops for dynamic feedback.
• Development of hybrid strategies leveraging LLM proposal with symbolic fallback verification.
• Exploration of more expressive prompting and adaptive sampling, particularly for complex or “hard” program instances.

Program verification via LLM-driven invariant synthesis remains an open challenge, with Qwen3-Coder-480B-A35B-Instruct constituting a significant milestone in the empirical evaluation and methodological advancement of this research frontier (Wei et al., 25 Sep 2025).