Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B

Abstract: This report describes our approach to the Efficient Qwen Competition, where the goal is to enable low-latency serving of Qwen3.5-4B on a resource-constrained NVIDIA A10G GPU. Our system combines a quantized target model with speculative decoding. To recover accuracy, we apply quantization-aware distillation to the target model while retaining the original quantization grid. To speed up decoding, a block-diffusion drafter specialized for the quantized target model is trained using a two-stage procedure: first learning from the high-precision target and then adapting to the low-precision target. Because the drafter is invoked at every speculative decoding step, we further reduce its overhead with quantization and sliding-window attention, preserving draft-token acceptance while improving long-context decoding latency. As a result, our submission achieves a 6.978$\times$ average speedup over the baseline while satisfying the required quality thresholds, ranking 3rd overall. We hope these results provide useful insights for practical LLM inference. The code and resources are available at https://github.com/nota-github/adaptfm-quant-dflash

• The paper introduces a grid-preserving quantization-aware distillation (QAD) to convert Qwen3.5-4B to INT4, maintaining high benchmark performance.
• It employs a two-stage training for the speculative decoder to ensure robust alignment and high draft-token acceptance while reducing per-step inference overhead.
• The system achieves nearly 7× inference speedup with sliding-window attention, highlighting efficiency improvements for LLM inference under tight memory constraints.

Quantized Speculative Decoding for Qwen3.5-4B: A Systematic Approach to Efficient LLM Inference

Motivation and Challenge

The paper addresses the challenge of achieving efficient, low-latency inference for the Qwen3.5-4B LLM under tight memory and compute constraints, specifically targeting single NVIDIA A10G GPU hardware. The imperative is to maximize throughput via aggressive model compression and speculative decoding, while rigorously maintaining accuracy thresholds imposed by standard LLM evaluation benchmarks. The methodology targets two primary system bottlenecks: (1) the high computational demand of transformer inference, especially with long-context inputs, and (2) the overheads introduced by speculative decoding drafters, which are invoked per decoding step.

System Overview and Key Techniques

The core of the presented system is the dual quantization of both the target LLM and the speculative decoder (drafter), systematically aligned to maintain output fidelity. The approach features the following key innovations:

• Quantization-Aware Distillation (QAD): The target model, Qwen3.5-4B, is first post-training quantized to INT4 using AWQ techniques, followed by fine-tuning via distillation from the original full-precision model. This process preserves the original quantization grid, ensuring deployment efficiency and maintaining model output distributions within accuracy bounds.
• Block Diffusion Drafter (DFlash) for Speculative Decoding: The drafter, central to speculative decoding, is trained using a two-stage regime: initial training with the full-precision (BF16) model for robust alignment, and a subsequent fine-tuning on the quantized INT4 target, with both stages using teacher-forced data to assure numerical alignment. The drafter is further quantized via GPTQ and adapted with sliding-window attention, both of which are critical in mitigating the per-step overhead and ensuring scaling to long contexts.

Figure 1: System employs speculative decoding with both target model and drafter quantized; sliding-window attention reduces long-context overhead.

Figure 2: Development pipeline: QAD produces the INT4 target; DFlash drafter trained and quantized in two stages, with SWA for efficiency.

Experimental Pipeline and Implementation Details

Quantization-Aware Distillation (QAD)

Direct INT4 post-training quantization (PTQ) of Qwen3.5-4B significantly degrades evaluation metrics, particularly IFEval and MMLU-Pro. QAD is applied by initializing the student with dequantized weights and optimizing against a frozen BF16 teacher using a per-token forward KL divergence objective. The quantization grid is held fixed by freezing per-group scales, thus maintaining deployment compatibility. After distillation, the updated weights are remapped to AWQ INT4 format for efficient inference.

Two-Stage Drafter Training

The DFlash drafter (537M parameters, block size 16) is trained in two concise phases:

1. Stage 1: Pretraining against the BF16 target using 220K conversations, forming a robust basis for block drafting.
2. Stage 2: Continued finetuning against the QAD-enhanced INT4 model with 400K conversations, strictly aligning the drafter to the quantized model’s token distribution.

Drafter Quantization and Sliding-Window Attention

Drafter inference cost is reduced through PTQ (evaluated using RTN, AWQ, GPTQ), with GPTQ chosen for superior trade-offs. Post-quantization, the drafter is further optimized using sliding-window attention (SWA), restricting attention computation to a local window of recent tokens (window sizes 1024–2048). This step both curtails compute requirements and, empirically, boosts mean draft-token acceptance lengths for long-context tasks, critical for overall system speedup.

Empirical Results

• Latency & Throughput: The fully optimized system achieves a 6.978× average speedup over the full-precision baseline for inference on NVIDIA A10G, satisfying all competition-imposed quality constraints.
• Accuracy Preservation: QAD on the INT4 model recovers benchmark performance to MMLU-Pro 0.6610, IFEval 0.8285, and GPQA-Diamond 0.6626, surpassing all required gates. Speculative decoding does not introduce any further accuracy degradation.
• Drafter Quality: Two-stage drafter training marginally improves mean acceptance length vs. direct INT4 training. Drafter quantization yields only minimal reductions in acceptance rates, and SWA with moderate window sizes yields substantial latency reduction in long-context regimes without penalizing acceptance.

Implications and Prospects

This system design and empirical evaluation underscore several important practical and theoretical points for future efficient LLM deployment:

• Grid-Preserving QAD: Preserving the quantization grid during QAD enables seamless low-bit deployment, with no need for backend-specific retraining or grid mismatch, a consideration critical in production LLM serving environments.
• Alignment-Optimized Speculative Decoding: Two-stage drafter alignment ensures high draft-token acceptance on quantized targets, emphasizing the importance of distributional alignment in speculative parallelization strategies.
• SWA as a Universal Optimization: The use of sliding-window attention presents a generalizable approach to reigning in quadratic attention complexity, which will likely transfer well to broader classes of hardware and LLM configurations.
• Systemic Quantization: Demonstrates that simultaneous low-bit quantization for both main and auxiliary models (e.g., speculative drafters) is viable—contradicting prior perceptions that auxiliary models must remain high-precision to maintain proposal quality under aggressive quantization.

Conclusion

This work presents a rigorously optimized inference pipeline for Qwen3.5-4B, combining QAD, two-stage drafter alignment, and system-wide INT4 quantization with SWA to attain near-lossless accuracy at nearly 7× speedup under memory-constrained conditions. These results will inform future architectural and systems research on quantized LLM serving, opening lines of investigation into broader auxiliary-model quantization, dynamic attention scaling, and advanced distillation for other model families.

Reference: "Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B" (2607.04244).