QuaLA-MiniLM: Adaptive, Quantized Transformer

• The model employs knowledge distillation along with a length-adaptive transformer mechanism to optimize computation across varied FLOPs budgets.
• It uses token dropping and evolutionary search to dynamically select key tokens per layer, achieving up to an 8.8× speedup on QA tasks with less than 1% F1 loss.
• Post-training 8-bit quantization reduces latency and model size, enabling robust inference in resource-constrained environments without significant accuracy degradation.

QuaLA-MiniLM is a transformer-based natural language processing model that combines three methodologies—knowledge distillation, length-adaptive token dropping, and post-training 8-bit quantization—into a single architecture. It enables a single trained model to serve a wide range of inference-time FLOPs or latency budgets via dynamic sequence length adaptation, while maintaining high accuracy with markedly reduced computational demands. Empirical evaluation on SQuAD1.1 demonstrates that QuaLA-MiniLM achieves up to an 8.8× real-world speedup on CPU (Intel Xeon Platinum 8280) with less than 1% F1 score loss relative to BERT-base, outperforming state-of-the-art efficient transformer variants across all tested computational budgets (Guskin et al., 2022).

1. Model Structure and Length Adaptation

QuaLA-MiniLM adopts a compact transformer architecture as its "MiniLM student," comprising 6 layers, hidden size $d=384$, intermediate feed-forward dimension $4d=1536$, and 12 attention heads (each of dimension $d_k = 32$), resulting in approximately 24 million parameters. Each layer computes: $$H^{(i)} = \mathrm{TransformerLayer}_i(H^{(i-1)}), \quad H^{(0)} = \text{input embeddings}$$

The model departs from fixed-length processing by implementing the Length-Adaptive Transformer (LAT) mechanism. In LAT, each of the $L=6$ layers retains only $l_i \leq N$ tokens, with the "length configuration" defined by

$\ell = (l_1, \dots, l_L)$

Unselected tokens are restored before the output layer for prediction. During training, "LengthDrop" randomly subsamples tokens at each layer, and "sandwich" distillation ensures that the full model provides supervision to short sub-models. At inference, evolutionary search determines an optimal length configuration $\ell^*(B)$ for any compute budget $B$ (in FLOPs), using token-wise importance scoring to select tokens per layer. Unlike dynamic-depth methods which involve binary layer gating, QuaLA-MiniLM fixes network depth ($g_i \equiv 1$) and adapts token lengths under the constraint: $$\mathrm{FLOPs}(\ell) = \sum_{i=1}^L c\,l_i^2 + \cdots \leq B_{\mathrm{FLOPs}}$$ The best configuration per budget is found by

$$\ell^*(B) = \arg\max_{\ell: \mathrm{FLOPs}(\ell)\leq B} \mathrm{F1}(\ell)$$

2. Knowledge Distillation Objective

QuaLA-MiniLM's student model is trained via joint distillation from a large pre-trained teacher (RoBERTa-Large or BERT-base). The training consists of two phases: task-agnostic MiniLM distillation (deep attention distillation on unlabeled text), then fine-tuning for span-based question answering (SQuAD1.1) with a composite loss. The loss components are:

• Attention-relation loss (MiniLM style):

$$\mathcal{L}_{\mathrm{attn}} = \frac{1}{L} \sum_{i=1}^L \bigl\| R(Q_t^{(i)}, K_t^{(i)}) - R(Q_s^{(i)}, K_s^{(i)}) \bigr\|_2^2$$

where $R(Q, K) = QK^\top/\sqrt{d_k}$.

• Hidden-state loss:

$$\mathcal{L}_{\mathrm{hid}} = \frac{1}{L}\sum_{i=1}^L \| H_t^{(i)} - H_s^{(i)} \|_2^2$$

• Logit-based distillation:

$$\mathcal{L}_{\mathrm{logit}} = T^2\, \mathrm{KL}\bigl(\mathrm{softmax}(z_t/T) \Vert \mathrm{softmax}(z_s/T)\bigr)$$

with temperature $T$.

• Supervised QA loss: cross-entropy on SQuAD1.1 start/end positions, $\mathcal{L}_{\mathrm{CE}}$.

During "Drop-and-Restore" training, an additional LAT distillation loss is included: $$\mathcal{L}_{\mathrm{LAT}} = \frac{1}{|\mathcal{S}|} \sum_{\ell'\in\mathcal{S}} \mathrm{KL}(p_{\mathrm{full}} || p_{\ell'})$$ where $\mathcal{S}$ is a set of sampled length configurations. The total loss is a weighted sum: $$\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \alpha\,\mathcal{L}_{\mathrm{logit}} + \beta\,\mathcal{L}_{\mathrm{attn}} + \gamma\,\mathcal{L}_{\mathrm{hid}} + \delta\,\mathcal{L}_{\mathrm{LAT}}$$

3. Quantization Methodology

QuaLA-MiniLM integrates post-training 8-bit uniform quantization of weights and activations. Quantization is performed per-tensor using min/max calibration on a small held-out set. Given floating-point element $r$: $$q = Q(r) = \mathrm{clip}\!\left(\mathrm{round}\left(\frac{r}{S}\right) + Z,\, q_{\min}, q_{\max}\right)$$

$r_{\mathrm{dequant}} = S\,(q - Z)$

where $S = (r_{\max} - r_{\min}) / (q_{\max} - q_{\min})$, $Z = q_{\min} - r_{\min}/S$, and $q \in \{q_{\min}, \dots, q_{\max}\}$ (256 bins for INT8). No quantization-aware retraining is applied. Accuracy loss from quantization alone remains below 1%.

4. Inference-Time Operation and Adaptivity

After single-pass training, QuaLA-MiniLM flexibly matches any target FLOPs or latency budget at inference. An offline evolutionary search computes the Pareto frontier of length configurations, mapping each budget $B$ to an optimal configuration $\ell^*(B)$. At inference, the model uses the precomputed configuration, retaining top-scoring tokens at each layer based on their embedding L2 norm, as tokens for the next layer. Quantization precision remains fixed to INT8 for all layers.

This approach enables dynamic adaptation to deployed platform constraints without the need for retraining multiple models for different budgets.

5. Performance and Comparative Analysis

Empirical evaluation on SQuAD1.1 (CPU, Intel Xeon Platinum 8280) demonstrates that QuaLA-MiniLM consistently outperforms peer models (including BERT-base, TinyBERT, Dynamic-TinyBERT, and various quantized/minimalist variants) in the trade-off between F1 score, latency, and FLOPs. The following table summarizes key results:

Model	Model Size (MB)	F1 Score	Latency (ms)	Speedup
BERT-base	415.5	88.58	56.6	1×
TinyBERT	253.2	88.40	32.4	1.74×
QuaTinyBERT	132.1	87.68	15.6	3.63×
MiniLMv2	115.0	88.70	18.2	3.10×
QuaMiniLMv2	84.9	88.55	9.1	6.18×
LA-MiniLM (drop)	115.0	87.76	11.4	4.94×
QuaLA-MiniLM (drop)	84.9	87.68	6.41	8.80×

QuaLA-MiniLM with combined length adaptation and quantization achieves up to 8.8× end-to-end speedup with less than 1 percentage point F1 drop. Length-adaptive MiniLM (without quantization) yields up to 4.9× speedup for ≈1% lower F1, while quantized MiniLMv2 yields ≈2× speedup with negligible loss. Ablation indicates quantization roughly halves latency at ≈0.4 pp F1 cost compared to length adaptation alone.

The F1 versus FLOPs trend demonstrates that QuaLA-MiniLM dominates Dynamic-TinyBERT across all computational budgets, and at high budgets matches or exceeds BERT-base accuracy (e.g., at ≈7× speedup, F1=88.8 versus 88.6 for BERT-base).

6. Significance and Broader Implications

QuaLA-MiniLM presents a robust approach for efficient NLP model deployment in resource-constrained environments. Its unified application of knowledge distillation, dynamic length adaptation, and post-training quantization allows practitioners to deploy a single model under arbitrary computational constraints, obviating the need for multiple model variants.

This design advances the Pareto frontier of speed–accuracy tradeoffs in transformer-based QA, demonstrating that length dynamically-adaptive and quantized models can largely close the performance gap to cumbersome teacher models even at aggressive acceleration factors (Guskin et al., 2022). A plausible implication is that similar methodologies could generalize to other transformer architectures and NLP tasks where dynamic budget constraints must be met.