Zero-Shot Adjustable Acceleration for LLMs

• Zero-Shot Adjustable Acceleration is a training-and-inference strategy that enables a single fine-tuned model to operate at multiple speed levels without additional retraining.
• It employs Dynamic Resource Attention to prune low-contribution word vectors, allowing configurable adjustments in computing resources and speed-accuracy trade-offs.
• The approach is validated on various LLM architectures, demonstrating significant speedup and controlled performance loss in diverse deployment scenarios.

Zero-Shot Adjustable Acceleration is a training-and-inference scheme for LLMs that allows a single fine-tuned model to operate at different speed levels after fine-tuning, without any extra retraining or re-finetuning. Introduced in "Efficient LLMs with Zero-Shot Adjustable Acceleration" (Kachuee et al., 1 Sep 2025), it is designed for deployments in which the latency, throughput, memory, and compute budget can vary over time. The method makes the model robust to a range of pruning settings during fine-tuning, then exposes an inference-time control parameter $\alpha$ that determines how aggressively computation is reduced. Its central mechanism, Dynamic Resource Attention, uses attention softmax scores to estimate token contribution and prunes low-contribution word vectors across layers, yielding adjustable acceleration with controlled performance loss.

1. Definition and deployment setting

The paper frames Zero-Shot Adjustable Acceleration as a response to a practical deployment problem: modern LLMs are powerful but expensive to train and serve, and real-world systems do not always require the same latency-throughput-accuracy trade-off (Kachuee et al., 1 Sep 2025). In settings such as industrial monitoring, safety systems, interactive services, and fluctuating workloads, the preferred operating point can shift across requests or hardware conditions. Sometimes the system can afford maximum accuracy; at other times it must reduce latency and accept some performance loss.

The defining claim is that the acceleration level can be changed after fine-tuning, in a zero-shot manner. In this usage, “zero-shot” does not mean that the model is never fine-tuned; it means that once the model has been fine-tuned under varying acceleration conditions, the inference-time speed level can be changed on the fly without additional retraining or re-finetuning. The paper’s goal is therefore not merely faster inference, but stable, adjustable acceleration from a single trained model.

The method is positioned against several weaknesses of prior acceleration strategies. Fixed-speed pruning or exiting methods work only at one setting and can degrade when the acceleration level is changed after training. Scheduled or progressively varying acceleration during fine-tuning can bias the model toward the last or most frequent settings. Many methods also require additional fine-tuning to move to a different speed-accuracy point. Zero-Shot Adjustable Acceleration is proposed specifically to avoid those constraints.

2. Dynamic Resource Attention and word-vector pruning

The mechanism is based on the observation that, within attention, not all token or word vectors contribute equally to the layer output (Kachuee et al., 1 Sep 2025). The paper uses the attention softmax scores to estimate contribution and then removes low-contribution word vectors in later layers. The attention formulation given is

$Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$

where $Q$, $K$, and $V$ are the query, key, and value matrices, $d_k$ is the key-dimension scaling term, and $M$ is the attention mask. For causal attention, $M$ is a lower-triangular mask with $-\infty$ on future positions; for bidirectional attention, $M=0$. The softmax output, described as the “soft-score,” is interpreted as a token-to-token importance distribution.

This contribution estimate drives Dynamic Resource Attention, the pruning mechanism inserted into the attention layer. At each layer, the number of remaining word vectors is

$Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$0

where $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$1 is the number of remaining word vectors at layer $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$2, $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$3 is the number of remaining vectors from the previous layer, $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$4 is the elimination-rate hyperparameter, and $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$5 is a minimum token threshold. Smaller $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$6 means more aggressive pruning and higher speedup. The threshold $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$7 prevents collapse to too few tokens, which would make the model unstable or unusable; the paper states that the clipping threshold improves stability, especially at high pruning rates.

The effect of this layerwise rule is a progressive reduction in token count across the network. The paper explicitly states that this lowers attention and downstream compute, and therefore reduces latency, MACs, and TTFT. A plausible implication is that the method is most naturally aligned with Transformer deployments in which token-level intermediate-state management dominates part of the inference budget.

3. Training policy and the meaning of “zero-shot adjustable”

What makes the method “zero-shot adjustable” is not only the pruning rule but the way the model is fine-tuned (Kachuee et al., 1 Sep 2025). Rather than training for a single pruning level, the acceleration rate is sampled from a uniform distribution for each batch:

$Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$8

This means that different batches can be trained with preservation ratios ranging from $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$9 to $Q$0 per layer. The paper argues that this prevents the model from specializing to one speed setting and avoids the bias that can arise under scheduled acceleration schemes. By exposing the model to a broad spectrum of compute budgets during fine-tuning, the procedure is intended to make inference-time speed control robust across the full range.

The reported training workflow is: modify the attention layer to support token pruning; for each batch, sample $Q$1; apply Dynamic Resource Attention and prune word vectors according to the sampled rate; keep at least $Q$2 tokens per layer; and fine-tune the model under these varying acceleration conditions. During inference, the operator chooses the desired $Q$3 according to hardware or latency needs, the model prunes layer by layer using $Q$4, and no further training is needed.

A common misunderstanding is to treat this as post-training approximation in the strict sense. The paper does not describe a purely post-training rewrite of the model; instead, it describes a fine-tuning and inference strategy in which the model is trained to tolerate a range of pruning rates and then deployed at any chosen point in that range. The “zero-shot” property therefore concerns inference-time adjustability after fine-tuning, not the absence of training.

4. Architectural scope and evaluation protocol

The paper does not propose a wholly new Transformer architecture. Instead, it keeps the core architecture unchanged and adds a pruning-based mechanism inside the attention layer (Kachuee et al., 1 Sep 2025). This design choice is central to the claim of broad applicability. The paper notes that many modern decoder-only LLMs are structurally similar, naming GPT-2, Gemma 2, Mistral, and LLaMA 3 as examples, and presents the method as a universal acceleration layer rather than a model-specific redesign.

The empirical evaluation uses GPT-2 as a legacy model and LLaMA 3 and Gemma 2 as modern state-of-the-art models. For LLaMA 3 and Gemma 2 fine-tuning, the paper uses LoRA for parameter-efficient tuning, BitsAndBytes quantization for reduced memory and compute, and 4-bit execution on GPU for some experiments. The setup is implemented with Hugging Face and PyTorch on an NVIDIA Tesla T4 GPU, with learning rate $Q$5, batch size $Q$6, epochs $Q$7, dropout $Q$8, and minimum preserved vectors $Q$9. The reported $K$0 range is $K$1 to $K$2 for GPT-2 and $K$3 to $K$4 for LLaMA 3 and Gemma 2.

Evaluation type	Benchmarks and datasets	Models
Classification / NLU	IMDB; GLUE: SST-2, MRPC, QQP, QNLI, RTE, MNLI-m, MNLI-mm	GPT-2
Text generation / LM	WikiText-103, Penn Treebank, One Billion Word, LAMBADA	GPT-2
Instruction tuning	MMLU	LLaMA 3, Gemma 2

The baselines are primarily the same model at different acceleration levels, with the $K$5 configuration as reference. The stated objective is not a new state-of-the-art comparison against unrelated methods, but an evaluation of whether a single model can preserve performance while being dynamically accelerated.

5. Reported empirical behavior

The main quantitative finding is that decreasing $K$6 increases speed, with up to $K$7 speedup on IMDB and about $K$8 average latency reduction for GPT-2 (Kachuee et al., 1 Sep 2025). On IMDB, the representative results reported are $K$9 at $V$0, $V$1 at $V$2, $V$3 at $V$4, $V$5 at $V$6, $V$7 at $V$8, $V$9 at $d_k$0, and $d_k$1 at $d_k$2. The pattern is the expected speed-accuracy trade-off: moderate acceleration preserves most performance, while very aggressive pruning causes a larger drop.

On GLUE, the paper reports similar behavior. Representative examples are MNLI-m at $d_k$3 for $d_k$4, $d_k$5 for $d_k$6, and $d_k$7 for $d_k$8; RTE at $d_k$9 for $M$0 and $M$1 for $M$2; and QQP at $M$3 for $M$4 and $M$5 for $M$6. These results are used to support the claim that adjustable acceleration remains functional over a broad operating range rather than only at a single tuned point.

For GPT-2 language modeling, perplexity increases as pruning becomes more aggressive, but TTFT speedups are reported up to $M$7. The examples given are WikiText-103 perplexity $M$8 at $M$9 and $M$0 at $M$1; LAMBADA perplexity $M$2 at $M$3 and $M$4 at $M$5; and PTB perplexity $M$6 at $M$7 and $M$8 at $M$9. The paper interprets this as evidence that word-vector pruning retains useful context.

For LLaMA 3 and Gemma 2 on MMLU, the paper reports MAC reduction rather than TTFT, describing this as more suitable for quantized GPU inference. The reported values are:

Model	Speed level	Reported value
LLaMA-3-8B-4bit	$-\infty$0, $-\infty$1, $-\infty$2, $-\infty$3	$-\infty$4, $-\infty$5, $-\infty$6, $-\infty$7
Gemma-2-2B-4bit	$-\infty$8, $-\infty$9, $M=0$0, $M=0$1	$M=0$2, $M=0$3, $M=0$4, $M=0$5

These results are presented as evidence that the method remains compatible with quantization and LoRA on modern decoder-only models.

6. Relation to prior adjustable acceleration work, limitations, and implications

The paper situates Zero-Shot Adjustable Acceleration within three broader families of efficiency methods: model compression, including weight pruning, quantization, and distillation; efficient architectures, including MoE, sparse attention, and long-context attention variants; and dynamic inference, including early exit, dynamic routing, token or layer skipping, and resource management (Kachuee et al., 1 Sep 2025). Its distinctives are stated as token-level pruning inside attention driven by actual contribution, zero-shot adjustability from one trained model to many speeds, no architecture replacement, compatibility with LoRA and quantization, and a stable acceleration range rather than a single speed point.

A useful contrast is Deep Learning Approximation, which is also presented as a zero-shot adjustable acceleration method but in a different sense (Pratusevich, 2018). DLA is a post-training method that rewrites a trained network into a faster one without retraining and without access to training data, using lossless structural simplifications and lossy SVD-based low-rank approximations selected by a single user parameter $M=0$6. On YOLO / PASCAL VOC 2007, it reports a $M=0$7 end-to-end speedup, with runtime changing from $M=0$8 ms to $M=0$9 ms and mAP from $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$00 to $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$01. The comparison clarifies that “adjustable acceleration” can denote either post-training structural approximation, as in DLA, or a training-and-inference regime that yields inference-time speed control after fine-tuning, as in Zero-Shot Adjustable Acceleration.

The terminology also differs from speed-adjustable control policies in robotics. In tactile in-hand manipulation, a policy can be conditioned at deployment on a target-speed-related signal and then transferred zero-shot to hardware without retraining, but the controlled quantity is task execution speed rather than LLM inference hardware usage (Pitz et al., 2024). This suggests that “zero-shot adjustable” is a broader design pattern whose precise meaning depends on whether the object of control is model computation or embodied behavior.

The limitations reported for Zero-Shot Adjustable Acceleration are straightforward. More pruning gives more speed but can reduce accuracy or worsen perplexity. The method depends on good calibration of $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$02 and $Att(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}+M})V$03. Very high pruning rates can still harm downstream performance. The results are shown on a specific set of tasks and models, so broader generalization is suggested but not exhaustively proven. Compatibility is strongest within Transformer architectures with similar decoder structure.

Its practical implication is that an LLM can be deployed once and then run at different compute budgets dynamically, without retraining for each speed setting. The paper explicitly identifies variable-load serving systems, low-latency applications, edge or resource-constrained environments, and mixed workloads as settings in which this property matters. Because the core model is not altered and the approach is combined with LoRA and 4-bit quantization in the reported experiments, the method is presented as deployment-oriented rather than architecture-specific.