CALM: Confident Adaptive Language Modeling

• Confident Adaptive Language Modeling (CALM) is a framework that assigns computation dynamically per token using early-exit decoding based on measured confidence.
• It employs diverse confidence measures, including softmax-response, hidden-state saturation, and learned exit classifiers, to determine when tokens need full-layer processing.
• Empirical results show CALM reduces average layers per token and speeds up tasks like summarization, translation, and QA while maintaining high performance.

Searching arXiv for the specified papers to ground the article in current preprint records. Confident Adaptive Language Modeling (CALM) is a framework for dynamically allocating different amounts of compute per input and generation timestep in Transformer-based LLMs. Introduced as an early-exit decoding method, it is motivated by the observation that some continuations can be produced confidently from earlier layers, whereas others benefit from the model’s full depth. CALM addresses three coupled problems: choosing a confidence measure, connecting sequence-level constraints to local per-token exit decisions, and attending back to missing hidden representations due to early exits in previous tokens. On three diverse text generation tasks, it reduces compute with potential speedup of up to $\times 3$ while provably maintaining high performance (Schuster et al., 2022).

1. Motivation and design objective

Modern autoregressive LLMs generate each token by running all $L$ decoder layers, so the per-token cost grows directly with model depth. CALM is built around the claim that not every token requires the model’s full capacity: “easy” continuations such as function words and punctuation can often be solved with reduced compute, while “hard” tokens such as rare words and content words benefit from deeper computation. The framework therefore treats inference as an adaptive process in which computation is allocated at token granularity rather than fixed uniformly across the sequence (Schuster et al., 2022).

The original formulation emphasizes that early-exit decoding is not just a heuristic for skipping layers. It must also preserve user-specified sequence-level quality constraints, such as requiring that a summary match full-model ROUGE within $\epsilon = 0.02$. This makes CALM a calibrated inference framework rather than only a systems optimization. In the paper’s terminology, the central problem is to reconcile local confidence decisions at each layer with global guarantees over the generated sequence.

2. Local confidence measures and the early-exit rule

At decoding step $t$, CALM assumes the prefix $y_{<t}$ has already been generated and processed through layers $1,\dots,i$ to obtain an intermediate representation $d_t^{(i)}$. It then defines a scalar confidence score $c_{t,i}\in[0,1]$ and exits at the smallest layer whose confidence exceeds a threshold $\tau(t)$. The rule is

$$\text{if}\;\; c_{t,i} \ge \tau(t)\;\; \Longrightarrow\;\; y_t = \arg\max_{v\in V} p(v\mid d_t^{(i)}), \;\;\text{use }i\text{ layers for token }t.$$

If the criterion is not met, decoding continues to layer $L$0 (Schuster et al., 2022).

CALM instantiates three practical confidence measures. The first is softmax-response, defined as the gap between the top two softmax scores:

$L$1

The second is hidden-state saturation, defined as

$L$2

which treats small representation change as a sign that the prediction has stabilized. The third is a learned exit classifier,

$L$3

where $L$4 is a small, frozen linear layer trained to predict whether layer $L$5 would agree with the final layer.

To control exit aggressiveness over the sequence, CALM often uses an exponentially decaying threshold:

$L$6

where $L$7 is the maximum generation length, $L$8 is an initial threshold, and $L$9 is a temperature controlling decay. This schedule reflects the empirical observation that perturbations early in the sequence have more cascading effect than later ones.

3. Sequence-level calibration and statistical guarantees

A defining feature of CALM is that local exit decisions are calibrated against sequence-level consistency constraints. The framework considers two kinds of global criteria. The first is textual consistency, formulated through a textual distance $\epsilon = 0.02$0 between the early-exit output and the full-model output. The second is risk consistency, formulated through a task loss $\epsilon = 0.02$1 relative to references $\epsilon = 0.02$2. In both cases, CALM asks for a threshold choice that satisfies a bound of the form “expected degradation is at most $\epsilon = 0.02$3” (Schuster et al., 2022).

For each candidate threshold $\epsilon = 0.02$4 and calibration prompt $\epsilon = 0.02$5, CALM defines a loss $\epsilon = 0.02$6 and an empirical average $\epsilon = 0.02$7. Using Hoeffding’s inequality, it constructs a p-value for the null hypothesis that threshold $\epsilon = 0.02$8 fails the desired consistency condition:

$\epsilon = 0.02$9

A Family-Wise Error Rate control procedure, such as fixed-sequence testing over the grid $t$0 at level $t$1, then selects the smallest acceptable threshold. Proposition 1 states that, with probability at least $t$2, the selected threshold satisfies the user’s global constraint.

The calibration procedure is a learn-then-test scheme. One first evaluates a grid of candidate thresholds on a calibration set, computes the empirical losses and p-values, and then accepts the most aggressive threshold that passes the statistical test. The paper’s proof sketch notes that for $t$3, early exiting is identical to the full model, so the null can never be true at that point. This anchors the validity of the sequential selection rule.

4. Missing representations, state propagation, and decoding workflow

Early exit creates a structural difficulty for self-attention. At layer $t$4 for token $t$5, the model requires keys and values from all previous tokens at that same layer. If an earlier token exited at a lower layer, higher-layer hidden states are absent. CALM resolves this with a state-copy approximation: if token $t$6 exited at layer $t$7, then for all layers $t$8 it sets

$t$9

This supplies usable hidden states for later attention even when prior tokens did not traverse the full stack (Schuster et al., 2022).

The paper reports that this approximation introduces negligible degradation. In oracle experiments, CALM still matches full-model ROUGE while using on average 1.5 of 8 layers. More generally, an oracle that exits as soon as the layer-$y_{<t}$0 top prediction matches the final layer achieves full-model quality with only 1.53 layers per token on an 8-layer model, implying up to $y_{<t}$1 compute reduction. These oracle numbers are not CALM’s realized operating point, but they quantify the headroom available if confidence estimation improves.

The decoding workflow has two stages. During calibration, the system evaluates candidate thresholds on prompts with references, computes loss statistics, and selects a threshold by the learn-then-test rule. During inference, it generates tokens sequentially, computing layer representations and confidence scores until the threshold is met or the final layer is reached. After each token, it propagates missing states by the copy rule. The result is a token-wise adaptive decoding procedure with explicit calibration to end-to-end quality constraints.

5. Empirical evaluation

The empirical study uses an 8-layer T5 model and repeats each experiment over 50 random 80/20 calibration/test splits with $y_{<t}$2. The three tasks are summarization on CNN/DailyMail measured by ROUGE-L, translation on WMT15 En$y_{<t}$3Fr measured by BLEU, and extractive question answering on SQuAD 1.1 measured by token-F1. Baselines include the full model with $y_{<t}$4 layers for every token, static $y_{<t}$5-layer decoding for $y_{<t}$6, and an oracle local exit upper bound (Schuster et al., 2022).

Task	Full metric	CALM at textual $y_{<t}$7
Summarization	ROUGE-L = 38.32	3.75 layers/token; $y_{<t}$8 end-to-end; 38.30
Translation	BLEU = 27.5	3.19 layers/token; $y_{<t}$9 end-to-end; 27.48
QA	F1 = 82.5	1.65 layers/token; $1,\dots,i$0 end-to-end; 82.1

These results show that CALM can reduce average layers per token substantially while maintaining metrics close to the full model under the selected tolerance. Under risk-consistency with $1,\dots,i$1, average layers per token drop further, for example from 2.62 to 1.73 on summarization, and end-to-end speedups reach up to $1,\dots,i$2. The paper also reports that oracle local exit uses only approximately 1.2–1.5 layers per token with no quality loss, reinforcing the view that confidence estimation, rather than architectural capacity, is the limiting factor in adaptive depth.

6. Limitations, applicability, and relation to later confidence-adaptive scaling

The original paper identifies several limitations. First, the Hoeffding-based p-values and FWER control are statistically rigorous but conservative; the summary notes that larger calibration sets or more refined bounds such as Bentkus can further shrink $1,\dots,i$3. Second, the state-copy approximation can introduce small errors even if the model is robust in practice. Third, branching logic and per-layer confidence computations add minimal overhead, but specialized fused kernels or hardware support for dynamic depth could further increase speedups. At the same time, CALM is presented as orthogonal to distillation, quantization, sparse Mixture-of-Experts, and efficient transformer architectures, so it can be combined with those approaches (Schuster et al., 2022).

The paper also sketches extensions, including learning per-token threshold schedules $1,\dots,i$4 via reinforcement learning under global constraints, designing richer confidence functions based on attention- or gradient-based uncertainty, and applying the method to non-autoregressive or retrieval-augmented generation. These directions keep the core CALM premise intact: adaptive inference should be confidence-driven and externally calibrated, not merely heuristic.

A later line of work revisits confidence-adaptive inference at a different operational level. “Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling” introduces Correctness and Confidence Calibration Reinforcement Learning (C3RL) and Confidence-based Adaptive Test-Time Scaling (CAS). Unlike CALM’s layer-wise local confidence and per-token early exit, CAS uses the model’s own verbalized estimate, has no access to internal logits or entropy, and stops repeated sampling through a Bayesian rule over confidence-weighted answer aggregates. Its experiments report that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times (Yang et al., 2 Jul 2026). This suggests that confidence-adaptive language modeling has developed along two distinct axes: token-wise layer allocation in CALM and sample-budget allocation in later adaptive test-time scaling.