---
title: Two-Pass Adaptive Inference
url: https://www.emergentmind.com/topics/two-pass-adaptive-inference
type: topic
---

# Two-Pass Adaptive Inference

Two-pass adaptive inference denotes a family of inference and training strategies in which computation is explicitly partitioned into two stages, with the first stage producing either a provisional prediction, a reduced candidate set, or a coarse representation, and the second stage applying additional computation only in a more targeted form. Across the literature, the term “adaptive” does not denote a single mechanism. In some systems it means asymmetric compute allocation between a linguistically difficult first stage and a cheaper realization stage; in others it means selective rescoring, coarse-to-fine refinement, hard-negative selection, or rapid reuse of intermediate summaries after local model changes. A common structural motif is that the second stage is constrained by artifacts from the first stage rather than recomputing the entire problem from scratch [2212.08055][1908.10992][1707.03073][2305.10845][1206.3234].

## 1. Conceptual scope and defining properties

In contemporary neural systems, two-pass adaptive inference usually refers to staged computation in which a first pass handles a cheaper or more latency-critical subproblem and a second pass refines, rescales, or reranks the result under a tighter search space. This pattern appears in speech recognition, speech-to-speech translation, incremental natural language understanding, coarse-to-fine video grounding, and large-label softmax training [1908.10992][2212.08055][2305.10845][2605.01512][1707.03073].

The family is heterogeneous. In "Two-Pass End-to-End Speech Recognition" [1908.10992], the first pass is a streaming RNN-T decoder and the second pass is a non-streaming LAS decoder used for rescoring or second-pass beam search. In "UnitY" [2212.08055], the first pass predicts target-language subwords and the second predicts discrete acoustic units, with a deliberately deep-shallow decoder allocation. In "TAPIR" [2305.10845], an LSTM produces immediate incremental outputs and a reviser is invoked only when a controller predicts revision is needed. In "TAPAS" [1707.03073], the first pass samples a candidate label pool from a squashed population distribution and the second adaptively resamples hard negatives for the current batch. In the surveillance-video pipeline of [2605.01512], the first pass scans a full video sparsely at 1 fps and the second pass revisits only a $\pm 3$ s window at 5 fps, with deterministic gates that may revert to the coarse estimate.

A broader historical usage predates deep cascades. "Adaptive Inference on General Graphical Models" [1206.3234] uses adaptive inference to mean maintaining a data structure that supports marginal computation and local updates more rapidly than recomputing exact inference from scratch after modifications to a factor graph. This suggests that the unifying idea is not any specific architecture, but the reuse of intermediate structure so that expensive computation is restricted to the subset of work that remains unresolved.

A concise way to distinguish two-pass adaptive inference from ordinary cascades is that the first pass does not merely precede the second chronologically; it changes the computational problem faced by the second pass. That change may occur through hypothesis pruning, representation transformation, localized temporal windows, or precomputed cluster functions [1908.10992][2212.08055][2605.01512][1206.3234].

## 2. Canonical two-pass patterns across domains

Several recurring design patterns can be identified in the literature.

| Pattern | First pass | Second pass / adaptive element |
|---|---|---|
| Streaming then rescoring | RNN-T streaming hypothesis | LAS rescoring on N-best or lattice [1908.10992] |
| Linguistic then acoustic | Subword text generation | Discrete unit prediction conditioned on text [2212.08055] |
| Incremental then revisable | LSTM online labeling | Transformer or Linear Transformer revision when triggered [2305.10845] |
| Random then adaptive sampling | Candidate set $S'$ from fixed $Q$ | Hard-negative set $S \subset S'$ for current batch [1707.03073] |
| Coarse then localized grounding | Full-video scan at 1 fps | Refinement in a $\pm 3$ s window with gates [2605.01512] |

Despite surface differences, these systems share a staged reduction of search complexity. In speech recognition, the second pass is applied to a compact candidate set rather than performing unrestricted sequence generation; rescoring a lattice is reported as more efficient than rescoring an N-best list because shared prefixes are processed only once [1908.10992]. In UnitY, the expensive beam search is applied to the shorter subword sequence, while the second-pass unit decoder is shallow and often decoded with beam size $1$ [2212.08055]. In TAPAS, adaptive search over the entire label space is avoided by first drawing a manageable candidate pool $S'$ and only then selecting the top-$n$ hard negatives for the current batch [1707.03073]. In the CCTV grounding system, the second pass is spatially and temporally localized around the coarse anchor produced by the first pass [2605.01512].

Not all systems are adaptive in the same sense. The residual-CNN method of "Leveraging Stochastic Depth Training for Adaptive Inference" [2505.17626] is explicitly two-phase rather than two-pass per sample: an offline design-time phase trains with Stochastic Depth and selects near Pareto-optimal skip configurations, and an online runtime phase chooses among those configurations based on workload and a minimum acceptable accuracy. The paper states that inference itself is single-pass and that the method is not a two-pass inference for each sample [2505.17626]. This distinction is central to the taxonomy: some adaptive systems stage computation within each example, whereas others stage configuration selection around a single execution path.

## 3. Speech and sequence-processing architectures

Speech has provided some of the clearest instantiations of two-pass adaptive inference. The speech-recognition architecture in [1908.10992] shares a single acoustic encoder between a streaming RNN-T first pass and a non-streaming LAS second pass. The encoder is an 8-layer LSTM with 2,048 hidden units and 640-dimensional projection per layer, with a time-reduction layer after the second LSTM. The combined training objective is
$$
L_{\mathrm{combined}(x, y^*) = \lambda L_{\mathrm{RNNT}(x, y^*) + (1-\lambda)L_{\mathrm{LAS}(x, y^*)
$$
with $\lambda$ set to $0.5$, followed by a multi-step strategy in which RNN-T is trained first, the encoder is frozen while LAS is trained, and then encoder and both decoders are deep-finetuned jointly [1908.10992]. For rescoring, LAS is trained with MWER using hypothesis sets produced by the RNN-T first pass, aligning training with the intended inference mode. Reported results show that LAS rescoring after MWER yields **5.7 SU / 3.5 LU / 7.0 Contacts**, compared with **6.9 SU / 4.5 LU** for the RNN-T baseline, corresponding to a **17% relative WER reduction on SU** and **22% relative WER reduction on LU** [1908.10992].

UnitY extends the two-pass principle from ASR to direct speech-to-speech translation [2212.08055]. Given source speech $X$, the speech encoder produces hidden states
$$
H = \text{SpeechEnc}(X).
$$
The first-pass text decoder $TDec$ generates the target-language subword sequence
$$
Y = (y_1,\dots,y_M)
$$
with an S2TT objective
$$
\mathcal{L}_{\text{S2TT} = -\frac{1}{M}\sum_{i=1}^{M}\log p(y_i \mid X, y_{<i}).
$$
The continuous states of the first pass are then transformed by a text-to-unit encoder,
$$
Z = T2UEnc(\cdot),
$$
and the second-pass unit decoder $UDec$ generates a discrete acoustic unit sequence
$$
U = (u_1,\dots,u_L)
$$
with
$$
\mathcal{L}_{\text{S2UT} = -\frac{1}{L}\sum_{i=1}^{L}\log p(u_i \mid X, Y, u_{<i}).
$$
The second pass does not attend directly to the speech encoder output $H$; it is conditioned on the first-pass linguistic representation so that text and units remain synchronized and the second pass does not become a repair mechanism for text errors [2212.08055]. The model’s total loss is
$$
\mathcal{L} = \mathcal{L}_{\text{S2UT} + \lambda\,\mathcal{L}_{\text{S2TT}.
$$
Architecturally, UnitY differs from Translatotron2 in five major ways: subwords instead of phonemes in the first pass, discrete units instead of spectrograms in the second pass, Transformer decoders instead of LSTMs, a T2U encoder between the decoders, and more capacity in the first pass than the second. The best configuration in the multi-domain En↔Es setting is often a **12-layer text decoder + 2-layer unit decoder** with t-mBART initialization [2212.08055].

TAPIR provides a sequence-labelling analogue of the same staged principle [2305.10845]. Its first pass is an LSTM incremental processor, and its optional second pass is a bidirectional Transformer encoder or Linear Transformer reviser. A recurrent controller decides between **WRITE** and **REVISE**:
$$
a_t = \begin{cases} \text{REVISE}, & \text{if } \sigma(\theta \kappa_t + b_k) \ge \tau \\
\text{WRITE}, & \text{otherwise}
\end{cases}
$$
with $\tau \in [0,1]$. The threshold yields a continuous speed-accuracy trade-off: $\tau = 1$ never revises and behaves like an RNN, whereas $\tau = 0$ always revises and behaves like a restart-incremental Transformer [2305.10845]. TAPIR is trained in two stages: first the reviser alone with cross-entropy, then the incremental processor and controller jointly with
$$
\mathcal{L} = \text{CE}(y^{gold}, y) + \text{BCE}(a^{LT}, a).
$$
The reported average sequence throughput is **1.032 sentences/sec** for the restart-incremental Transformer, **4.594 sentences/sec** for TAPIR-Trf, and **5.044 sentences/sec** for TAPIR-LT, for an average speed-up of about **4.5×** [2305.10845].

## 4. Search, gating, and compute allocation mechanisms

Two-pass adaptive systems are often defined less by their backbone models than by how they allocate search and trust between passes. UnitY makes this explicit. Its decoding procedure is: encode speech, beam search over the text decoder, retrieve cached hidden states for the selected $Y$, transform them with the T2U encoder, beam search over the unit decoder, and convert units to waveform with a unit-based vocoder. The hidden states of the winning text hypothesis are reused rather than recomputed [2212.08055]. Empirically, increasing first-pass beam width up to $10$ helps, while reducing second-pass beam width to $1$ barely hurts. The main setting uses beam width **10** for the first pass and **1** for the second. This supports the paper’s claim that the first pass is harder and deserves more search [2212.08055].

In the RNN-T/LAS system, the adaptive element lies in candidate management rather than architectural asymmetry alone. The first-pass beam can be fixed or adaptive, adaptive pruning decides which hypotheses survive based on thresholding, and rescoring can operate on an N-best list or a lattice. The paper estimates latency using
$$
\mathrm{latency} = K \cdot H \cdot N \cdot M_{\mathrm{decoder}
$$
with $K = 10\ \mathrm{GB/s}$, and for adaptive beam decoding replaces $H \cdot N$ by the number of lattice arcs. Reported latencies are **369.6 ms** for fixed-beam N-best rescoring and **247.5 ms** for adaptive-beam lattice rescoring; with further lattice pruning and limiting rescored arcs, the system fits within the **200 ms budget** for interactive latency [1908.10992].

The CCTV grounding system of [2605.01512] makes the adaptive logic deterministic and externally inspectable. After the coarse pass returns $(t_1,\tilde x_1,\tilde y_1,c_1)$, the refinement window is
$$
W = [\max(0,t_1-\Delta),\min(D,t_1+\Delta)]
$$
with $\Delta = 3$ s. A temporal fallback gate with boundary tolerance $\tau = 0.3$ s chooses
$$
t^* = \begin{cases} t_1 & \text{if } t_2 < 0 \text{ or } |t_2-W_{\min}|<\tau \text{ or } |t_2-W_{\max}|<\tau,\\
t_2 & \text{otherwise.}
\end{cases}
$$
A spatial merge gate with margin $m=10$ accepts refined coordinates only if they are not near the edges of the native $[0,1000]^2$ grid; otherwise it reverts to the coarse location [2605.01512]. This is a different style of adaptivity from learned gating: the refinement pass always runs on the localized window, but the system decides whether to trust it.

The stochastic-depth method in [2505.17626] offers a contrasting notion of control. It inserts simple, weightless skip-or-not gates before skippable residual blocks, defines the configuration as an externally supplied binary array
$$
S = [s_1,s_2,..,s_{B_s}], \qquad s_i \in \{0,1\},
$$
and selects operating points via sensitivity analysis plus a Pareto front. Runtime adaptation then changes the skip level based on whether the device is busy or idle and whether candidate configurations satisfy a minimum acceptable accuracy. Because configuration is controlled externally rather than by an input-dependent gate network, the paper emphasizes **fully user-controllable** and **time-predictable** execution [2505.17626]. A plausible implication is that two-pass adaptive inference can be situated on a spectrum ranging from learned, per-example escalation to explicit, systems-level scheduling.

## 5. Training regimes, regularization, and approximation objectives

Two-pass adaptive inference commonly requires training strategies that match the staged inference procedure. In speech recognition, direct training from scratch of the combined RNN-T and LAS objective was found unstable, which led to the multi-step train-freeze-finetune schedule and later MWER training using first-pass hypotheses [1908.10992]. This is an instance of pass-aware optimization: the second pass is not merely trained as a standalone model, but as a model operating over the support induced by the first pass.

UnitY likewise relies on training interventions tailored to the staged architecture [2212.08055]. The paper applies **R-Drop** to discrete-symbol prediction tasks, duplicating inputs and adding KL penalties for both the unit and text decoders. It reports that R-Drop improves the first-pass text decoder and thereby improves final S2ST quality, while an auxiliary CTC loss on the unit decoder is not helpful because the first-pass decoder already makes monotonic alignment easier. Pretraining is especially important for the first-pass decoder. The system uses self-supervised speech encoder pretraining such as **wav2vec 2.0** and **w2v-BERT**, and text-only unlabeled data through **text-based mBART (t-mBART)** to initialize the first-pass text decoder. The paper finds that t-mBART pretraining is very effective for the first-pass decoder, pretraining the unit decoder with u-mBART does not help as much, and the best strategy is to pretrain the deep text decoder and keep the unit decoder shallow [2212.08055].

TAPAS is noteworthy because its second pass deliberately introduces bias into the gradient approximation [1707.03073]. For a mini-batch $B=\{(x_i,y_i)\}$, after first-pass pre-sampling of
$$
S' \subset [V], \qquad |S'| = \min(rn, |V|),
$$
the adaptive second pass chooses
$$
S = \arg\max^n_{y\in S'} \sum_{i\in B} \exp(\phi(x_i)\cdot \psi(y)/\tau).
$$
The method then computes sampled softmax gradients using only $S$, intentionally not correcting for the bias introduced by adaptive resampling because the goal is to emphasize hard negatives and improve rank loss rather than exact softmax optimization [1707.03073]. The reported real-data overhead of increasing the pre-sample factor from $r=1$ to $r=8$ is about **10%** at fixed $n$, while MAP@20 improves from **0.050 to 0.068** for $n=1000$ and from **0.067 to 0.075** for $n=8000$ [1707.03073]. This illustrates that in two-pass adaptive methods, the objective optimized by the second stage may differ from the nominal full-model objective.

A related optimization issue appears in low-resource NLP adaptive inference [2306.02307]. The paper compares Early-Exit and Multi-Model methods and attributes the underperformance of Early-Exit in low-resource settings to conflicting gradients from multiple classifiers sharing lower layers. Individual Multi-Model classifiers outperform Early-Exit classifiers by **2.3% on average**, and the proposed SWEET method improves Early-Exit individual classifiers by **1.1% on average** by assigning each classifier its own subset of parameters [2306.02307]. Although SWEET is not a classic two-pass method, it indicates that staged inference can fail when training induces destructive coupling across stages. This suggests that parameter ownership and optimization geometry are central design variables even when the runtime adaptation mechanism appears simple.

## 6. Efficiency, theoretical interpretation, and limits

The literature consistently frames two-pass adaptive inference as a speed-quality or resource-quality trade-off, but the quantitative form of that trade-off varies by task. UnitY reports **+4.2** ASR-BLEU over S2UT on Fisher, **+3.7** on CVSS-C, and **+2.5** on multi-domain En↔Es, together with **2.83× faster decoding** than S2UT and **2.51× faster decoding** than S2SpecT2. It also reports **1.65× fewer FLOPs** than S2SpecT2 and **3.19× fewer FLOPs** than S2UT [2212.08055]. The paper’s key conclusion is that allocating compute to the text stage is more beneficial than spending it on the acoustic stage.

The traffic-grounding pipeline of [2605.01512] shows a coarse-to-fine analogue of the same principle. On **ACCIDENT@CVPR 2026** with **2,027 real CCTV test videos**, the full pipeline reaches
$$
\mathrm{ACC}^S = 0.539 \quad [0.525, 0.553]
$$
with $T = 0.497$, $S = 0.538$, and $C = 0.591$. This is reported as **+0.127** over the benchmark paper’s best-of-baselines oracle (**0.412**), **+0.143** over the strongest single-VLM baseline (**Molmo-7B, 0.396**), and **+0.250** over the naive baseline (**0.289**) [2605.01512]. The VLM path uses up to **three API calls per video**, with Pass 1 costing $\sim \$4$, Pass 2 $\sim \$6$, and Gemini typing $\sim \$10$, for a total of $\sim \$20$ over 2,027 videos, or about **\$0.01 per video** [2605.01512]. Here adaptation is not primarily about minimizing FLOPs inside one model, but about spending high-resolution inference only on the temporal neighborhood proposed by the coarse stage.

A theoretical account of such staged systems is developed in "Adaptive Inference: Theoretical Limits and Unexplored Opportunities" [2402.04359]. The paper models adaptive inference as a state machine over models or configurations $\{S_i\}_{i=1}^N$ with ordered resource costs $R_i$ and accuracies $A_i$. In the two-state case, the first and second passes can be interpreted as $S_1$ and $S_2$. Under the conservative $\alpha=1$ regime, the oracle’s expected resource use becomes
$$
R_{oracle}=R_1+(A_2-A_1)(R_2-R_1), \qquad A_{oracle}=A_2.
$$
The exact framework emphasizes that attainable gains depend not only on the accuracies of the two passes, but on the overlap of their error sets, encoded by $\alpha_i$ [2402.04359]. The paper reports potential **10–100x efficiency improvements** without performance penalties across Computer Vision and Natural Language Processing tasks, including conservative gains of about **63.43x** for ImageNet / EfficientNet, **43.80x** for ImageNet / ViT, and **9.56x** for HellaSwag / Pythia [2402.04359]. These are oracle opportunity estimates rather than realized gains of any specific two-pass method.

The theory clarifies several misconceptions. First, adaptive inference is not guaranteed to improve accuracy; gains require favorable error complementarity, and if smaller and larger models fail on largely the same inputs, the opportunity is limited [2402.04359]. Second, two-pass structure alone does not ensure per-sample adaptivity: [2505.17626] is adaptive through offline/online configuration selection but single-pass at inference time. Third, “adaptive inference” can also refer to repeated exact inference under local model changes rather than dynamic compute allocation. In the graphical-model setting, exact marginal queries and updates can be supported in expected
$$
O(\alpha \mu_T(G)\log n)
$$
time, where $\alpha = d^{k+1}$ and $\mu_T(G)$ measures the spanning tree’s cut complexity [1206.3234]. This older formulation shows that adaptive inference originally concerned incremental reuse of inference structure, a theme that remains present in modern two-pass systems whenever coarse hypotheses, cached hidden states, or localized windows prevent global recomputation.

Overall, two-pass adaptive inference is best understood as a general staged-computation paradigm. Its mature forms combine asymmetric model capacity, pass-aware training, search-space restriction, and explicit policies for deciding how much trust to place in refinement. The strongest empirical results arise when the first pass is cheap but informative, the second pass is constrained by the first rather than unconstrained, and the two stages have sufficiently complementary failure modes to justify escalation [2212.08055][1908.10992][2305.10845][2605.01512][2402.04359].

Source: https://www.emergentmind.com/topics/two-pass-adaptive-inference