Training-Inference Mismatch (TIM)

• Training-Inference Mismatch (TIM) refers to the inconsistency between the conditions, distributions, objectives, or parameters of a model during training and those during inference, leading to performance degradation in AI applications.
• TIMs are of different types: distribution mismatch, objective mismatch, parameter-state mismatch, context mismatch, input-condition mismatch, numerical and systems mismatch, and trigger mismatch, each affecting the model in distinct ways across different architectures and tasks.
• Researchers address TIM using various methods, such as task-aware regularization, architectural alignment, numerical and systems alignment, and gradient and objective correction, each aimed at minimizing the discrepancies between training and inference.

Training–Inference Mismatch (TIM) is a discrepancy between the conditions, distributions, objectives, parameter states, numerical computations, or contextual information available during model training and those encountered during inference. The term encompasses temporal distribution drift, incompatible adapted and frozen model components, streaming-context differences, masked-versus-unmasked inputs, autoregressive exposure bias, numerical inconsistencies between rollout and training engines, quantized-rollout discrepancies, and inference configurations absent from training. Although these phenomena arise in different architectures and tasks, they share a common structure: the model is optimized under one computational or statistical regime and deployed under another.

1. Definition, scope, and conceptual structure

TIM is broader than an ordinary train/test split. In conventional supervised learning, the training and test distributions may differ, but the deployed model usually has the same architecture, objective, parameter state, and input-processing procedure at both stages. TIM arises when one or more of these conditions change between training and inference.

The mismatch may concern several dimensions:

• Distribution mismatch: inference data are drawn from a population whose distribution differs from the training population, as in gradual temporal drift. A model may be trained on labeled historical snapshots $\mathcal{D}^1,\ldots,\mathcal{D}^T$ and deployed on a future distribution $P(\mathbf{x}\mid t_{T+1})$ (Nasery et al., 2021).
• Objective mismatch: training optimizes one loss, while inference-time adaptation or deployment behavior is governed by another objective. Test-time training, for example, updates a feature extractor using an auxiliary loss while the main classifier remains frozen (Zhang et al., 2022).
• Parameter-state mismatch: some model components are updated while others retain their training-time parameters. The resulting hybrid model may not have been optimized as a coherent system (Zhang et al., 2022).
• Context mismatch: the model receives contexts during inference that were absent during training, including partially filled streaming segments, self-generated autoregressive prefixes, or diffusion denoising configurations outside the training support (Raffel et al., 2023, Cen et al., 2024, Jain, 28 Jun 2026).
• Input-condition mismatch: masked language-model pre-training may use masked representations although downstream inference uses unmasked representations (Yadav et al., 2024).
• Numerical and systems mismatch: rollout and training engines may assign different probabilities to the same sequence despite identical nominal parameters, because of different kernels, reduction orders, precisions, tensor-parallel configurations, or quantization schemes (Qi et al., 30 Oct 2025, Zhong et al., 14 May 2026).
• Trigger mismatch: in backdoor attacks, the trigger intensity used during poisoning may differ from the intensity used during inference (Lin et al., 15 Mar 2025).

A useful formal abstraction is to distinguish the training-side predictor or policy $p$ from the inference-side predictor or policy $q$. TIM exists whenever

$q \ne p$

under the relevant input, state, or configuration. In LLM reinforcement learning, for example, the sampler policy $\mu$ and learner policy $\pi$ may satisfy

$$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$

even when both engines load the same parameters (Qi et al., 30 Oct 2025). In diffusion LLMs, the analogous discrepancy is between the training distribution $\pi$ over prefix-window configurations and the inference distribution $\Pi_{\mathrm{inf}}$ (Jain, 28 Jun 2026).

TIM is therefore not a single failure mechanism or a single metric. It is a family of training-deployment inconsistencies whose effect depends on how the mismatch propagates through prediction, adaptation, autoregressive feedback, optimization, or deployment.

2. Principal manifestations across learning systems

Temporal extrapolation

In temporal generalization, the model observes labeled examples at times $P(\mathbf{x}\mid t_{T+1})$0 and is evaluated at a future time $P(\mathbf{x}\mid t_{T+1})$1. The future distribution is assumed to be relatively close to the latest training period, but it need not be stationary: $P(\mathbf{x}\mid t_{T+1})$2, $P(\mathbf{x}\mid t_{T+1})$3, or both may change over time (Nasery et al., 2021).

A sufficiently expressive time-aware network can fit each historical snapshot while learning an implausible or oscillatory function between and beyond the observed timestamps. Ordinary empirical risk minimization constrains the model at observed times but does not directly constrain its temporal extrapolation. Gradient Interpolation (GI) addresses this form of TIM by supervising both the predictor value and its temporal derivative. Its objective uses a first-order approximation,

$P(\mathbf{x}\mid t_{T+1})$4

with adversarially selected $P(\mathbf{x}\mid t_{T+1})$5. GI permits task-consistent temporal change rather than forcing the predictor to become time-invariant (Nasery et al., 2021).

Adapted representations paired with static heads

Test-time training introduces an auxiliary optimization stage between source training and main-task inference:

$P(\mathbf{x}\mid t_{T+1})$6

Typically, the shared feature extractor $P(\mathbf{x}\mid t_{T+1})$7 is updated using an auxiliary loss, while the main classifier $P(\mathbf{x}\mid t_{T+1})$8 remains frozen. Inference therefore uses

$P(\mathbf{x}\mid t_{T+1})$9

rather than the jointly trained source model. If the feature geometry moves too far, the static classifier is no longer calibrated to the representation. Mixup for Test-Time Training (MixTTT) reduces this model mismatch by interpolating test samples with source training images, thereby regularizing the magnitude and direction of feature adaptation (Zhang et al., 2022).

Streaming-context mismatch

Segment-based simultaneous speech translation processes complete segments during training but partially filled segments during streaming inference. An Augmented Memory Transformer may be trained with a fixed configuration such as left context, center context, and right context, but inference can expose it to incomplete center or right contexts and missing left context at sentence onset (Raffel et al., 2023).

Shiftable Context reallocates already available tokens among these regions so that inference-time segments retain the training segment length and, where possible, the training center-context length. It does not use unavailable future input. Instead, missing right context is replaced with additional left context, partial center context is completed by shifting tokens from the left, and missing initial left context is replaced with available right context (Raffel et al., 2023).

Masked-versus-unmasked representation mismatch

HuBERT pre-training masks approximately half of the convolutional feature vectors before the Transformer predicts discrete pseudo-labels. During downstream representation extraction or ASR inference, no mask tokens are inserted. The Transformer is therefore trained primarily on $p$0 but used on $p$1 (Yadav et al., 2024).

MS-HuBERT’s Swap mechanism processes masked and unmasked copies together and exchanges their representations at masked positions after every Transformer layer. Swap is architectural rather than an additional similarity loss. It exposes the network to both input views during pre-training and reduces dependence on the artificial mask token (Yadav et al., 2024).

Autoregressive exposure bias

Teacher-forced autoregressive models receive ground-truth prefixes during training but their own generated prefixes during inference. For a reference continuation $p$2 and generated continuation $p$3,

$p$4

whereas

$p$5

A small early prediction error changes the prefix and thereby changes all subsequent conditional distributions. This is especially consequential in long sequences, including speech-token sequences and reasoning traces (Cen et al., 2024, Zhang et al., 21 Sep 2025).

Batch-Scheduled Sampling exposes LLMs to offline contexts containing self-generated tokens while retaining reference targets. Reference-Answer-based Correction trains the model to transform its own generated response into a reference-informed correction. Prompt-guided hybrid training applies the analogous principle to LM-based TTS by combining protected ground-truth prefixes with self-generated speech-token prefixes and using EOS prediction to control free-running exposure (Cen et al., 2024, Zhang et al., 21 Sep 2025).

Numerical policy mismatch in LLM reinforcement learning

Modern LLM-RL systems commonly separate rollout generation from policy optimization. vLLM or another inference stack generates responses, while FSDP, DeepSpeed, Megatron, or another training stack recomputes probabilities and applies gradients. Different floating-point reduction trees, kernels, tensor-parallel sizes, batch layouts, attention implementations, MoE routing, or quantization schemes can cause the two systems to assign different probabilities to the same token (Zhang et al., 21 Nov 2025, Zhong et al., 14 May 2026).

This makes nominally on-policy RL effectively off-policy. If responses are sampled from $p$6 but gradients use $p$7, the practical gradient is

$p$8

rather than the intended expectation under $p$9 (Qi et al., 30 Oct 2025).

3. Mechanisms of error propagation

TIM is often amplified rather than remaining a local discrepancy.

In autoregressive generation, token probabilities are multiplied across a sequence:

$q$0

Consequently, small token-level log-probability differences accumulate over sequence length. A changed token also changes the subsequent hidden state, so later probabilities are evaluated on a different prefix. In RL, the resulting sequence-probability ratio can become extremely large or small, producing high-variance importance weights and biased or unstable updates (Qi et al., 30 Oct 2025).

A related formal analysis bounds the gradient discrepancy under a per-state total-variation mismatch $q$1 by a term proportional to $q$2:

$q$3

where $q$4 bounds the token-level score function (Zhang et al., 2 Feb 2026). The quadratic dependence reflects both the accumulation of state-distribution divergence and the summation of gradients over the response.

In numerical RL systems, rare probability disagreements can be more consequential than average disagreement. A single token may have a substantially different log probability under the learner and rollout engines, potentially changing the top-ranked token. PPO clipping does not eliminate this problem because it constrains movement relative to a denominator policy but does not ensure that the denominator is the policy that generated the data (Zhong et al., 14 May 2026).

Persistent mismatch can also form a feedback loop. A biased update changes the trainer, the updated trainer is synchronized back to the sampler, and the sampler then generates data under a distribution containing the previous bias. Score Centering Stabilizes Off-policy Reinforcement Learning identifies the expected-score term

$q$5

as a drift component. Subtracting it from the score removes the sampler-induced mean update at each prefix (Marek et al., 17 Sep 2026).

Optimization can further amplify TIM. As response lengths increase, gradient noise and mismatch may rise together. Large learning-rate updates can move the model into regions where numerical differences have greater behavioral impact. A response-length-triggered scheduler therefore reduces the learning rate when average response length exhibits a surge, repeatedly halving it until a specified floor (Zhang et al., 2 Feb 2026).

4. Mitigation methodologies

TIM mitigation methods operate at different levels: the data distribution, architecture, loss, numerical implementation, optimization schedule, or deployment policy.

Training-distribution alignment

The simplest strategy is to expose the model during training to the configurations it will encounter at inference.

For diffusion LLMs, Adaptive Block Diffusion samples prefix-window configurations $q$6 from a training distribution $q$7. Its theoretical alignment condition is

$q$8

meaning that every inference configuration with nonzero probability is covered by the training distribution (Jain, 28 Jun 2026). This prevents fixed-block specialists from being unconstrained at off-grid block sizes, although configuration density still affects performance.

For autoregressive models, BASH and prompt-guided hybrid training introduce self-generated prefixes during training. Both preserve reference supervision while modifying the conditioning distribution (Cen et al., 2024, Zhang et al., 21 Sep 2025).

Task-aware regularization

GI regularizes temporal evolution by evaluating a gradient-interpolated prediction rather than merely penalizing the temporal derivative. Its strength $q$9 controls the contribution of the interpolated prediction, while $q \ne p$0 controls the temporal neighborhood (Nasery et al., 2021).

MixTTT applies input mixup before the auxiliary test-time objective. Its first-order expansion introduces an implicit regularization term proportional to

$q \ne p$1

This constrains the sensitivity of the auxiliary loss and indirectly limits representation drift while retaining the original test-time adaptation objective (Zhang et al., 2022).

Architectural alignment

Swap in MS-HuBERT couples masked and unmasked representation streams throughout the Transformer (Yadav et al., 2024). Shiftable Context changes the allocation of streaming context tokens so that the encoder sees configurations closer to those used during training (Raffel et al., 2023).

These methods modify the representation or context construction rather than relying only on a penalty. Their common principle is to make inference-relevant states part of the training computation.

Numerical and systems alignment

Several methods target the numerical source of TIM.

• FP16 alignment: using FP16 uniformly in rollout and training can reduce local rounding discrepancies relative to BF16, whose seven fraction bits provide coarser local resolution (Qi et al., 30 Oct 2025).
• Tree-Based Invariant Kernels: TBIK aligns local GEMM reductions and inter-GPU reductions through a shared binary reduction tree. Under specified conditions, it produces bit-wise identical results across tensor-parallel sizes (Zhang et al., 21 Nov 2025).
• VeXact: VeXact uses shared model implementations, deterministic batch-invariant kernels, fixed tiling, deterministic RMSNorm and MoE operations, and attention with KV splitting disabled to provide a zero-mismatch diagnostic baseline (Zhong et al., 14 May 2026).
• QaRL: QaRL performs actual low-bit forward computation in the learner to match quantized rollout computation, maintains BF16 master weights, uses a straight-through estimator, and synchronizes low-bit tensors to the rollout engine (Gu et al., 9 Apr 2026).

These approaches differ in ambition. Some seek bit-wise or near-bit-wise equality, while others reduce the dominant precision discrepancy and tolerate residual kernel differences.

Gradient and objective correction

Importance sampling corrects for the difference between sampler and learner distributions, but long sequence ratios can have high variance. Truncation and masking reduce variance at the cost of bias (Qi et al., 30 Oct 2025, Zhong et al., 14 May 2026).

Score Centering adds an estimator correction rather than multiplying by a potentially heavy-tailed ratio. Its corrected score is

$q \ne p$2

The correction removes the drift term exactly under the sampler distribution while retaining a covariance-based learning signal (Marek et al., 17 Sep 2026).

Trust-Band Policy Optimization (TBPO) treats a response as a sequence-level action, uses geometric-mean sequence probabilities, clips mismatch weights, and applies dual clipping to negative-advantage responses. Its purpose is to suppress entire error-token trajectories rather than clipping only individual tokens (Gu et al., 9 Apr 2026).

Deployment-aware policy selection

MIPU distinguishes improving the training policy from improving the inference policy. It first constructs a sampler-referenced candidate using truncated mismatch correction, then synchronizes the candidate and evaluates an inference-side gap proxy. Candidates whose proxy is sufficiently negative are rejected and both model and optimizer state are rolled back (Liang et al., 28 Jun 2026).

This approach addresses a distinct problem from numerical alignment: even if a training-side update is stable, it may not improve the deployed inference policy. MIPU therefore makes candidate acceptance deployment-aware.

5. Empirical evidence and evaluation patterns

The empirical literature shows that TIM is measurable and can materially affect performance, but the appropriate metric depends on the manifestation.

In temporal generalization, GI was evaluated on nine datasets, including Rotated 2-Moons, Rotated MNIST, Online News Popularity, Electrical Demand, Shuttle, Reuters, House prices, M5-Hobbies, and M5-Household. GI achieved test errors of $q \ne p$3 on Rotated 2-Moons, $q \ne p$4 on Rotated MNIST, and $q \ne p$5 MAE on M5-House under the reported protocol (Nasery et al., 2021). Its gains were strongest in settings vulnerable to temporal overfitting and did not require future labeled or unlabeled samples.

For streaming speech translation, Shiftable Context improved average BLEU over wait-$q \ne p$6, wait-$q \ne p$7, wait-$q \ne p$8, and wait-$q \ne p$9 baselines by $\mu$0 for English–German, $\mu$1 for English–French, and approximately $\mu$2 for English–Spanish on tst-COMMON, with small computation-aware latency increases (Raffel et al., 2023). The largest isolated ablation gain came from shiftable center context, but left- and right-context shifting addressed distinct mismatch sources.

For HuBERT representation learning, MS-HuBERT with Swap obtained one-hour fine-tuning WERs of $\mu$3 on test-clean and $\mu$4 on test-other, while removing Swap produced $\mu$5 and $\mu$6, respectively (Yadav et al., 2024). The results support a contribution from Swap but do not fully disentangle it from Multicluster masked prediction.

For autoregressive LLM fine-tuning, BASH and RAC improved summarization, general QA, and mathematical QA relative to SFT in the reported experiments. BASH achieved $\mu$7 on GSM8K and $\mu$8 on MATH, while RAC achieved $\mu$9 and $\pi$0 (Cen et al., 2024). RAC was strongest in the reported general-QA evaluation, whereas BASH was strongest on the mathematical benchmarks.

For LM-based TTS, the prompt-guided hybrid method improved CosyVoice2 LibriSpeech WER from $\pi$1 to $\pi$2 and improved SIM from $\pi$3 to $\pi$47.25$\pi$54.98$\pi$64.21$$for the complete method (<a href="/papers/2509.17021" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Zhang et al., 21 Sep 2025</a>).</p> <p>For backdoor attacks, mismatched training and inference trigger intensities sometimes improved attack robustness and stealth. In one CIFAR-10 experiment, mixed training opacities raised worst-case ASR from$$\pi$7 for the single-intensity baseline to $\pi$8 for the best reported mixture (Lin et al., 15 Mar 2025). A training/inference opacity pair of $\pi$9 reduced Scale-Up defense AUC from $\mu(y_t\mid x,y_{$191.62\%$.

In LLM-RL systems, VeXact experiments isolated TIM from ordinary algorithmic off-policy effects. In a Qwen3-30B-A3B MoE REINFORCE experiment, the vLLM run’s training reward decreased from $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$2 to $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$3 and validation reward from $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$4 to $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$5, whereas VeXact reached $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$6 training reward and $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$7 validation reward (Zhong et al., 14 May 2026). Other studies reported that FP16 reduced sequence-level mismatch by approximately $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$8 relative to BF16 in controlled offline analysis (Qi et al., 30 Oct 2025), while TBIK achieved zero reported probability divergence across tested tensor-parallel and batch-size configurations (Zhang et al., 21 Nov 2025).

The newer optimization-oriented results emphasize that mitigation efficacy depends on mismatch severity and duration. A response-length-triggered scheduler stabilized Qwen3-4B and Qwen3-8B RL runs when decay began after response-length surges (Zhang et al., 2 Feb 2026). QaRL plus TBPO improved Qwen3-30B-A3B quantized-rollout performance from $$\mu(y_t\mid x,y_{<t},\theta)\ne\pi(y_t\mid x,y_{<t},\theta),$$9 to $\pi$0, approaching the BF16 baseline of $\pi$1 while retaining low-bit rollout benefits (Gu et al., 9 Apr 2026). MIPU improved average benchmark scores from $\pi$2 to $\pi$3 for Qwen3-4B and from $\pi$4 to $\pi$5 for Qwen3-1.7B under severe FP8 rollout mismatch (Liang et al., 28 Jun 2026). Score centering remained stable under severe quantization conditions in experiments ranging from $\pi$6B to $\pi$7B parameters and composed effectively with importance sampling under sampler staleness (Marek et al., 17 Sep 2026).

6. Limitations, distinctions, and open problems

TIM mitigation is not universal. Methods are generally tailored to specific mismatch structures.

Temporal GI assumes gradual drift and nearby future deployment. Abrupt regime changes, far-future extrapolation, inappropriate $\pi$8 or $\pi$9, and a large number of temporal domains can reduce effectiveness (Nasery et al., 2021). Shiftable Context assumes segment-based architectures with identifiable context regions and may require adaptation for architectures without right context (Raffel et al., 2023).

Input mixup can generate semantically ambiguous examples and requires access to source data at test time (Zhang et al., 2022). BASH and RAC incur offline generation and storage costs, and RAC can produce local token corrections without repairing deeper reasoning errors (Cen et al., 2024). Prompt-guided TTS training has incompletely specified schedules and uses EOS as a proxy that does not fully characterize acoustic or prosodic quality (Zhang et al., 21 Sep 2025).

Numerical alignment can be expensive. TBIK introduces substantial overhead relative to conventional GEMM and collective operations, and its formal guarantees depend on fixed arithmetic, compatible tiling, specified tensor-parallel sizes, and controlled execution (Zhang et al., 21 Nov 2025). FP16 has narrower dynamic range than BF16 and may require loss scaling or selective higher precision (Qi et al., 30 Oct 2025). QaRL reduces but does not eliminate sampler-learner differences and adds training-side quantization overhead (Gu et al., 9 Apr 2026).

Post-hoc corrections have their own trade-offs. Importance sampling can become high variance on long responses; clipping and masking introduce bias; sequence-level rejection discards potentially useful data (Zhong et al., 14 May 2026). Score Centering removes the drift term but does not convert a sampler-distribution covariance into an exact trainer-distribution policy gradient under substantial staleness (Marek et al., 17 Sep 2026). MIPU’s inference-side acceptance proxy is not a formal monotonic-improvement guarantee, requires additional inference evaluation, and is sensitive to its tolerance parameter (Liang et al., 28 Jun 2026).

The theoretical status of several claims is also limited. ABD’s support-based guarantee assumes global population-risk minimization and does not provide finite-sample or practical optimization guarantees (Jain, 28 Jun 2026). The response-length analysis gives an $\Pi_{\mathrm{inf}}$0 upper bound under stated assumptions rather than an exact scaling law for every RL system (Zhang et al., 2 Feb 2026). The causal contribution of individual components is not always isolated when architectural, loss, and optimization changes are combined (Yadav et al., 2024).

A recurring unresolved issue is the distinction between reducing the source of TIM and making the system robust to its consequences. Shared kernels, precision alignment, configuration coverage, and context matching reduce the discrepancy itself. Importance sampling, score centering, trust-region objectives, learning-rate scheduling, and rollback instead control how the remaining discrepancy affects optimization. These strategies can be complementary rather than interchangeable.

Future work concerns broader architectures, non-mathematical RL tasks, structured prediction, source-free adaptation, long-horizon streaming, lower-precision execution, asynchronous systems, quantized and MoE models, and deployment policies whose inference behavior cannot be fully reproduced by the training engine. Across these settings, the central methodological requirement is to measure the conditions actually encountered at inference—probability distributions, contexts, parameter states, numerical paths, or configurations—rather than assuming that nominal equality of models implies equality of behavior.