Inference-Time Trajectory Alignment

Updated 5 July 2026

Inference-time trajectory alignment is a framework that leverages mid-generation interventions to refine latent trajectories for improved output quality.
It employs methods like joint optimization, token-level nudging, and population resampling to adapt to non-stationary dynamics across various modalities.
Empirical evidence indicates these techniques enhance safety and diversity while mitigating reward hacking, though they face context-specific challenges.

Inference-time trajectory alignment denotes a family of methods that improve model behavior by intervening along the generation or execution path during sampling, decoding, or rollout, rather than only retraining parameters or ranking completed outputs. In the recent literature, the aligned object may be a denoising trajectory in diffusion and flow-matching models, an autoregressive token sequence in LLMs, a stochastic audio-generation path in video-to-audio generation, or a harness-conditioned execution trace in agents. The unifying premise is that alignment is determined by the evolving trajectory itself: a single fixed prompt embedding, a one-shot latent perturbation, or terminal-only selection is often insufficient when the process is non-stationary, uncertainty is localized, or small mid-course interventions can redirect the final outcome (Sun et al., 22 Jun 2026, Lin et al., 8 Jun 2026, Zhang et al., 7 Jun 2026, Wang et al., 15 May 2026).

1. Problem formulation and conceptual scope

A central motivation for inference-time trajectory alignment is the claim that many existing alignment procedures are too single-axis and too static. In text-to-image diffusion, conditioning-side methods edit cross-attention or embeddings without updating the latent trajectory, latent-side methods perturb $z_t$ while keeping conditioning fixed, and many methods act only once at the start rather than adapting as denoising evolves. PG-MAP states that this is problematic because the denoising trajectory is non-stationary: early steps emphasize global structure / composition, whereas later steps emphasize fine detail / texture; a single fixed prompt embedding $c_0$ reused across all steps cannot reflect this changing information need (Sun et al., 22 Jun 2026).

In LLM inference, the same trajectory-level objection appears in a different form. GGRO argues that Best-of- $N$ and rejection sampling treat alignment as a search problem over complete responses: they sample multiple candidates from the base model, score them with a reward model, and return the best one. The paper identifies two limitations: coverage bound by the base model, because search can only select from what the model already produces, and reward hacking vulnerability, because maximizing an imperfect reward over many samples can overfit reward-model quirks (Lin et al., 8 Jun 2026).

Safety work sharpens the trajectory perspective further. The paper on inference-time vulnerability argues that so-called shallow safety is only a special case of a broader phenomenon: short token injections at essentially any generation step can redirect subsequent safety behavior. Harmful injections increase attack success rate, refusal injections suppress it, and the effect is strongest early but persists later in decoding. The paper therefore treats safety not as a property of the initial prefix or a static hidden-state direction, but as a property of the entire generation trajectory (Park et al., 3 Jun 2026).

Across these settings, inference-time trajectory alignment is best understood as the attempt to control an evolving path $\tau$ rather than a single state. This suggests a shared abstraction: the model’s intermediate states are not incidental bookkeeping, but the locus at which alignment, misalignment, and recovery are decided.

2. Optimization principles for aligned trajectories

One major line of work formulates trajectory alignment as explicit optimization over latent states, conditioning variables, or path distributions. PG-MAP is the clearest statement of this view. It treats each denoising step as a joint optimization problem over the text conditioning $c$ and the current latent $z_t$ and solves

$(c_t^\star, z_t^\star)=\arg\max_{c,z_t}\mathcal J_t(c,z_t).$

Its key coupling term is the forward-consistency residual

$r_t(c,z_t)=z_t-\sqrt{a_{t\mid s}\,\hat z_{s,\theta}(z_t,t,c)},$

which forces the optimized pair $(c,z_t)$ to remain consistent with the denoiser’s own predicted trajectory. The paper interprets the objective as a Gibbs-MAP / proximal energy optimization: the residual acts as a virtual observation term, Gaussian anchors impose a proximal trust region, and an optional frozen reward $Q(\hat x_0(z_t,c),y)$ biases the solution toward higher preference scores. For diffusion, the latent trust region follows the noise schedule via

$c_0$ 0

whereas for flow matching the framework reduces to the latent-only variant UG-FM with

$c_0$ 1

and an Euler ODE residual (Sun et al., 22 Jun 2026).

A second formulation casts alignment as sampling from a reward-tilted distribution. FVD defines the target as

$c_0$ 2

and argues that inference-time alignment for diffusion should preserve diversity while exploring this tilted distribution. Its solution is a Fleming–Viot birth-death process over parallel denoising trajectories: particles survive or die independently according to intermediate reward proxies, dead particles are reborn from surviving donors, and stochastic rebirth noise prevents deterministic cloning under DDIM with $c_0$ 3. The method is explicitly designed to avoid the diversity collapse and lineage collapse associated with multinomial SMC resampling (Shekhar et al., 8 Apr 2026).

Learnable Chernoff Baselines formulate the same KL-regularized reward-alignment problem through exponential tilting of the reverse process. The ideal aligned marginal is

$c_0$ 4

and the aligned reverse kernels are tilted by a soft value function $c_0$ 5. LCBs then approximate these kernels with rejection sampling whose acceptance threshold is a learned, state-dependent baseline $c_0$ 6. The paper’s stated goal is to retain a principled compute–accuracy tradeoff while requiring only black-box sampling access to the pretrained model (Madhow et al., 8 Feb 2026).

MIRA introduces a distinct constraint: the aligned trajectory should remain close to the image distribution induced by the frozen backbone. Its conceptual objective is

$c_0$ 7

motivated by the claim that noise-space regularization is insufficient because small changes in noise can still induce large changes in image space. The tractable surrogate replaces the image-space KL with a score-based quantity accumulated along the reverse trajectory (Zhai et al., 2 Oct 2025).

These formulations differ in mechanism, but they share two structural commitments: alignment is defined over a path rather than a terminal sample, and the path is constrained by either model dynamics, proximal trust regions, reward tilting, or distributional regularization.

3. Intervention strategies during decoding and sampling

A second organizing axis concerns where and how often intervention occurs. Some methods intervene locally at a small number of critical points; others maintain a population of trajectories and repeatedly reallocate compute.

GGRO is a minimal-intervention LLM method. It computes token-level entropy

$c_0$ 8

marks positions with $c_0$ 9 as insertion points, and uses reward-model gradients to choose a deterministic nudging token that is inserted before decoding continues. The method therefore shifts from post hoc selection over complete responses to active, local trajectory steering during decoding (Lin et al., 8 Jun 2026).

SIA adopts a related sparsity principle. It defines a gated policy

$N$ 0

and uses entropy-based gating $N$ 1 to intervene only at high-entropy junctions. The paper’s claim is that only a minority of steps are truly alignment-critical, so dense steering is both unnecessarily expensive and potentially harmful to generation quality (Hu et al., 30 Jan 2026).

Darwin treats decode-time alignment as a heuristic-guided tree search. It combines instruction mutation for exploration with reward-guided beam replacement for exploitation. Partial trajectories are scored by a reward model, low-value states are periodically replaced by copies of top states every $N$ 2 tokens, and an archive stores high-performing mutated instructions across search rounds. In this view, inference-time trajectory alignment becomes search over both response continuations and instruction-induced branches (Hung et al., 2024).

Population methods intervene at an even coarser level. SMC-ITA formulates flow-matching video-to-audio alignment as a particle-filtering problem over stochastic denoising trajectories. At predefined search times, each trajectory is propagated, rolled forward with lookahead, scored by a z-normalized average of semantic, AV-align, temporal, and quality rewards, reweighted by

$N$ 3

and resampled, with systematic resampling used as the default (Zhang et al., 7 Jun 2026). FVD uses a related population logic but replaces standard multinomial resampling with independent survival/death events and donor rebirth to preserve lineage diversity (Shekhar et al., 8 Apr 2026).

The intervention patterns can be summarized compactly.

Intervention locus	Representative methods	Stated mechanism
Sparse local decoding steps	GGRO, SIA	Entropy-triggered token insertion or gated value steering
Periodic partial-trajectory pruning	Darwin	Reward-guided replacement and instruction mutation
Population-level resampling	FVD, SMC-ITA	Birth-death or SMC reallocation over parallel trajectories
Per-step latent/conditioning optimization	PG-MAP, MIRA	Joint MAP refinement or score-regularized reward ascent

The literature therefore distinguishes between local corrective steering, search over partial continuations, and trajectory-distribution reweighting. The choice is largely determined by the underlying generative transport and the available reward signal.

4. Modalities, transports, and the object being aligned

The phrase “trajectory alignment” is used across multiple technical settings, but the aligned object is not always the same. In pretrained text-to-image models, the aligned trajectory is usually the reverse path of a diffusion or flow-matching sampler. PG-MAP explicitly adapts its active set by transport—joint $N$ 4 refinement for diffusion, latent-only data-side refinement for flow matching—because the paper argues that deterministic FM ODE dynamics amplify perturbations strongly along the remaining trajectory and make conditioning refinement less useful (Sun et al., 22 Jun 2026).

In LLMs, the aligned trajectory is the token-by-token decoding path. GGRO steers local continuations with nudging tokens, SIA gates reward guidance at high-entropy decision points, and the safety-vulnerability paper treats a harmful or refusal injection as a perturbation that moves the model between refusal and compliance regions in trajectory space (Lin et al., 8 Jun 2026, Hu et al., 30 Jan 2026, Park et al., 3 Jun 2026).

In audiovisual generation, SMC-ITA aligns stochastic denoising trajectories in a flow-matching video-to-audio model, but the target is explicitly multi-dimensional cross-modal reward rather than a single scalar preference. The trajectory must jointly satisfy audiovisual alignment, semantic consistency, temporal synchronization, and perceptual quality (Zhang et al., 7 Jun 2026).

Agent work generalizes the concept again. The harness paper defines a harness-conditioned execution trajectory

$N$ 5

where task decomposition sets sub-goals and guided execution reweights local action distributions. Final task success is factorized into stagewise recoverability, so the alignment problem becomes the preservation of a recoverable execution path rather than token-level preference alone (Wang et al., 15 May 2026).

This heterogeneity matters. In some papers, inference-time trajectory alignment means steering a frozen generative transport; in others, it means maintaining robustness of an autoregressive path under perturbation or keeping an agent’s workflow on a recoverable execution trace. The commonality is structural rather than modal: the intervention operates on the process, not merely on the endpoint.

5. Empirical evidence across domains

The recent literature reports improvements on preference alignment, safety robustness, multimodal synchronization, and efficiency, but the reported quantities depend heavily on the domain and evaluation protocol.

PG-MAP reports consistent gains on diffusion backbones and stronger effects on flow matching. On PartiPrompts ( $N$ 6), the paper summarizes consistent +5 to +7 pp gains on PickScore / Aesthetic for PG-MAP variants on diffusion backbones. On SD3.5-medium, the flow-matching specialization UG-FM reaches 91.9% PickScore and 75.7% HPS win rates against a static baseline; on SDXL, human evaluation with 62 PartiPrompts, 100 raters, and 6200 judgments prefers PG-MAP over the static baseline at 60.2%, over Tuned-CFG at 56.0%, and over NFE-matched universal guidance at 66.8% (Sun et al., 22 Jun 2026).

FVD reports gains in both text-to-image and class-conditional posterior sampling. Its abstract states that on DrawBench it outperforms prior methods by 7% in ImageReward, on class-conditional tasks it improves FID by roughly 14–20% over strong baselines, and it is up to 66 times faster than value-based approaches. The paper also reports that with $N$ 7 and $N$ 8 on CIFAR-10, FKD collapses to 5 distinct lineages, whereas FVD retains 52 (Shekhar et al., 8 Apr 2026).

SMC-ITA reports improvements for flow-matching video-to-audio generation under a matched NFE = 800 budget. Relative to naive single-trajectory sampling, it reports a 55.67% relative reduction in DeSync, a 20.23% improvement in IB-score, and a 15.44% improvement in Audio Quality. In the main table it is best or near-best on KL $N$ 9 = 1.578, KL $\tau$ 0 = 1.578, AQ = 21.764, IB-score = 34.114, AVCC = 21.666, DeSync = 0.219, PQ = 6.126, CU = 5.530, and CE = 4.103 (Zhang et al., 7 Jun 2026).

On the LLM side, GGRO reports its strongest gains on safety: on the 8B model and HEx-PHI with prefilling attacks, it achieves 26.2% ASR, better than BoN, RS, CARDS, CBS, and SEA; it also reports competitive or stronger performance on HH-RLHF, ARC-Challenge, and MMLU-Pro (Lin et al., 8 Jun 2026). SIA reports that steering only 20% to 80% of tokens achieves superior alignment-efficiency trade-offs, that for strong base models such as Qwen3 intervening on as few as 20% of tokens can match or surpass heavily post-trained instruct models, and that sparsity reduces computational cost by up to 6x (Hu et al., 30 Jan 2026). Darwin reports improvements over ARGS on AlpacaEval 2 and MT-Bench, with representative results such as LC 30.47, WR 33.90 for Llama3-Instruct under Mutation(iter=3) + Replacement, compared with LC 29.91, WR 30.68 for Sample-15 + Best-of-15 (Hung et al., 2024).

A compact selection of reported outcomes is useful for orientation.

Setting	Method	Reported outcome
Diffusion / flow-matching T2I	PG-MAP	91.9% PickScore and 75.7% HPS win rates on SD3.5-medium
Diffusion alignment with population resampling	FVD	7% ImageReward gain on DrawBench; up to 66 times faster than value-based approaches
Flow-matching V2A	SMC-ITA	55.67% relative reduction in DeSync; 20.23% improvement in IB-score
LLM safety decoding	GGRO	26.2% ASR on HEx-PHI with prefilling attacks on the 8B model
Sparse LLM steering	SIA	Steering 20%–80% of tokens; up to 6x cost reduction
Decode-time tree search	Darwin	LC 30.47, WR 33.90 on Llama3-Instruct with Mutation(iter=3) + Replacement

The empirical record is therefore not uniform in metric or protocol, but it consistently supports the narrower claim that trajectory-level intervention can outperform terminal-only selection or static one-axis control within the tested regimes.

6. Limitations, failure modes, and adjacent training-time approaches

The literature also converges on several recurring failure modes. The first is reward hacking. GGRO motivates local steering partly as a defense against the tendency of large-sample search to overfit imperfect reward models (Lin et al., 8 Jun 2026). MIRA argues more strongly that reward hacking in text-to-image diffusion is essentially distributional drift: reward increases can coincide with prompt failure, oversaturated images, or nearly black/white outputs, and preventing this requires an explicit image-space constraint rather than merely a noise-space one. The paper reports CMMD = 0.281 for DNO versus CMMD = 0.063 for MIRA and emphasizes that higher raw reward need not imply better images (Zhai et al., 2 Oct 2025).

A second failure mode is trajectory collapse under strong selection pressure. FVD frames multinomial SMC resampling as structurally prone to diversity loss, with roughly $\tau$ 1 of ancestors lost in one uniform resampling step asymptotically, independent of $\tau$ 2. Its birth-death dynamics, donor rebirth, and stochastic rebirth noise are all introduced to slow lineage collapse (Shekhar et al., 8 Apr 2026). A related reliability problem appears in SMC-ITA, where early-step rewards correlate poorly with final reward and lookahead is needed to make intermediate estimates more predictive (Zhang et al., 7 Jun 2026).

A third limitation is that trajectory control is often highly context-dependent. PG-MAP’s oracle-routing analysis shows that the relative importance of conditioning and latent optimization depends on prompt types: for SDXL, an oracle router over $\tau$ 3 reaches 72.7% PickScore, 73.5% HPS, 63.8% CLIP, and 68.2% Aesthetic; for SD 1.5, the oracle ceiling is 75.2%, 76.9%, 65.6%, and 66.7% respectively. The paper presents this as evidence of further headroom for a per-prompt selector (Sun et al., 22 Jun 2026).

In LLM safety, the vulnerability paper adds a different caution: alignment with refusal directions in hidden states does not predict robustness. The model may have “refused but didn’t resist,” meaning harmful injection still succeeds despite refusal-like internal states. The proposed remedy is not another inference-time controller but trajectory-level safety alignment during training, using augmented generation trajectories with simulated mid-sequence perturbation and a SimPO objective (Park et al., 3 Jun 2026).

This connects inference-time alignment to a broader family of training-time trajectory methods. Diffusion-Sharpening explicitly moves trajectory optimization into training so that inference is amortized: it samples multiple candidate denoising trajectories, aggregates reward over the trajectory, selects the best one, and fine-tunes on that trajectory, thereby avoiding extra test-time NFEs (Tian et al., 17 Feb 2025). SwiftVideo uses trajectory alignment across inference steps as a post-training strategy for few-step video generation: higher-step generations are treated as preferred trajectories, lower-step generations as less preferred ones, and a flow-based DPO objective plus a reflow regularizer transfer quality from longer to shorter Euler trajectories (Sun et al., 8 Aug 2025).

These adjacent works do not eliminate the importance of inference-time alignment; rather, they delimit its scope. Some trajectory problems are best addressed by training on aligned or perturbed paths, others by sparse or population-based test-time control, and some by a combination of both. A plausible implication is that future systems will mix these strategies: amortize what is stable, intervene only at genuinely critical junctions, and reserve expensive trajectory search for prompts or tasks where static alignment remains brittle.