---
title: Inference-Time Trajectory Alignment
url: https://www.emergentmind.com/topics/inference-time-trajectory-alignment
type: topic
---

# Inference-Time Trajectory Alignment

Inference-time trajectory alignment denotes a family of methods that improve model behavior by intervening along the generation or execution path during sampling, decoding, or rollout, rather than only retraining parameters or ranking completed outputs. In the recent literature, the aligned object may be a denoising trajectory in diffusion and flow-matching models, an autoregressive token sequence in LLMs, a stochastic audio-generation path in video-to-audio generation, or a harness-conditioned execution trace in agents. The unifying premise is that alignment is determined by the evolving trajectory itself: a single fixed prompt embedding, a one-shot latent perturbation, or terminal-only selection is often insufficient when the process is non-stationary, uncertainty is localized, or small mid-course interventions can redirect the final outcome [2606.22958][2606.09635][2606.08393][2605.21516].

## 1. Problem formulation and conceptual scope

A central motivation for inference-time trajectory alignment is the claim that many existing alignment procedures are too **single-axis** and too **static**. In text-to-image diffusion, conditioning-side methods edit cross-attention or embeddings without updating the latent trajectory, latent-side methods perturb \(z_t\) while keeping conditioning fixed, and many methods act only once at the start rather than adapting as denoising evolves. PG-MAP states that this is problematic because the denoising trajectory is **non-stationary**: early steps emphasize **global structure / composition**, whereas later steps emphasize **fine detail / texture**; a single fixed prompt embedding \(c_0\) reused across all steps cannot reflect this changing information need [2606.22958].

In LLM inference, the same trajectory-level objection appears in a different form. GGRO argues that **Best-of-\(N\)** and **rejection sampling** treat alignment as a search problem over complete responses: they sample multiple candidates from the base model, score them with a reward model, and return the best one. The paper identifies two limitations: **coverage bound by the base model**, because search can only select from what the model already produces, and **reward hacking vulnerability**, because maximizing an imperfect reward over many samples can overfit reward-model quirks [2606.09635].

Safety work sharpens the trajectory perspective further. The paper on inference-time vulnerability argues that so-called **shallow safety** is only a special case of a broader phenomenon: short token injections at essentially any generation step can redirect subsequent safety behavior. Harmful injections increase attack success rate, refusal injections suppress it, and the effect is strongest early but persists later in decoding. The paper therefore treats safety not as a property of the initial prefix or a static hidden-state direction, but as a property of the **entire generation trajectory** [2606.04778].

Across these settings, inference-time trajectory alignment is best understood as the attempt to control an evolving path \( \tau \) rather than a single state. This suggests a shared abstraction: the model’s intermediate states are not incidental bookkeeping, but the locus at which alignment, misalignment, and recovery are decided.

## 2. Optimization principles for aligned trajectories

One major line of work formulates trajectory alignment as explicit optimization over latent states, conditioning variables, or path distributions. PG-MAP is the clearest statement of this view. It treats each denoising step as a **joint optimization problem over the text conditioning \(c\) and the current latent \(z_t\)** and solves
\[
(c_t^\star, z_t^\star)=\arg\max_{c,z_t}\mathcal J_t(c,z_t).
\]
Its key coupling term is the **forward-consistency residual**
\[
r_t(c,z_t)=z_t-\sqrt{a_{t\mid s}\,\hat z_{s,\theta}(z_t,t,c)},
\]
which forces the optimized pair \((c,z_t)\) to remain consistent with the denoiser’s own predicted trajectory. The paper interprets the objective as a **Gibbs-MAP / proximal energy optimization**: the residual acts as a virtual observation term, Gaussian anchors impose a proximal trust region, and an optional frozen reward \(Q(\hat x_0(z_t,c),y)\) biases the solution toward higher preference scores. For diffusion, the latent trust region follows the noise schedule via
\[
\sigma_z(t)=\gamma\sqrt{1-\bar\alpha_t},
\]
whereas for flow matching the framework reduces to the latent-only variant **UG-FM** with
\[
\mathcal A_t=\{z_t\}, \qquad \sigma_z(t)=\gamma(1-t)
\]
and an Euler ODE residual [2606.22958].

A second formulation casts alignment as sampling from a **reward-tilted distribution**. FVD defines the target as
\[
\pi^*(x_0)\propto p_\theta(x_0)\exp(\lambda r(x_0)),
\]
and argues that inference-time alignment for diffusion should preserve diversity while exploring this tilted distribution. Its solution is a **Fleming–Viot birth-death process** over parallel denoising trajectories: particles survive or die independently according to intermediate reward proxies, dead particles are reborn from surviving donors, and stochastic rebirth noise prevents deterministic cloning under DDIM with \(\eta=0\). The method is explicitly designed to avoid the **diversity collapse** and **lineage collapse** associated with multinomial SMC resampling [2604.06779].

Learnable Chernoff Baselines formulate the same KL-regularized reward-alignment problem through exponential tilting of the reverse process. The ideal aligned marginal is
\[
p^*(x_0)\propto e^{r(x_0)/\alpha}p^{\mathrm{pre}}(x_0),
\]
and the aligned reverse kernels are tilted by a soft value function \(v_t\). LCBs then approximate these kernels with rejection sampling whose acceptance threshold is a learned, state-dependent baseline \(B_{t+1}(x_{t+1})\). The paper’s stated goal is to retain a principled compute–accuracy tradeoff while requiring only black-box sampling access to the pretrained model [2602.07738].

MIRA introduces a distinct constraint: the aligned trajectory should remain close to the **image distribution** induced by the frozen backbone. Its conceptual objective is
\[
\mathcal L_{\text{MIRA}}(z,c)= -r(x_0,c)+\beta\, D_{\mathrm{KL}}\!\left(p_\theta(x_0\mid z,c)\,\|\,p_\theta(x_0\mid z_0,c)\right),
\]
motivated by the claim that **noise-space regularization is insufficient** because small changes in noise can still induce large changes in image space. The tractable surrogate replaces the image-space KL with a score-based quantity accumulated along the reverse trajectory [2510.01549].

These formulations differ in mechanism, but they share two structural commitments: alignment is defined over a path rather than a terminal sample, and the path is constrained by either model dynamics, proximal trust regions, reward tilting, or distributional regularization.

## 3. Intervention strategies during decoding and sampling

A second organizing axis concerns **where** and **how often** intervention occurs. Some methods intervene locally at a small number of critical points; others maintain a population of trajectories and repeatedly reallocate compute.

GGRO is a minimal-intervention LLM method. It computes token-level entropy
\[
H_i=-\sum_{y_j\in V}\pi(y_j\mid y_{<i},X)\log \pi(y_j\mid y_{<i},X),
\]
marks positions with \(H_i>\tau_H\) as insertion points, and uses reward-model gradients to choose a deterministic **nudging token** that is inserted before decoding continues. The method therefore shifts from post hoc selection over complete responses to **active, local trajectory steering during decoding** [2606.09635].

SIA adopts a related sparsity principle. It defines a gated policy
\[
\pi_{\text{sparse}}(y_t\mid s_t)=
\begin{cases}
\frac{1}{Z}\,\pi_{\text{base}}(y_t\mid s_t)\exp\!\big(\beta V^*(s_{t+1})\big), & \mathcal G(s_t)=1,\\
\pi_{\text{base}}(y_t\mid s_t), & \mathcal G(s_t)=0,
\end{cases}
\]
and uses entropy-based gating \(g_t=\mathbb I(H_t>\tau_H)\) to intervene only at high-entropy **junctions**. The paper’s claim is that only a minority of steps are truly alignment-critical, so dense steering is both unnecessarily expensive and potentially harmful to generation quality [2602.21215].

Darwin treats decode-time alignment as a **heuristic-guided tree search**. It combines **instruction mutation** for exploration with **reward-guided beam replacement** for exploitation. Partial trajectories are scored by a reward model, low-value states are periodically replaced by copies of top states every \(m\) tokens, and an archive stores high-performing mutated instructions across search rounds. In this view, inference-time trajectory alignment becomes search over both response continuations and instruction-induced branches [2406.15193].

Population methods intervene at an even coarser level. SMC-ITA formulates flow-matching video-to-audio alignment as a particle-filtering problem over stochastic denoising trajectories. At predefined search times, each trajectory is propagated, rolled forward with lookahead, scored by a z-normalized average of semantic, AV-align, temporal, and quality rewards, reweighted by
\[
w_{t_i}^{(n)}=
\frac{\exp(\tau R(\hat x^{(n)}_{1\mid t_i},c))}
{\sum_{\ell=1}^N \exp(\tau R(\hat x^{(\ell)}_{1\mid t_i},c))},
\]
and resampled, with **systematic resampling** used as the default [2606.08393]. FVD uses a related population logic but replaces standard multinomial resampling with independent survival/death events and donor rebirth to preserve lineage diversity [2604.06779].

The intervention patterns can be summarized compactly.

| Intervention locus | Representative methods | Stated mechanism |
|---|---|---|
| Sparse local decoding steps | GGRO, SIA | Entropy-triggered token insertion or gated value steering |
| Periodic partial-trajectory pruning | Darwin | Reward-guided replacement and instruction mutation |
| Population-level resampling | FVD, SMC-ITA | Birth-death or SMC reallocation over parallel trajectories |
| Per-step latent/conditioning optimization | PG-MAP, MIRA | Joint MAP refinement or score-regularized reward ascent |

The literature therefore distinguishes between **local corrective steering**, **search over partial continuations**, and **trajectory-distribution reweighting**. The choice is largely determined by the underlying generative transport and the available reward signal.

## 4. Modalities, transports, and the object being aligned

The phrase “trajectory alignment” is used across multiple technical settings, but the aligned object is not always the same. In pretrained text-to-image models, the aligned trajectory is usually the reverse path of a diffusion or flow-matching sampler. PG-MAP explicitly adapts its active set by transport—joint \((c,z_t)\) refinement for diffusion, latent-only data-side refinement for flow matching—because the paper argues that deterministic FM ODE dynamics amplify perturbations strongly along the remaining trajectory and make conditioning refinement less useful [2606.22958].

In LLMs, the aligned trajectory is the token-by-token decoding path. GGRO steers local continuations with nudging tokens, SIA gates reward guidance at high-entropy decision points, and the safety-vulnerability paper treats a harmful or refusal injection as a perturbation that moves the model between refusal and compliance regions in trajectory space [2606.09635][2602.21215][2606.04778].

In audiovisual generation, SMC-ITA aligns stochastic denoising trajectories in a flow-matching video-to-audio model, but the target is explicitly **multi-dimensional cross-modal reward** rather than a single scalar preference. The trajectory must jointly satisfy audiovisual alignment, semantic consistency, temporal synchronization, and perceptual quality [2606.08393].

Agent work generalizes the concept again. The harness paper defines a harness-conditioned execution trajectory
\[
\tau_h(x)=(g_1,\tau_1,\ldots,g_T,\tau_T),
\]
where task decomposition sets sub-goals and guided execution reweights local action distributions. Final task success is factorized into stagewise **recoverability**, so the alignment problem becomes the preservation of a recoverable execution path rather than token-level preference alone [2605.21516].

This heterogeneity matters. In some papers, inference-time trajectory alignment means **steering a frozen generative transport**; in others, it means **maintaining robustness of an autoregressive path under perturbation** or **keeping an agent’s workflow on a recoverable execution trace**. The commonality is structural rather than modal: the intervention operates on the process, not merely on the endpoint.

## 5. Empirical evidence across domains

The recent literature reports improvements on preference alignment, safety robustness, multimodal synchronization, and efficiency, but the reported quantities depend heavily on the domain and evaluation protocol.

PG-MAP reports consistent gains on diffusion backbones and stronger effects on flow matching. On **PartiPrompts (\(n=1632\))**, the paper summarizes **consistent +5 to +7 pp gains** on PickScore / Aesthetic for PG-MAP variants on diffusion backbones. On **SD3.5-medium**, the flow-matching specialization **UG-FM** reaches **91.9%** PickScore and **75.7%** HPS win rates against a static baseline; on **SDXL**, human evaluation with **62 PartiPrompts**, **100 raters**, and **6200 judgments** prefers PG-MAP over the static baseline at **60.2%**, over Tuned-CFG at **56.0%**, and over NFE-matched universal guidance at **66.8%** [2606.22958].

FVD reports gains in both text-to-image and class-conditional posterior sampling. Its abstract states that on **DrawBench** it outperforms prior methods by **7% in ImageReward**, on class-conditional tasks it improves **FID by roughly 14–20%** over strong baselines, and it is up to **66 times faster** than value-based approaches. The paper also reports that with \(K=1000\) and \(\lambda=1\) on CIFAR-10, FKD collapses to **5 distinct lineages**, whereas FVD retains **52** [2604.06779].

SMC-ITA reports improvements for flow-matching video-to-audio generation under a matched **NFE = 800** budget. Relative to naive single-trajectory sampling, it reports a **55.67% relative reduction in DeSync**, a **20.23% improvement in IB-score**, and a **15.44% improvement in Audio Quality**. In the main table it is best or near-best on **KL\(_{\text{PANNs}}\) = 1.578**, **KL\(_{\text{PaSST}}\) = 1.578**, **AQ = 21.764**, **IB-score = 34.114**, **AVCC = 21.666**, **DeSync = 0.219**, **PQ = 6.126**, **CU = 5.530**, and **CE = 4.103** [2606.08393].

On the LLM side, GGRO reports its strongest gains on safety: on the **8B** model and **HEx-PHI with prefilling attacks**, it achieves **26.2% ASR**, better than BoN, RS, CARDS, CBS, and SEA; it also reports competitive or stronger performance on HH-RLHF, ARC-Challenge, and MMLU-Pro [2606.09635]. SIA reports that steering only **20% to 80%** of tokens achieves superior alignment-efficiency trade-offs, that for strong base models such as **Qwen3** intervening on as few as **20%** of tokens can match or surpass heavily post-trained instruct models, and that sparsity reduces computational cost by up to **6x** [2602.21215]. Darwin reports improvements over ARGS on **AlpacaEval 2** and **MT-Bench**, with representative results such as **LC 30.47, WR 33.90** for **Llama3-Instruct** under **Mutation(iter=3) + Replacement**, compared with **LC 29.91, WR 30.68** for **Sample-15 + Best-of-15** [2406.15193].

A compact selection of reported outcomes is useful for orientation.

| Setting | Method | Reported outcome |
|---|---|---|
| Diffusion / flow-matching T2I | PG-MAP | **91.9%** PickScore and **75.7%** HPS win rates on SD3.5-medium |
| Diffusion alignment with population resampling | FVD | **7%** ImageReward gain on DrawBench; up to **66 times faster** than value-based approaches |
| Flow-matching V2A | SMC-ITA | **55.67%** relative reduction in DeSync; **20.23%** improvement in IB-score |
| LLM safety decoding | GGRO | **26.2% ASR** on HEx-PHI with prefilling attacks on the 8B model |
| Sparse LLM steering | SIA | Steering **20%–80%** of tokens; up to **6x** cost reduction |
| Decode-time tree search | Darwin | **LC 30.47, WR 33.90** on Llama3-Instruct with Mutation(iter=3) + Replacement |

The empirical record is therefore not uniform in metric or protocol, but it consistently supports the narrower claim that trajectory-level intervention can outperform terminal-only selection or static one-axis control within the tested regimes.

## 6. Limitations, failure modes, and adjacent training-time approaches

The literature also converges on several recurring failure modes. The first is **reward hacking**. GGRO motivates local steering partly as a defense against the tendency of large-sample search to overfit imperfect reward models [2606.09635]. MIRA argues more strongly that reward hacking in text-to-image diffusion is essentially **distributional drift**: reward increases can coincide with prompt failure, oversaturated images, or nearly black/white outputs, and preventing this requires an explicit image-space constraint rather than merely a noise-space one. The paper reports **CMMD = 0.281** for DNO versus **CMMD = 0.063** for MIRA and emphasizes that higher raw reward need not imply better images [2510.01549].

A second failure mode is **trajectory collapse under strong selection pressure**. FVD frames multinomial SMC resampling as structurally prone to diversity loss, with roughly \(36.8\%\) of ancestors lost in one uniform resampling step asymptotically, independent of \(K\). Its birth-death dynamics, donor rebirth, and stochastic rebirth noise are all introduced to slow lineage collapse [2604.06779]. A related reliability problem appears in SMC-ITA, where early-step rewards correlate poorly with final reward and lookahead is needed to make intermediate estimates more predictive [2606.08393].

A third limitation is that trajectory control is often highly **context-dependent**. PG-MAP’s oracle-routing analysis shows that the relative importance of conditioning and latent optimization depends on prompt types: for **SDXL**, an oracle router over \(\{f_c,f_{cz},f_{\text{tcfg}}\}\) reaches **72.7%** PickScore, **73.5%** HPS, **63.8%** CLIP, and **68.2%** Aesthetic; for **SD 1.5**, the oracle ceiling is **75.2%**, **76.9%**, **65.6%**, and **66.7%** respectively. The paper presents this as evidence of further headroom for a per-prompt selector [2606.22958].

In LLM safety, the vulnerability paper adds a different caution: **alignment with refusal directions in hidden states does not predict robustness**. The model may have “refused but didn’t resist,” meaning harmful injection still succeeds despite refusal-like internal states. The proposed remedy is not another inference-time controller but **trajectory-level safety alignment** during training, using augmented generation trajectories with simulated mid-sequence perturbation and a SimPO objective [2606.04778].

This connects inference-time alignment to a broader family of **training-time trajectory methods**. Diffusion-Sharpening explicitly moves trajectory optimization into training so that inference is amortized: it samples multiple candidate denoising trajectories, aggregates reward over the trajectory, selects the best one, and fine-tunes on that trajectory, thereby avoiding extra test-time NFEs [2502.12146]. SwiftVideo uses **trajectory alignment across inference steps** as a post-training strategy for few-step video generation: higher-step generations are treated as preferred trajectories, lower-step generations as less preferred ones, and a flow-based DPO objective plus a reflow regularizer transfer quality from longer to shorter Euler trajectories [2508.06082].

These adjacent works do not eliminate the importance of inference-time alignment; rather, they delimit its scope. Some trajectory problems are best addressed by training on aligned or perturbed paths, others by sparse or population-based test-time control, and some by a combination of both. A plausible implication is that future systems will mix these strategies: amortize what is stable, intervene only at genuinely critical junctions, and reserve expensive trajectory search for prompts or tasks where static alignment remains brittle.

Source: https://www.emergentmind.com/topics/inference-time-trajectory-alignment