Prompt-Guided Dual Latent Steering

• The paper introduces PDLS, a training-free dual-latent inversion framework that balances pixel-level structure and prompt-guided semantics using a closed-form LQR controller.
• It decomposes inversion into structural and semantic paths, dynamically fusing them via Rectified Flow models to mitigate semantic drift in corrupted images.
• Experimental results on deblurring, super-resolution, and inpainting show PDLS achieves consistently superior LPIPS, PSNR, and SSIM, validating its robust performance.

Prompt-Guided Dual Latent Steering (PDLS) is a training-free framework for inversion problems in diffusion models that addresses the difficulty of inverting corrupted images into the latent space while preserving both structural fidelity and semantic accuracy. The method is introduced for settings in which single-latent inversion methods tend to exhibit semantic drift, including blurred details and incorrect attributes, because they struggle to balance pixel-level consistency against semantic cues. PDLS builds on Rectified Flow models for their stable inversion paths, decomposes inversion into structural and semantic streams, and fuses these streams during reverse generation through an optimal-control controller with a closed-form Linear Quadratic Regulator (LQR) solution (Wu et al., 23 Sep 2025).

1. Problem setting and motivation

Inversion in diffusion models is defined here as reconstructing a clean, high-fidelity image from a degraded or corrupted observation by mapping the observed image back into the model’s latent space. The motivating degradations include blurred, low-resolution, and masked inputs. Prevailing methods encode the image into a single latent vector through deterministic processes such as DDIM inversion and RF-Inversion, primarily enforcing pixel-level consistency. The reported consequence is a fundamental trade-off: reconstructions may preserve structure while drifting semantically, or may pursue semantics while losing source integrity (Wu et al., 23 Sep 2025).

The paper identifies three limitations of this single-latent regime. First, semantic drift produces artifacts such as blurred textures or incorrect attributes. Second, reconstruction becomes unfaithful when the input is severely degraded or off-manifold. Third, existing diffusion-based techniques rarely leverage textual semantics, or do so via expensive per-image optimization. PDLS is proposed specifically to fuse pixel-level fidelity and prompt-based semantics without retraining and without costly per-image optimization (Wu et al., 23 Sep 2025).

This framing suggests that PDLS is not merely a restoration heuristic. It is positioned as a response to a structural limitation of single-path inversion: one latent trajectory is forced to compromise between low-level support from the observation and high-level semantic plausibility.

2. Dual-path latent construction

The core design of PDLS is a dual-path inversion framework with two complementary trajectories. The structural path inverts the corrupted image using a null prompt and places emphasis on preserving geometric and pixel-level details. This path leverages attention maps without semantic injection. The semantic path performs inversion guided by a user-provided text prompt, infusing high-level, class- or instance-specific semantic cues such as “green sea turtle under daylight” (Wu et al., 23 Sep 2025).

Both paths are computed with a Rectified Flow model’s ODE solver but under different prompt conditions, producing two latent trajectories:

• $Y_t$: structural path latent at time $t$, under a null prompt.
• $Y'_t$: semantic path latent at time $t$, under the guided prompt.

The reverse process, which generates from latent to image, does not select one path over the other. Instead, it fuses them “on the fly” through adaptive steering. The target anchor is defined as the average of the two path latents,

$\bar{y}_t = \frac{1}{2}(y_t + y'_t).$

A common misunderstanding is to reduce PDLS to simple latent averaging. The method does define an averaged target latent, but the paper’s operative mechanism is dynamic steering at every denoising step rather than a one-shot interpolation between two endpoints (Wu et al., 23 Sep 2025).

3. Rectified Flow foundation and control-theoretic formulation

PDLS is built on the Rectified Flow (RF) model, described by

$$\frac{d\mathbf{x}_t}{dt} = \mathbf{v}_\theta(\mathbf{x}_t, t), \qquad t \in [0, 1],\quad \mathbf{x}_0 \sim \mathcal{N}(0, I).$$

RF models are characterized as learning a nearly straight, deterministic transport from noise to data, trained via ODEs rather than stochastic SDEs. The stated motivation for this choice is that the straight path avoids the curvature and instability of classic diffusion, offering numerically stable and structure-preserving inversions. RF-Inversion is cited as demonstrating inversion via optimal control, but only along a single path, which compounds semantic drift when the input is noisy (Wu et al., 23 Sep 2025).

PDLS formulates prompt-guided inversion as an optimal control problem. The controlled ODE for the reverse process is

$$dX_t = \left[\, v_t(X_t) + \eta \big(v_t(X_t \mid \bar{y}_t) - v_t(X_t)\big) \right] dt,$$

where $v_t(\cdot)$ is the drift of the reverse SDE/ODE, $v_t(X_t \mid \bar{y}_t)$ is the drift targeting the averaged dual-anchor, and $\eta$ is the controller strength. The controller is derived by minimizing the control energy and the deviation from the dual path at final time through an LQR objective,

$t$0

subject to

$t$1

The resulting optimal control has the closed form

$t$2

The interpretation given in the paper is that the controller exerts a stronger pull toward the reference as the process nears completion. PDLS also uses a cosine decay for $t$3, with steering maximal at $t$4 and vanishing at $t$5, so that global structure is imposed early while fine details emerge near the end without further perturbation (Wu et al., 23 Sep 2025).

4. Workflow, architecture, and mechanism of drift mitigation

The workflow consists of two stages. First, dual inversion is performed: a structural ODE inverts the observed image with a null prompt to produce $t$6, and a semantic ODE inverts the image with the text prompt to produce $t$7. Second, reverse denoising with latent steering is carried out by initializing from $t$8, the structural path latent, and steering the current latent toward $t$9 at each timestep until the image is recovered at $Y'_t$0 (Wu et al., 23 Sep 2025).

The model backbone is the Flux RF model architecture. No fine-tuning or network modifications are needed. The paper therefore characterizes the framework as training-free and broadly compatible with existing pretrained RF models (Wu et al., 23 Sep 2025).

The semantic-drift argument is explicit. Single-latent methods drift because one inversion path is forced to compromise: it can anchor to pixels, which causes poor semantics, or anchor to the prompt, which loses structure. PDLS mitigates this through dual anchoring, so that the generated image both preserves the input’s structural content and aligns with the prompt’s semantics at every denoising step. The closed-form LQR solution is presented as a theoretically grounded, globally consistent interpolation along the reverse path rather than only at the endpoints. The time-decay schedule is used to prevent over-imposing prompt semantics late in generation, which could otherwise lead to overfitting or hallucinations (Wu et al., 23 Sep 2025).

This suggests that PDLS should be understood as a trajectory-level control method rather than a post hoc semantic correction.

5. Experimental validation

PDLS is evaluated on multiple inverse problems: super-resolution with $Y'_t$1 upscaling from highly downsampled inputs, motion deblurring, Gaussian deblurring, and free-form inpainting with arbitrarily masked-out regions. The datasets are FFHQ-1K and ImageNet-1K, each comprising 1000 images at $Y'_t$2. Evaluation uses LPIPS for perceptual quality, PSNR for pixel-level fidelity, and SSIM for structural similarity (Wu et al., 23 Sep 2025).

The reported quantitative findings are that PDLS consistently achieves top or near-top scores across tasks. For motion deblurring, the paper reports LPIPS $Y'_t$3, PSNR $Y'_t$4, and SSIM $Y'_t$5. For Gaussian deblurring, it reports LPIPS $Y'_t$6, PSNR $Y'_t$7, and SSIM $Y'_t$8. For $Y'_t$9 super-resolution, it reports LPIPS $t$0, PSNR $t$1, and SSIM $t$2. For inpainting, it reports LPIPS $t$3, PSNR $t$4, and SSIM $t$5 (Wu et al., 23 Sep 2025).

The qualitative findings are described in terms of images that preserve correct pose and shapes while recovering textures and colors aligned with the semantic prompt. The ablation results state that using only structure or only prompt reintroduces earlier failure modes, whereas combining both through PDLS retains the strengths of both branches. The reported generalization claim is that PDLS works across multiple degradation types and on both face-centric and generic datasets (Wu et al., 23 Sep 2025).

6. Interpretation, misconceptions, and related developments

PDLS is explicitly not a retraining-based method. It is also not a purely prompt-based inversion method. The paper’s stated advantages are unified structure–semantic guidance at every generation step, training-free operation with existing pretrained RF models, a closed-form controller with negligible computational overhead, robustness to noisy, blurry, and damaged inputs, and fine-grained control through prompt-length tuning to balance stylistic hallucination and structural realism (Wu et al., 23 Sep 2025).

A second misconception is that prompt guidance alone is sufficient to resolve inversion failures. In the PDLS formulation, prompt guidance is one branch of a dual system. The structural branch is retained precisely because prompt-only guidance can lose source integrity, while structure-only guidance can drift semantically. The method’s central claim is that the point of control lies in dynamically steering toward where structure and semantics best agree (Wu et al., 23 Sep 2025).

Later work in adjacent domains illustrates how dual-latent or dual-steering ideas have been adapted beyond the original setting. CARE, a training-free controllable restoration framework for medical images, uses a dual-latent restoration strategy in which one branch enforces data fidelity and anatomical consistency while the other leverages a generative prior to recover missing or degraded information; a risk-aware adaptive controller adjusts their contribution based on restoration uncertainty, local structural reliability, and a user-supplied control parameter $t$6 (Liu, 26 Mar 2026). In a different modality, a symbolic music generation framework introduces Dual Steering with Gram-Schmidt Orthogonalization to reduce conceptual interference between pitch and duration steering vectors in the residual stream of a Multitrack Music Transformer (Prokopiou et al., 29 May 2026). These are distinct methods rather than variants of PDLS, but they indicate that dual-path control and disentanglement have become recurring design patterns in inference-time steering.

Within that broader landscape, PDLS occupies the specific niche of prompt-guided inversion for corrupted natural images under Rectified Flow dynamics, with dual latent trajectories and LQR-based steering as its defining technical components (Wu et al., 23 Sep 2025).