Inverse Prompting in Diffusion Models

• Inverse Prompt is a reverse process that recovers discrete vocabulary tokens to form readable text prompts which best explain generated images.
• The method employs a delayed projection technique using L-BFGS optimization over high-noise timesteps to navigate a discrete, non-differentiable search space.
• Applications include concept removal, evolutionary multi-concept generation, and pseudo-segmentation, making prompt inversion valuable in controlled text-to-image synthesis.

Searching arXiv for the target paper and a few closely related inverse-prompting papers to ground the article in current arXiv metadata. Inverse prompt denotes a reverse-direction formulation in which a model, signal, or observation is used to recover, optimize, or verify the prompt-like condition that would have produced it. In text-to-image diffusion, the term most specifically refers to prompt inversion: given a target image and a frozen text-conditioned latent diffusion model, recover a sequence of discrete vocabulary tokens whose conditioning best explains that image, rather than learning a continuous pseudo-token embedding. In this setting, the central object is a readable, semantically interpretable prompt recovered from the model’s own vocabulary, and the main technical difficulty is that the search space is discrete, combinatorial, and non-differentiable (Mahajan et al., 2023).

1. Prompt inversion in text-to-image diffusion

In the Stable Diffusion–style setting considered by Mahajan et al., the problem is: given a pre-trained latent diffusion model $\epsilon_\theta$, its text encoder $f(\cdot)$, and a target image $\mathbf{I}$, find a text prompt $\mathbf{S}$ made of discrete vocabulary tokens such that, under the frozen model, the prompt best explains the image. The objective is written as

$$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$

where $\mathbf{x}=E(\mathbf{I})$ is the latent image and $\mathbf{x}_t$ is the noisy latent at timestep $t$ (Mahajan et al., 2023).

This formulation differs fundamentally from embedding inversion methods such as Textual Inversion, DreamBooth, and related approaches. Those methods optimize continuous embeddings in the text encoder’s embedding space and often introduce new pseudo-tokens that do not correspond to actual vocabulary items. PH2P instead inverts directly to existing vocabulary tokens: no new tokens are added, the result is a sequence of vocabulary indices, and the prompt is readable text. This yields interpretability and permits downstream editing, composition, and concept removal at the language level (Mahajan et al., 2023).

The distinction is consequential. Continuous embedding inversion can produce representations that are far from the original vocabulary manifold, making them uninterpretable and sometimes entangled or non-generalizable. Hard prompt inversion constrains the solution to the model’s vocabulary and therefore emphasizes semantic legibility. This suggests that the price of interpretability is a harder optimization problem, because the search domain is $V^L$ with $|V|^L$ possible sequences.

2. Latent diffusion formulation and discrete search space

The model assumes a Stable Diffusion–style latent diffusion backbone. The input RGB image $f(\cdot)$0 is mapped by an encoder $f(\cdot)$1 to a latent $f(\cdot)$2, and a decoder $f(\cdot)$3 maps latents back to image space. The U-Net denoiser $f(\cdot)$4 is conditioned on a CLIP text encoding $f(\cdot)$5. The latent diffusion training loss is

$f(\cdot)$6

Prompt inversion reuses this loss with fixed $f(\cdot)$7 and optimizes $f(\cdot)$8 instead (Mahajan et al., 2023).

The prompt pipeline is discrete. Tokenization maps a prompt string $f(\cdot)$9 to indices $\mathbf{I}$0, each $\mathbf{I}$1. An embedding lookup maps each token index to $\mathbf{I}$2, with embedding matrix $\mathbf{I}$3. Gradients can flow through $\mathbf{I}$4 with respect to embeddings, but not directly with respect to token indices. Naive projected SGD therefore becomes unstable: optimizing continuous embeddings and projecting them to nearest vocabulary items at every step can lead to vanishing gradients and high-variance updates due to the deep CLIP+U-Net stack and the stochastic diffusion process (Mahajan et al., 2023).

PH2P further restricts optimization over timesteps. Rather than optimize over all $\mathbf{I}$5, it minimizes

$\mathbf{I}$6

with $\mathbf{I}$7. For concept removal, it replaces the objective by a difference of two LDM losses, driving the prompt to better explain a positive image and worse explain a negative image: $\mathbf{I}$8 No explicit regularization on text is added; interpretability is imposed by the vocabulary projection itself (Mahajan et al., 2023).

3. Delayed projection and semantic timesteps

The core technical device in PH2P is a delayed projection scheme over continuous prompt embeddings. Instead of optimizing token indices directly, the method introduces a continuous variable $\mathbf{I}$9. A projection operator maps each position to its nearest vocabulary embedding: $\mathbf{S}$0 Losses and gradients are computed on the projected embeddings $\mathbf{S}$1, but the parameter actually updated is the underlying continuous $\mathbf{S}$2, using L-BFGS rather than Adam: $\mathbf{S}$3 This allows $\mathbf{S}$4 to wander in continuous space while retaining discrete, vocabulary-valid evaluations (Mahajan et al., 2023).

A second empirical observation is that diffusion timesteps separate semantics from detail. Early timesteps, with low noise, are relatively insensitive to prompt changes and mainly refine fine-grained details and textures. Late timesteps, with high noise, are much more sensitive to semantic content. In the paper’s loss-versus-timestep analysis, semantically aligned and random prompts have similar loss values at early timesteps, while the loss curves diverge at later timesteps; prompts that include words like “cabin” or “snowy” approach the original prompt’s loss only at high $\mathbf{S}$5. Prompt inversion over very high-noise timesteps recovers coarse semantics, extending to mid-to-high timesteps refines them, and including very early timesteps can degrade prompt quality by introducing special characters or non-informative tokens (Mahajan et al., 2023).

This yields a specific algorithmic stance: optimize only over the high-noise interval $\mathbf{S}$6, use L-BFGS with delayed projection, and recover tokens representative of image semantics rather than low-level texture. The practical effect is a prompt that remains readable and content-relevant.

4. Empirical behavior and evaluation

PH2P produces prompts that are semantically interpretable rather than pseudo-token strings. Scene images can yield prompts containing words such as “shops”, “volkswagen”, and “headlights”; temple and architecture examples contain “pavilion”, “pagoda”, “japanese”, and “hilltop”; sports images contain “surfing” and “wave”; ski-resort images contain “alps”, “skiing”, and “resort”. When these recovered prompts are used to synthesize images with the same Stable Diffusion model, the outputs are semantically aligned with the target image while remaining diverse in pose, layout, and appearance (Mahajan et al., 2023).

On COCO and LSUN, the paper reports gains over PEZ in reconstruction quality, textual semantic quality, and diversity. The quantitative comparison is as follows (Mahajan et al., 2023):

Setting	PEZ	PH2P
CLIP similarity, COCO / LSUN	0.72 / 0.70	0.77 / 0.77
LPIPS similarity, COCO / LSUN	0.477 / 0.480	0.462 / 0.463
LPIPS diversity, COCO / LSUN	0.417 / 0.420	0.435 / 0.422
BERTScore Precision / Recall / F1	0.772 / 0.835 / 0.802	0.803 / 0.838 / 0.820

The ablations isolate both design choices. On COCO, PH2P reaches CLIP similarity $\mathbf{S}$7, LPIPS similarity $\mathbf{S}$8, and LPIPS diversity $\mathbf{S}$9, compared with $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$0 for LDM+Adam and $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$1 for LDM+all $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$2. These numbers support two conclusions: restricting optimization to late timesteps matters, and L-BFGS with delayed projection matters (Mahajan et al., 2023).

5. Applications and downstream manipulations

The recovered prompts are not only reconstructions; they are reusable control objects. One application is evolutionary multi-concept generation. A user gathers reference images, runs PH2P on each to obtain prompts $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$3, composes or edits them into a composite prompt, generates new images, and can then invert one of those new images again to refine the prompt in multiple hops. The paper illustrates sequences such as flooded street $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$4 flooded city street with boat, showing that complex prompts can emerge from purely visual exemplars (Mahajan et al., 2023).

A second application is concept removal via negative image prompting. Here the loss is the difference between positive and negative LDM losses, so the prompt is optimized to retain the target image content while failing to explain the negative concept. Examples include removing “bird” from a bird-on-branch image or removing a specific food item from a table scene while preserving the rest of the scene (Mahajan et al., 2023).

A third application is unsupervised segmentation or localized editing through cross-attention. After inverting an image to a prompt $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$5, one can visualize token-wise cross-attention maps in Stable Diffusion. Tokens such as “dog”, “bus”, and “pizza” attend to appropriate image regions, which indicates that the recovered prompt tokens correspond to spatially disentangled concepts and can serve as pseudo-segmentation masks (Mahajan et al., 2023).

6. Broader meanings of inverse prompting

The term inverse prompt is not uniform across the literature. In PH2P it means image-to-prompt recovery in a text-to-image diffusion model, but other arXiv papers use the term for different reverse-direction constructions.

Domain	Forward direction	Inverse direction
Few-shot slot tagging	span $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$6 label	label $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$7 span (Hou et al., 2022)
Cross-domain slot filling	slot type $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$8 slot value	slot value $$\mathbf{S}^* = \arg\min_{\mathbf{S}} \;\mathbb{E}_{t,\mathbf{x},\epsilon}\left[\bigl\|\epsilon - \epsilon_\theta(\mathbf{x}_t,t,f(\mathbf{S}))\bigr\|_2^2\right],$$9 slot type (Li et al., 2023)
Controllable generation	prompt $\mathbf{x}=E(\mathbf{I})$0 text	generated text $\mathbf{x}=E(\mathbf{I})$1 prompt during beam search (Zou et al., 2021)
LLM self-improvement	select correct candidates	identify incorrect candidates (Ahn et al., 2024)
Reverse prompt engineering	prompt $\mathbf{x}=E(\mathbf{I})$2 outputs	outputs $\mathbf{x}=E(\mathbf{I})$3 prompt (Li et al., 2024)
AI detection	prompt $\mathbf{x}=E(\mathbf{I})$4 generated text	text $\mathbf{x}=E(\mathbf{I})$5 predicted prompt (Chen et al., 21 Feb 2025)
NLG evaluation	instruction $\mathbf{x}=E(\mathbf{I})$6 evaluation score	outputs and scores $\mathbf{x}=E(\mathbf{I})$7 evaluation prompt (Hong et al., 29 Apr 2025)
Imaging inverse problems	prompt $\mathbf{x}=E(\mathbf{I})$8 image prior	measurement or image $\mathbf{x}=E(\mathbf{I})$9 prompt embedding (Chung et al., 2023, Spagnoletti et al., 16 Mar 2025, Spagnoletti et al., 8 May 2026)

In few-shot slot tagging, “inverse prompting” means “given the sentence and a slot type, generate the token span(s) that fill that slot,” reducing complexity from $\mathbf{x}_t$0 span enumeration to $\mathbf{x}_t$1 per label (Hou et al., 2022). In zero-shot cross-domain slot filling, it is an auxiliary task that reverses entity spans to slot types in order to reduce multiple-prediction conflicts and improve unseen-slot generalization by $\mathbf{x}_t$2 F1 over a prior SOTA on unseen slots (Li et al., 2023). In controllable generation, it means scoring beam candidates by how well generated text can inversely predict the original prompt, which improves relevance in long-form Chinese QA and poem generation (Zou et al., 2021). In discriminative self-improvement for math reasoning, “inverse prompt” means asking which candidate solutions are incorrect rather than which are correct, and its effectiveness depends strongly on model family and instruction-following strength (Ahn et al., 2024).

The imaging literature adds a nearby but distinct usage. P2L jointly optimizes the text embedding on-the-fly while running reverse diffusion, treating the prompt as a latent variable for inverse problems (Chung et al., 2023). LATINO-PRO estimates the prompt embedding from measurements by marginal maximum likelihood, turning “prompt $\mathbf{x}_t$3 image” into “measurement $\mathbf{x}_t$4 prompt $\mathbf{x}_t$5 image” (Spagnoletti et al., 16 Mar 2025). A later Euclidean–Wasserstein-2 gradient-flow formulation couples posterior sampling and prompt optimization in a single latent-space flow (Spagnoletti et al., 8 May 2026).

7. Limitations and open directions

PH2P inherits several structural limitations. Prompt length is fixed, so complex scenes may need more tokens than the available positions. The recovered prompt is constrained to the existing CLIP vocabulary, which means rare proper names or novel stylistic descriptors cannot be recovered explicitly and must be approximated by nearby words. The inversion objective is only an approximate inverse of the generative process, because diffusion is stochastic and many-to-one; secondary objects may be omitted, and some prompts remain partly generic or contain redundant tokens. Performance is also sensitive to the timestep range $\mathbf{x}_t$6, learning rate $\mathbf{x}_t$7, number of steps $\mathbf{x}_t$8, and initialization, and using all timesteps or Adam leads to suboptimal prompts (Mahajan et al., 2023).

These limitations clarify the scope of “hard prompting” as a vocabulary-constrained inverse problem rather than a perfect captioning mechanism. The paper suggests better initialization strategies, language-model post-processing, smarter token-level search, and extensions to video or multi-frame generation. A plausible implication is that prompt inversion occupies a middle ground between captioning and embedding inversion: it is more interpretable than the latter and more model-respecting than the former, but it is bounded by the vocabulary, conditioning interface, and inductive biases of the specific diffusion model (Mahajan et al., 2023).