Prompt-in-Image: Embedded Prompt Integration

• Prompt-in-Image is a family of methods that integrate prompt data into image computation, merging textual cues with internal image features.
• These approaches enable internal conditioning, multimodal quality assessment, and feedback-driven prompt optimization in various imaging tasks.
• Key systems like PIP, IP-IQA, and VisualPrompter demonstrate improved control, efficiency, and interactive prompt refinement across diverse applications.

Prompt-in-Image denotes a family of methods in which prompting is embedded into image computation rather than treated only as external text. In current literature, the term covers at least three distinct constructions: using prompts as “internal, learnable conditioning signals inside an image restoration network” (Li et al., 2023), treating the textual prompt as “an intrinsic, inseparable part of the quality of an AI-generated image” (Qu et al., 2024), and using the generated image itself as feedback for prompt revision through “self-reflection” and prompt optimization (Wu et al., 29 Jun 2025). Across low-level restoration, multimodal assessment, prompt engineering, and interactive systems, the unifying idea is that prompt information is carried inside the image-processing pipeline, the image representation, or the prompt–image loop rather than remaining a purely textual front end.

1. Conceptual scope and taxonomy

A common misconception is that prompt-based vision methods are necessarily text-conditioned generation systems. The literature instead uses “prompt-in-image” for feature-level prompts, structural prompts derived from the image itself, prompt-aware evaluators, and interfaces that analyze prompt histories or use generated images as feedback for subsequent prompting (Li et al., 2023, Li et al., 10 Feb 2025, Qu et al., 2024, Guo et al., 2024).

Usage	Core mechanism	Representative papers
Internal conditioning	“feature-level objects” or structural codes injected into restoration backbones	PIP (Li et al., 2023), Prompt-SID (Li et al., 10 Feb 2025)
Intrinsic multimodal evaluation	image quality defined jointly over image and prompt	IP-IQA (Qu et al., 2024)
Feedback-driven prompt engineering	generated images used to identify failures and rewrite prompts	VisualPrompter (Wu et al., 29 Jun 2025), PRISM (He et al., 2024), RATTPO (Kim et al., 20 Jun 2025)
Interactive prompt exploration	prompt histories, semantic maps, and mixed-initiative refinement	Promptify (Brade et al., 2023), PromptCrafter (Baek et al., 2023), PromptMap (Adamkiewicz et al., 12 Mar 2025), PrompTHis (Guo et al., 2024)

This taxonomy suggests that prompt-in-image is best understood as a general conditioning paradigm. The prompt may be a learned vector tied to degradation type, a multimodal token summarizing a textual instruction, a set of image-grounded questions, or a visualized history of prompt edits. The common thread is not modality but control: the prompt becomes part of the image model’s internal state or part of the formal definition of image quality.

2. Internal prompts inside image models

In universal image restoration, PIP formalizes “Prompt-in-Image” as learned prompts injected directly into the feature processing pipeline of a U-shaped restoration network (Li et al., 2023). The method defines a universal restoration operator

$\widehat Z = PIP(Z,\omega),$

where $Z$ is a skip-connection feature tensor and $\omega$ is a degradation weight vector over tasks such as denoising, deraining, dehazing, deblurring, and low-light enhancement. Its architecture separates a degradation-aware prompt $d_t \in \mathbb{R}^{C_d}$ from a basic restoration prompt $B \in \mathbb{R}^{c \times h \times w}$, fuses them through a Prompt-to-Prompt interaction module, and then applies a selective Prompt-to-Feature interaction to modulate degradation-relevant channels. The degradation-aware prompts are explicitly pushed apart with a directional decoupled loss,

$$\mathcal L = \mathcal L_1 + \alpha \mathcal L_{\text{ddl}},$$

with $\alpha=0.002$ and $\theta_{\text{thre}} = 90^\circ$ in experiments. The result is a single universal restoration model that handles multiple degradations with one backbone and exposes degradation weights $\omega$ as an interpretable control interface.

PIP’s empirical profile is defined by multi-task training on BSD400 + WED, Rain100L, SOTS, GoPro, and LOL, and by consistent gains over PromptIR and multi-task Restormer or NAFNet baselines (Li et al., 2023). In the 3-task setting, PIP$_{\text{Restormer}}$ reaches 32.91 dB / 0.920 average versus PromptIR’s 32.06 dB / 0.913, with especially large gains on SOTS and Rain100L. In the 5-task setting, PIP$Z$0 reaches 30.81 dB / 0.901 and improves Restormer deblurring from 27.31 to 28.61 dB. The method also exposes a parameter-efficiency claim: on a Restormer backbone, PIP adds +2.8% parameters, +9.7% FLOPS, and +8.6% latency, whereas PromptIR adds +26.3% parameters, +12.2% FLOPS, and +11.6% latency.

Prompt-SID extends the same general idea to self-supervised denoising, but here the prompt is a structural representation extracted from the noisy image itself (Li et al., 10 Feb 2025). A Pixel Structure Encoder produces latent codes from the original image and from downsampled views, a latent diffusion process reconstructs an original-scale structural code, and a Structural Attention Module injects that prompt into a transformer denoiser. The diffusion branch is trained with

$Z$1

and the total training objective combines reconstruction, scale replay, and diffusion losses. The central claim is that this avoids the pixel information loss of blind-spot or sub-image-pair methods while preserving structural details. On SIDD, Prompt-SID reaches 51.02 dB / 0.991 on benchmark and 51.55 dB / 0.992 on validation, and on fluorescence imaging it reaches 15.89 dB at 1 Hz and 21.10 dB at 30 Hz, matching or exceeding supervised baselines in some settings.

Taken together, these two lines of work establish an internalist interpretation of prompt-in-image. Prompts are not raw tokens but latent conditioning objects aligned with restoration operators, structural priors, or degradation semantics.

3. Prompt-aware evaluation and prediction

A second meaning of prompt-in-image arises in quality assessment, where the prompt is not merely an auxiliary input but part of the target itself. IP-IQA argues that AI-generated image quality must be defined over image–prompt pairs because AGIQA labels explicitly couple perceptual quality, aesthetics, and text–image correspondence (Qu et al., 2024). On AGIQA-1k, each image has a single MOS in $Z$2; on AGIQA-3k, each image has two MOSs in $Z$3 for perceptual quality and text–image alignment. This makes unimodal IQA models structurally mismatched to the task.

IP-IQA addresses that mismatch with a CLIP-based dual-stream architecture plus two additions (Qu et al., 2024). First, Image2Prompt incrementally pretrains the image encoder on 560K DiffusionDB pairs selected with CLIP similarity above 0.35, using a cosine-similarity objective against a frozen text encoder. Second, a fusion module introduces a trainable [QA] token and prompt-guided attention pooling over visual tokens. The [QA] token becomes a quality-focused summary of the prompt, and its embedding is used as a query over visual tokens to produce a fused representation for regression. On AGIQA-1k, IP-IQA reaches SRCC 0.8401, PLCC 0.8922, and KRCC 0.6635; on AGIQA-3k, it reaches SRCC 0.8634, PLCC 0.9116, and KRCC 0.6844. On AGIQA-3k alignment MOS, it reaches SRCC 0.7578, PLCC 0.8544, and KRCC 0.5734, slightly surpassing StairReward on PLCC while improving SRCC and KRCC.

Prompt Performance Prediction generalizes the evaluation perspective further by asking whether prompt quality can be predicted before any image is generated (Bizzozzero et al., 2023). The task defines prompt performance as

$Z$4

where $Z$5 is an image-level score such as aesthetics, memorability, or compositionality, and then learns a regressor $Z$6 from prompt text alone. Across DALL·E 2, Midjourney, and Stable Diffusion datasets, CLIP text encoders outperform BART, GPT-2, BLOOM-560M, and Sentence-T5 variants, and the analysis shows a modality gap: image-based aesthetic predictors cannot simply be applied to prompt embeddings. The paper’s complementary painting experiment further shows that text-based predictors using description, painter, epoch, and valence can approach image-based prediction of human appreciation.

This evaluation-oriented branch shifts the notion of prompt-in-image from control to definition. The prompt becomes part of what the image is judged to be, not just a cause of how it was produced.

4. Optimization and control in the prompt–image loop

A third major branch treats prompt-in-image as a closed loop in which images reveal how prompts were interpreted, and that signal is then used to rewrite prompts. VisualPrompter is the clearest instance: it “treats the image produced by a text-to-image model as feedback about how the model interpreted the prompt” and operationalizes this with SERE and TSPO (Wu et al., 29 Jun 2025). SERE converts a prompt into a Davidsonian Scene Graph, asks Qwen2-VL binary questions about the generated image, and computes a semantic score as the fraction of “yes” answers. Missing concepts become the targets for TSPO, which expands only the failed concepts and regenerates a refined prompt. The method is training-free, uses Qwen2 and Qwen2-VL, and achieves 84.3 on DSG-1k for Flux-dev versus a 79.1 baseline, 77.0 versus 72.1 on Stable Diffusion v2.1, and 93.8 versus 87.9 on TIFA for Flux-dev.

OPT2I pursues the same loop for consistency rather than generic alignment (Mañas et al., 2024). It treats prompt rewriting as an optimization problem over paraphrases, uses an LLM to propose revised prompts, and scores them with decomposed CLIPScore or Davidsonian Scene Graph metrics computed against the original user prompt. The reported outcome is a boost of up to 24.9% in terms of DSG score while preserving FID and increasing recall between generated and real data.

Promptist and NeuroPrompts represent reward-specific prompt optimization for aesthetics or preference (Hao et al., 2022, Rosenman et al., 2023). Promptist learns a GPT-2–based mapping from user-style prompts to model-preferred prompts using supervised fine-tuning and PPO with a reward that combines aesthetic improvement, relevance, and KL regularization. On DiffusionDB, it reaches aesthetic 6.26 versus 5.47 for user input and 5.87 for human engineered prompts. NeuroPrompts uses a GPT-2 LLM trained on 600k DiffusionDB prompts, PPO with PickScore, and NeuroLogic constrained decoding over style, artist, format, booster, vibe, and perspective categories; its full system reaches aesthetic score 6.27 versus 5.64 for the original prefix and 5.92 for the original human prompt.

PRISM and RATTPO move toward black-box and reward-agnostic control (He et al., 2024, Kim et al., 20 Jun 2025). PRISM starts from reference images rather than text alone, uses GPT-4V both as prompt engineer and judge, and iteratively searches for natural-language prompts that transfer across SD 2.1, SDXL-Turbo, DALL·E 2, and DALL·E 3. RATTPO, by contrast, assumes an arbitrary reward model specified at test time and performs optimization with two LLMs: an optimizer and a hint generator. The optimizer never receives reward-specific task descriptions; it sees only prompt histories, scores, and a reward-aware “hint.” The method is reported to surpass other test-time search baselines in search efficiency, using up to 3.5 times less inference budget, and to become comparable to reward-specific learning-based baselines when the budget is sufficiently large.

Prompt Expansion and PromptMoG emphasize diversity rather than single-prompt optimization (Datta et al., 2023, Ruan et al., 25 Nov 2025). Prompt Expansion maps a short query to a set of expanded prompts and reports MUSIQ 6.185 ± 0.474 and diversity 0.00746 ± 0.00354 for the re-fine-tuned model, versus 5.121 ± 0.519 and 0.00582 ± 0.00275 for straight-query generation. PromptMoG studies long prompts and shows that Vendi Score drops as prompt length increases across SD3.5-Large, Flux.1-Krea-Dev, CogView4, and Qwen-Image. It then samples prompt embeddings from a Mixture-of-Gaussians around the original embedding, yielding consistent diversity gains without semantic drifting.

A concise view of this optimization literature is useful:

Method	Mechanism	Reported outcome
Promptist	GPT-2 + SFT + PPO for model-preferred prompts	aesthetic 6.26 vs 5.47 user input (Hao et al., 2022)
VisualPrompter	SERE + TSPO with Qwen2 and Qwen2-VL	Flux-dev DSG 84.3 vs 79.1 baseline (Wu et al., 29 Jun 2025)
OPT2I	iterative LLM paraphrase search with DSG/dCS scoring	up to 24.9% DSG boost (Mañas et al., 2024)
RATTPO	reward-agnostic test-time prompt optimization with hints	up to 3.5 times less inference budget (Kim et al., 20 Jun 2025)
PromptMoG	prompt embedding Mixture-of-Gaussians	improves long-prompt generation diversity without semantic drifting (Ruan et al., 25 Nov 2025)

5. Interactive systems and prompt-space exploration

Interactive systems make prompt-in-image legible to human users by externalizing prompt histories, example spaces, and prompt–image relations. Promptify frames prompt creation as interactive prompt exploration with GPT-3.5-Turbo, Stable Diffusion v1.5, Sentence-BERT retrieval over DiffusionDB, CLIP embeddings for layout, and CLIP Interrogator for modifier extraction (Brade et al., 2023). In a within-subject study against Automatic1111, Promptify was rated more useful overall, reduced mental demand and frustration, and made it easier to compare and ignore images. Initial prompts constructed with Promptify were longer after stopword removal—34.43 words versus 8.6 words—and first-attempt images were rated significantly more aesthetically pleasing.

PromptCrafter decomposes prompt engineering into mixed-initiative dialogue with GPT-3 and DALL·E2 (Baek et al., 2023). Its design goals are to decompose prompt completion into smaller steps, provide examples and multiple results that can be compared simultaneously, and visualize the workflow so users can branch or revert. The interface centers on clarifying questions, multiple candidate answers, six generated images per answer, and a history panel that supports reversion to earlier states.

PromptMap changes the interaction style more radically by replacing the blank prompt box with a semantic map of prompt–image pairs (Adamkiewicz et al., 12 Mar 2025). It constructs a synthetic corpus of approximately 12.4M prompts and generated images, embeds main-subject captions with Sentence-BERT, lays them out with UMAP, and uses semantic zoom plus manually placed labels to expose topic regions. The between-subject online study with $Z$7 and the qualitative within-subject study with $Z$8 did not find significant differences on CSI, NASA-TLX, UMUX-Lite, or satisfaction scores, but the qualitative analysis showed a shift from trial-and-error prompting toward example-driven prompting, and participants perceived the synthetic prompt set as more varied and useful than nearest-neighbor search over DiffusionDB.

PrompTHis addresses the analysis of prompt editing itself (Guo et al., 2024). Its Image Variant Graph embeds prompt–image pairs in a joint CLIP-based space, models prompt differences as edges, and assigns edge weights through Jaccard-thresholded prompt comparisons, token-level edit detection, and cluster-based bundling. The quantitative study reports 82.21% accuracy on identifying exploration stages, 96.92% on identifying key words that led to differences between clusters, and 78.12% on summarizing how the model responds to given words. Qualitative interviews indicate that users employed the graph to identify “magic words,” diagnose unstable prompts, and plan new prompt branches.

These systems collectively show that prompt-in-image is not only a modeling concept but also an interface problem. Prompt histories, clusters, maps, and graphs transform prompt engineering from opaque trial-and-error into a visually analyzable search over prompt-conditioned image manifolds.

6. Limits, trade-offs, and open directions

Across the literature, prompt-in-image methods inherit the limitations of the representations and evaluators they rely on. In restoration, PIP improves universal handling of known degradations but remains limited on completely unknown degradations and arbitrary combinations, and its design is tied to U-shaped backbones with skip connections (Li et al., 2023). Prompt-SID adds a latent diffusion branch and scale replay mechanism, which preserve structural information, but the framework introduces extra complexity and depends on hyperparameters such as the diffusion schedule and loss weights (Li et al., 10 Feb 2025).

In multimodal assessment, IP-IQA mainly uses global alignment and prompt-guided attention over visual tokens; it does not explicitly model compositional structures, long narrative prompts, or richer edit instructions (Qu et al., 2024). Prompt Performance Prediction shows that prompt-only prediction is plausible, but it also documents a modality gap between prompt and image embeddings, which prevents simple reuse of image-side predictors on text inputs (Bizzozzero et al., 2023).

For optimization systems, the main trade-offs are compute, evaluator reliability, and model specificity. VisualPrompter depends on Qwen2 and Qwen2-VL and is exposed to LLM/VLM hallucination, prompt-length constraints, and inference latency from multiple LLM and VLM calls (Wu et al., 29 Jun 2025). OPT2I can substantially improve consistency, but it inherits the weaknesses of decomposed CLIPScore and DSG as optimization targets (Mañas et al., 2024). RATTPO relies on strong in-context learning and a “sane” reward signal, and its cost remains far above one-shot generation even though it is more sample-efficient than brute-force search (Kim et al., 20 Jun 2025). PromptMoG is explicitly sensitive to $Z$9, $\omega$0, and $\omega$1; its diversity gains are bounded because strong semantic preservation is enforced, and the method is presently validated on rectified-flow systems rather than autoregressive generators (Ruan et al., 25 Nov 2025).

A broader implication is that prompt-in-image is increasingly less about prompt text alone and more about prompt geometry, prompt histories, and prompt-conditioned internal states. Internal prompts in restoration, [QA] tokens in AGI assessment, DSG question sets for semantic reflection, graph edges that link word edits to image transitions, and Mixture-of-Gaussians over prompt embeddings all point in the same direction: prompt information is being redistributed across feature spaces, latent spaces, evaluators, and interfaces. This suggests that future work will likely focus less on isolated prompt strings and more on structured prompt representations that are jointly optimized with image models, reward models, and user workflows (Li et al., 2023, Qu et al., 2024, Wu et al., 29 Jun 2025, Guo et al., 2024, Ruan et al., 25 Nov 2025).