Scale-Aware Prompt Encoder

• The paper proposes a scale-aware prompt encoder that dynamically selects optimal guidance scales by fusing prompt embeddings, complexity metrics, and scale variables.
• It leverages multi-scale information from visual, textual, and budgeted features to enhance performance in diffusion, segmentation, and federated learning tasks.
• Empirical results show improved diffusion quality, better generalization in federated settings, and efficient token compression across diverse benchmarks.

A scale-aware prompt encoder is a prompt-processing module that makes prompt-conditioned decisions depend on an explicit notion of scale. In current arXiv usage, that notion of scale appears in several forms: the classifier-free guidance scale in diffusion sampling, the low/mid/high-level depth of visual features, the coarse-to-fine semantic granularity of textual descriptions, and the compression budget of long-context prompts. The most explicit formulation treats the prompt encoder as a mapping $$p \mapsto \arg\max_{\omega \in \mathcal{S}} U(p,\omega)$$, where a lightweight predictor estimates prompt-dependent quality over candidate guidance scales and selects the optimum for each prompt (Zhang et al., 25 Sep 2025). Related work realizes analogous ideas by fusing multi-scale visual cues into prompt tokens (Prasad et al., 17 Aug 2025), organizing text prompts from global to local semantic scales (Shan et al., 2 Apr 2025), and ranking sentence-level prompt units under explicit length constraints (Liskavets et al., 2024).

1. Conceptual scope

Within diffusion models, a scale-aware prompt encoder is defined most concretely as a prompt-aware, scale-aware network $g_\phi$ that receives a prompt $p$ and a candidate guidance scale $\omega$, predicts a multi-metric quality vector $\widehat{\mathbf{q}(p,\omega)}$, and chooses the prompt-specific optimum through a regularized utility. The prompt representation combines a semantic embedding $\mathbf{e}(p)$, a linguistic complexity embedding $\mathbf{c}(p)$, and the scale itself, yielding $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$ (Zhang et al., 25 Sep 2025).

Across other domains, scale is encoded differently but remains explicit. In federated vision-language prompt generation, scale denotes low, mid, and high-level visual features extracted from frozen CLIP layers and fused with client-specific style statistics before prompt-token injection (Prasad et al., 17 Aug 2025). In medical segmentation, scale denotes either the last four transformer layers of a MedSAM backbone used as multi-scale visual features or the progressive semantic enrichment of text prompts from global to local clinical descriptions (Zhang et al., 18 Mar 2025, Shan et al., 2 Apr 2025). In prompt compression, scale denotes the granularity of selection or the target retention fraction: sentence-level selection in CPC and budget-controlled token retention in TACO-RL (Liskavets et al., 2024, Shandilya et al., 2024). This suggests that the term does not identify a single architecture; rather, it identifies a family of prompt encoders that expose scale as an input variable, a hierarchy, or a budgeted control dimension.

2. Canonical formulation in diffusion guidance

The diffusion formulation begins from classifier-free guidance (CFG), where the guided noise estimate is

$$\hat{\epsilon}_\theta(x_t, p, t; \omega) = (1-\omega)\,\epsilon_\theta(x_t, \emptyset, t) + \omega\,\epsilon_\theta(x_t, p, t).$$

The central claim is that a fixed guidance scale is suboptimal because simple prompts and long compositional prompts exhibit different optima: high scales can cause oversaturation and artifacts, while low scales can weaken alignment. The proposed encoder therefore models the prompt-scale pair directly (Zhang et al., 25 Sep 2025).

The predictor is defined by

$$\mathbf{e}(p) = E_{\text{text}}(p), \qquad \mathbf{c}(p) = W_c\,\mathbf{r}(p)+\mathbf{b}_c,$$

followed by

$g_\phi$0

For images, $g_\phi$1 is CLIP’s text encoder; for audio, it is CLAP’s text encoder. The raw complexity statistics $g_\phi$2 include examples such as prompt length and entropy. The selected scale is

$g_\phi$3

This utility combines predicted quality with quadratic regularization around an anchor scale $g_\phi$4, so the encoder becomes a prompt-to-scale controller rather than a prompt-to-text embedding alone (Zhang et al., 25 Sep 2025).

Training supervision is synthetic. Samples are generated across multiple scales with frozen backbones—SDXL 1.0 for images and AudioLDM2 for audio—using prompts from MSCOCO 2014 and AudioCaps. For each $g_\phi$5, multiple generations are scored and averaged into oracle labels $g_\phi$6. Image supervision uses KID, CLIP score, ImageReward, and Precision/Recall; audio supervision uses AudioBox-Aesthetics sub-scores CE, CU, PC, and PQ. The predictor is optimized with mean squared error, while the backbone remains frozen, which is why the method is described as a training-free enhancement with respect to the pretrained diffusion model (Zhang et al., 25 Sep 2025).

3. Architectural patterns across domains

The idea of scale-aware prompt encoding appears in several recurring architectural forms: explicit scale variables, multi-scale feature fusion, hierarchical prompt semantics, and budget-conditioned selection. The following representative systems instantiate those forms differently.

Work	Scale notion	Prompt-encoder mechanism
(Zhang et al., 25 Sep 2025)	CFG guidance scale $g_\phi$7	Prompt embedding + complexity + scale $g_\phi$8 quality-curve predictor
(Prasad et al., 17 Aug 2025)	Low/mid/high visual depth + style	Cross-attention text prompts + SE-style injection block
(Shan et al., 2 Apr 2025)	Global-to-local textual semantics	Retrieved text prompts progressively averaged across four semantic levels

A closely related prompt-learning formulation is PRE, which reparameterizes learnable prompt tokens through a shared residual encoder,

$g_\phi$9

with a one-layer BiLSTM as the default choice. PRE is not presented as a scale-aware method, but it establishes an important prompt-encoder pattern: prompts need not be optimized as free vectors and can instead be passed through a lightweight shared encoder that models inter-token dependencies while preserving pretrained semantics through a residual path (Pham et al., 2023). This is relevant because many scale-aware systems also use small auxiliary encoders rather than modifying the full pretrained backbone.

In federated vision-language learning, FedCSAP makes scale explicit by extracting $p$0 from multiple CLIP layers and concatenating them into a multi-scale content vector $p$1. Batch-wise style means $p$2 are appended to form $p$3, which is then passed through an SE-style Injection Block: $p$4 The resulting descriptor is projected into visual tokens $p$5, while a cross-attention prompt generator produces context tokens $p$6, and the final prompts are formed by $p$7. Here the prompt encoder is scale-aware because different feature depths and style statistics directly modulate prompt-token generation (Prasad et al., 17 Aug 2025).

Medical segmentation offers two further variants. OMT-SAM adds a CLIP-based image-text prompt encoder to MedSAM and exposes the last four transformer layers as multi-scale visual features before fusion in the mask decoder (Zhang et al., 18 Mar 2025). STPNet uses a discrete text repository with four semantic levels—Infection, Num, Left Loc, and Right Loc—and defines hierarchical prompts by

$p$8

These prompts are injected into scale-matched encoder and transformer blocks, coupling semantic scale in language with spatial scale in the segmentation hierarchy (Shan et al., 2 Apr 2025).

A precursor outside prompt learning is the scale-aware encoder in crowd counting, where ASPP provides static multi-scale context and CAN provides dynamic per-location scale attention. The paper explicitly frames CAN as adaptive encoding of contextual scales and distinguishes it from ASPP’s globally shared multi-scale fusion. This distinction—static multi-scale fusion versus dynamic scale selection—has become useful for understanding later prompt encoders as well (Thanasutives et al., 2020).

4. Objectives, supervision, and inference policies

Scale-aware prompt encoders are supervised according to the scale variable they model. In diffusion guidance, the target is a multi-metric quality curve over prompt-scale pairs, and the loss is a standard multi-task regression objective,

$p$9

Inference is then utility maximization over a discrete scale set $\omega$0 (Zhang et al., 25 Sep 2025).

In prompt learning for CLIP, PRE uses standard vision-language cross-entropy on Base classes while freezing CLIP and updating only the prompt vectors and the reparameterization encoder. Its regularization is architectural rather than explicit: BiLSTM sequence modeling, residual connections, and parameter sharing constrain the prompt space and improve base-to-new transfer (Pham et al., 2023).

In federated prompt generation, FedCSAP optimizes a CLIP-style cosine-similarity classification loss together with a Context Redundancy Penalizing loss,

$\omega$1

where $\omega$2 encourages different prompt tokens to be distinct and non-redundant. Training proceeds with local client updates and global aggregation through FedAvg, while CLIP’s image and text encoders remain frozen (Prasad et al., 17 Aug 2025).

In medical segmentation, OMT-SAM uses a combined Dice and BCE objective for final masks, while STPNet adds retrieval and focal losses to segmentation loss: $\omega$3 with $\omega$4. STPNet’s distinctive feature is that text is used only during training; at inference, the segmentation network runs without text input while retaining the effects of cross-modal training in its parameters (Shan et al., 2 Apr 2025).

Compression-oriented prompt encoders use budget-aware policies. TACO-RL formulates token retention as a Bernoulli policy with reward

$\omega$5

where $\omega$6. The policy is trained with REINFORCE and entropy regularization, and exact compression is enforced at inference by selecting the top $\omega$7 tokens by probability (Shandilya et al., 2024). CPC instead learns context-aware sentence embeddings with a contrastive loss $\omega$8 and a masked next token prediction auxiliary loss $\omega$9, then ranks whole sentences by cosine similarity to the question embedding under a token budget (Liskavets et al., 2024). These two systems show that scale-awareness can be budget-driven rather than geometry-driven.

5. Empirical behavior and application domains

The diffusion case provides a direct demonstration of prompt-specific scale selection. On MSCOCO with SDXL, prompt-aware CFG improves over non-adaptive CFG from FID $\widehat{\mathbf{q}(p,\omega)}$0 and CLIP $\widehat{\mathbf{q}(p,\omega)}$1 to FID $\widehat{\mathbf{q}(p,\omega)}$2 and CLIP $\widehat{\mathbf{q}(p,\omega)}$3; on AudioCaps with AudioLDM2, prompt-aware guidance raises CE from $\widehat{\mathbf{q}(p,\omega)}$4 to $\widehat{\mathbf{q}(p,\omega)}$5, PC from $\widehat{\mathbf{q}(p,\omega)}$6 to $\widehat{\mathbf{q}(p,\omega)}$7, and PQ from $\widehat{\mathbf{q}(p,\omega)}$8 to $\widehat{\mathbf{q}(p,\omega)}$9. The ablation also shows that using only KID and CLIP for supervision can worsen FID to $\mathbf{e}(p)$0, whereas “All metrics” yields the best reported FID and CLIP combination (Zhang et al., 25 Sep 2025).

Prompt reparameterization in PRE demonstrates that a lightweight prompt encoder can improve generalization even without an explicit scale variable. On eight datasets in the 16-shot setting, PRE reports average New accuracy $\mathbf{e}(p)$1 and Harmonic mean $\mathbf{e}(p)$2, compared with CoOp’s $\mathbf{e}(p)$3 and $\mathbf{e}(p)$4. The default BiLSTM with residual connection yields the best Harmonic mean among the reported encoder variants, while maintaining training time near CoOp at $\mathbf{e}(p)$5 ms per image (Pham et al., 2023).

FedCSAP shows that prompt tokens enriched with multi-scale visual cues and style statistics improve federated generalization. Averaged over nine datasets, FedCSAP reports Harmonic Mean $\mathbf{e}(p)$6, compared with $\mathbf{e}(p)$7 for FedCoOp and $\mathbf{e}(p)$8 for FedMaple. The paper further states that FedCSAP outperforms FedCoOp by $\mathbf{e}(p)$9 on unseen classes and $\mathbf{c}(p)$0 on unseen datasets on average, and its ablations show an $\mathbf{c}(p)$1 average-accuracy drop when the Injection Block is removed and a $\mathbf{c}(p)$2 drop when the prompt generator is removed (Prasad et al., 17 Aug 2025).

In medical segmentation, OMT-SAM reports mean Dice Similarity Coefficient $\mathbf{c}(p)$3 on FLARE 2021, outperforming MedSAM’s $\mathbf{c}(p)$4, with especially large pancreas gains ($\mathbf{c}(p)$5 versus $\mathbf{c}(p)$6). STPNet reports Dice/IoU $\mathbf{c}(p)$7 on COVID-CT, $\mathbf{c}(p)$8 on COVID-Xray, and $\mathbf{c}(p)$9 on Kvasir-SEG, while requiring no text input at inference. Its ablations show progressively better Dice as additional semantic text scales are added, culminating in the best performance when Infection, Num, and bilateral localization descriptions are all used (Zhang et al., 18 Mar 2025, Shan et al., 2 Apr 2025).

Prompt compression studies show a different but consistent advantage for scale-aware encoding. CPC reports average LongBench score $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$0 at a 3,000-token budget and $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$1 at a 2,000-token budget, outperforming LongLLMLingua at both budgets, and it is reported to be up to $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$2 faster at inference than the best token-level compression method (Liskavets et al., 2024). TACO-RL reports improvements of $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$3 over state-of-the-art compression techniques across summarization, question answering, and code summarization, including a $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$4 BLEU gain over LLMLingua-2 at 6$$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$5 compression on MeetingBank and large F1/EM gains on SQuAD 2.0 under severe compression (Shandilya et al., 2024). Taken together, these results indicate that scale-aware prompt encoders are effective in generation, classification, segmentation, federated learning, and long-context inference.

6. Limitations, misconceptions, and open directions

A common misconception is that a scale-aware prompt encoder must be a heavy prompt-tuning stack. The literature shows otherwise: the diffusion predictor is described as a lightweight agent with $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$6M parameters, PRE adds a small reparameterization encoder on top of frozen CLIP, and FedCSAP trains only prompt modules while leaving CLIP frozen (Zhang et al., 25 Sep 2025, Pham et al., 2023, Prasad et al., 17 Aug 2025). Another misconception is that scale-awareness is synonymous with spatial pyramids. The surveyed systems use spatial scale, semantic scale, guidance scale, and budget scale, sometimes without any spatial pyramid at all (Shan et al., 2 Apr 2025, Liskavets et al., 2024).

The current limitations are equally heterogeneous. Diffusion-based scale selection depends on a discrete candidate set $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$7 and a quadratic anchor prior, so out-of-distribution prompts may default toward $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$8 rather than discovering novel optima (Zhang et al., 25 Sep 2025). FedCSAP discards spatial structure through global average pooling and models style only with batch means, which the paper identifies indirectly by using simple style descriptors and global vectors rather than token-wise alignment (Prasad et al., 17 Aug 2025). OMT-SAM inherits a CLIP domain gap because CLIP is trained on natural image-text data while the target domain is CT (Zhang et al., 18 Mar 2025). CPC is sentence-level only, so it cannot refine within long partially relevant sentences (Liskavets et al., 2024). TACO-RL incurs expensive RL training because each update requires downstream LLM calls and is sensitive to reward design (Shandilya et al., 2024). The crowd-counting precursor explicitly notes fixed dilation and pooling rates as a limitation of hand-selected scale hyperparameters (Thanasutives et al., 2020).

Several directions recur across the literature. The diffusion paper proposes continuous scale modeling, distributions over scales, and joint hyperparameter tuning beyond CFG scale (Zhang et al., 25 Sep 2025). The crowd-counting work points toward learnable scales and dynamic dilation (Thanasutives et al., 2020). TACO-RL suggests explicit conditioning on the target compression rate $$\mathbf{h}(p,\omega) = [\mathbf{e}(p);\mathbf{c}(p);\omega]$$9 and task type $$\hat{\epsilon}_\theta(x_t, p, t; \omega) = (1-\omega)\,\epsilon_\theta(x_t, \emptyset, t) + \omega\,\epsilon_\theta(x_t, p, t).$$0, while CPC suggests extending from sentence-only decisions to more general multi-granularity selection (Shandilya et al., 2024, Liskavets et al., 2024). Medical and federated systems suggest richer token-wise cross-modal alignment and more expressive style descriptors than simple batch statistics or pooled global vectors (Zhang et al., 18 Mar 2025, Prasad et al., 17 Aug 2025). This suggests that the next generation of scale-aware prompt encoders will likely be hierarchical, budget-conditioned, and explicitly multi-granular rather than tied to a single notion of scale.