Prompt-Aware Scale Selection

• Prompt-Aware Scale Selection is a framework that adaptively selects scales, granularity, or operating regimes based on prompt-specific features like length, complexity, and task type.
• It is applied across diffusion model guidance, text-to-image approximation, prompt optimization, and model routing to replace static global settings with tailored, prompt-conditioned decisions.
• Empirical results show measurable improvements in metrics such as FID, CLIP score, and evaluation stability, underscoring benefits in fidelity, efficiency, and system robustness.

Prompt-aware scale selection denotes a family of prompt-conditioned decision procedures that replace a single global operating point with a per-prompt choice of scale, granularity, or operating regime. In recent work, the phrase has been used for selecting classifier-free guidance strength in diffusion models, approximation levels in text-to-image serving, latent quality scales in unsupervised prompt optimization, model choice in prompt routing, hierarchical compression granularity in long-context inference, and evaluation subset scale in automatic prompt optimization. Across these settings, the common claim is that prompt length, complexity, semantic content, task type, or evaluation uncertainty can make a fixed global choice intrinsically suboptimal (Zhang et al., 25 Sep 2025).

1. Conceptual scope and problem formulation

In diffusion sampling, prompt-aware scale selection means choosing the classifier-free guidance strength separately for each prompt rather than using a single global value. In text-to-image systems based on Approximate Caching, it means predicting the optimal approximation level $K$ for each incoming prompt and routing the request accordingly. In prompt optimization, it can mean constructing a latent quality scale over candidate prompts from local pairwise preferences, or selecting prompts with uncertainty-aware criteria rather than raw mean scores. In prompt routing, it refers to assigning each prompt to a model from a large pool using task-aware quality estimates. In long-context compression, it denotes query-conditioned selection of token-, sentence-, or page-level representations. In evaluation scheduling, it means selecting which examples to evaluate on, and how many, as a function of the currently strongest prompts (Peng et al., 30 Jan 2026).

The negative case is also consistent across these lines of work. A fixed guidance scale can oversaturate simple prompts and under-condition complex ones; a prompt-agnostic approximation policy can degrade sensitive prompts; a single monolithic router can blur subtle task-model affinities; token-only compression can fragment discourse; and prompt rankings can appear globally stable while the identity of the top prompt changes across seeds and evaluation subsets. The shared motivation is therefore not merely efficiency, but the preservation of fidelity, alignment, robustness, or discriminative power under prompt heterogeneity (Agarwal et al., 29 Jan 2025).

A plausible implication is that “scale” is best understood as an abstract control variable rather than a single modality-specific hyperparameter. Depending on the system, that variable may be a scalar guidance coefficient, a discrete approximation level, a latent Bradley-Terry-Luce quality parameter, a model index, a page size and page subset, or an evaluation subset of size $k$.

2. Diffusion guidance and prompt-conditioned generative control

The most explicit formulation appears in prompt-aware classifier-free guidance for diffusion models. Standard classifier-free guidance combines unconditional and conditional denoisers as

$$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$

so the guidance scale $\omega$ directly controls how strongly sampling is pulled toward the prompt. The prompt-aware variant learns a lightweight predictor conditioned on semantic embeddings and linguistic complexity, estimates multi-metric quality curves over candidate $\omega$, and selects

$$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$

The synthetic supervision uses samples generated at multiple scales and metrics including KID, CLIP score, ImageReward, and Precision / Recall for images, and AudioBox-Aesthetics for audio. On SDXL with MSCOCO 2014, prompt-aware guidance improved FID from $31.04$ to $30.74$ and CLIP from $0.31$ to $0.33$; on AudioLDM2 with AudioCaps it improved CE from $k$0 to $k$1, PC from $k$2 to $k$3, and PQ from $k$4 to $k$5. The predictor is described as a small neural network of about $k$6M parameters, and the method is presented as training-free with respect to the pretrained diffusion backbone (Zhang et al., 25 Sep 2025).

This formulation makes prompt dependence explicit at several levels. The predictor uses CLIP or CLAP text embeddings together with handcrafted features such as prompt length, entropy or perplexity, modifier diversity, lexical statistics, object counts, and the presence of compositional phrases. The reported qualitative behavior is that short prompts such as “A white car” often benefit from lower $k$7, while long compositional prompts benefit from higher $k$8. The ablation that uses only KID and CLIP supervision yields worse FID than the fixed baseline, whereas using all metrics produces the best result, supporting the claim that the operating scale is tied to a multi-objective trade-off rather than a single alignment score (Zhang et al., 25 Sep 2025).

A second generative interpretation appears in SPARKE, where the relevant scale is not only the classifier-free guidance scale $k$9 but also a diversity-guidance scale $$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$0. SPARKE replaces matrix-based conditional entropy guidance with Conditional Inverse-RKE in latent space, reducing entropy-related computation from $$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$1 to $$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$2 for the per-new-sample gradient. Its prompt-aware gradient weights latent interactions by prompt similarity through a prompt kernel $$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$3, so diversity pressure is concentrated among semantically similar prompts rather than spread uniformly across all prompts. The sampling update is of the form

$$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$4

with $$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$5 itself built from CFG. SPARKE reports improvements in prompt-aware diversity metrics such as Conditional-Vendi while maintaining CLIPScore and KID near baseline levels, and describes $$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$6 as a practical trade-off point for SDXL and SD 2.1 (Jalali et al., 11 Jun 2025).

Taken together, these works define prompt-aware scale selection in generative modeling as inference-time control of prompt alignment, fidelity, and diversity through prompt-conditioned choice of one or more guidance strengths. One notable limitation, stated directly in the literature, is dependence on proxy metrics such as CLIP, KID, ImageReward, AudioBox, Vendi, and Cond-Vendi, which may not perfectly match human preference or prompt-specific notions of quality (Zhang et al., 25 Sep 2025).

3. Latent prompt quality, ranking stability, and uncertainty-aware selection

In unsupervised prompt optimization, prompt-aware scale selection refers to the construction of a global latent quality scale from local pairwise prompt comparisons. UPA frames prompt optimization as sequential decision-making over a tree of prompt refinements. Because the available supervision is a judge LLM that returns local, fine-grained, order-invariant pairwise preferences rather than absolute scores, UPA separates search from scale construction. It first performs path-wise Bayesian aggregation over parent-child comparisons using Beta posteriors and BTL log-odds, with

$$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$7

to filter candidates; it then runs a global round-robin tournament on the top-$$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$8 prompts and fits a Bradley-Terry-Luce model

$$\hat{\epsilon}_\theta(x_t,p,t;\omega) = \epsilon_\theta(x_t,\emptyset,t) + \omega\left(\epsilon_\theta(x_t,p,t)-\epsilon_\theta(x_t,\emptyset,t)\right),$$9

by MM updates to recover latent quality parameters $\omega$0. On five closed-ended benchmarks, UPA reports average accuracy $\omega$1, compared with about $\omega$2–$\omega$3 for the strongest baselines, and its selection ablations show $\omega$4 without LCB, $\omega$5 without Stage II, $\omega$6 for Max $\omega$7, and $\omega$8 for Max local win rate (Peng et al., 30 Jan 2026).

The stability literature approaches the same problem from another angle: a scale of prompt quality exists only insofar as it is stable under realistic evaluation variation. Under evaluation conditions $\omega$9, prompt $\omega$0 receives a score $\omega$1, inducing a ranking $\omega$2. Empirically, rank correlations can be moderate to high while top-1 identity remains unstable. For example, Phi on GSM8K at $\omega$3 examples reaches $\omega$4 and $\omega$5, yet top-1 consistency can be as low as $\omega$6 in other settings such as Phi–MMLU–50, with five distinct top prompts across five seeds. To address this, the paper proposes lower-confidence-bound selection,

$\omega$7

instead of mean-only selection. In leave-one-seed-out evaluation, Qwen–GSM8K–50 improves from $\omega$8 under mean selection to $\omega$9 under LCB, while more stable settings remain competitive (Du et al., 23 Jun 2026).

A third robustness-based scale is prompt flatness. For a fixed pretrained model $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$0, the robust prompt-selection objective is

$$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$1

with first-order approximation

$$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$2

The paper introduces the label-free surrogate $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$3, based on prediction changes under Gaussian parameter perturbations, and shows that combining prompt flatness with MI or Sen improves prompt selection by an average increase of $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$4 in accuracy and $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$5 in Pearson correlation across six classification benchmarks. It also reports that the gain from adding pFlat increases as model size grows from GPT-2 base to GPT-2 xl, suggesting that prompt-quality scales become more sensitive to robustness as parameter count increases (Shen et al., 2023).

These three lines of work make different assumptions, but they converge on the same principle: prompt-aware selection is not only a matter of estimating expected performance, but of defining a scale that remains meaningful under noise, local comparisons, or parameter perturbations.

4. Systems, routing, and approximation-level selection

In high-load text-to-image serving, prompt-aware scale selection appears as per-prompt choice of approximation level. The SD-XL scheduling system based on Approximate Caching defines a discrete set of scales indexed by $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$6, where $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$7 is the full model and larger $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$8 skips more denoising steps. Each prompt is assigned an optimal $$\omega^\star(p)=\arg\max_{\omega\in\mathcal{S}} \left[ \mathbf{w}^\top \widehat{\mathbf{q}}(p,\omega)-\alpha(\omega-\mu_\omega)^2 \right].$$9 by nearest-cache retrieval, and the controller predicts the workload distribution

$31.04$0

together with query fractions $31.04$1 and a redirection plan $31.04$2. The optimization target is the expected quality degradation

$31.04$3

The reported system runs SD-XL on 8 NVIDIA A100 GPUs, avoids model switching by keeping the same base model resident on all workers, and reports consistent quality $31.04$4, lowest SLO violations $31.04$5, up to $31.04$6 higher quality, $31.04$7 higher throughput, and up to $31.04$8 fewer SLO violations than the baselines named in the poster text (Agarwal et al., 29 Jan 2025).

Prompt routing generalizes the same idea from approximation levels to model pools. FineRouter formulates routing as

$31.04$9

but replaces monolithic prediction with a two-stage architecture. Stage 1 discovers latent task types by graph-based clustering over LLM-generated task descriptions and reward-model-induced model rankings; Stage 2 uses a mixture-of-experts quality estimator with general per-model adapters and task-specific adapters. Final routing combines task-level priors and prompt-level estimates through

$30.74$0

On about $30.74$1k training prompts, FineRouter discovers $30.74$2 latent task types, reduces the effective candidate set to an average of $30.74$3 models per task, and reports average quality $30.74$4, compared with $30.74$5 for IPR and $30.74$6 for the strongest single model, Claude Sonnet. It also reports that it achieves better quality than Claude Sonnet while using less than half its cost (Zhang et al., 19 Mar 2026).

Both systems treat prompt-conditioned scale selection as a resource-allocation problem under external constraints. In one case the scale is denoising-step approximation level $30.74$7; in the other it is model capability and cost. In both, the prompt is first mapped to a latent requirement signal—optimal $30.74$8 or latent task type—and that signal is then reconciled with global system constraints.

5. Hierarchical compression and prompt-aware evaluation budgets

Long-context compression introduces a structural notion of scale. BEAVER explicitly describes itself as a query-conditioned choice among token, sentence, and page scales. It segments context into logical spans, packs them into fixed-capacity pages with default $30.74$9, constructs dense page tensors, and forms page representations by fusing ITF-weighted average pooling with max pooling:

$0.31$0

Pages are then scored by a hybrid semantic and lexical planner,

$0.31$1

and selected under three structural priors: Anchors, Flow, and Flash. Sentence smoothing expands selected page spans to sentence boundaries before reconstruction. The method is training-free, reports a $0.31$2 latency reduction on $0.31$3k contexts, and on RULER reaches average $0.31$4 at $0.31$5k context and $0.31$6k budget, compared with $0.31$7 for LLMLingua-2 and $0.31$8 for LongLLMLingua (Hu et al., 20 Mar 2026).

A different form of scale selection appears in automatic prompt optimization, where the scale variable is the evaluation subset size $0.31$9 and the identity of the chosen examples. POES models APO as an online adaptive testing problem with prompts as examinees and examples as test items. At round $0.33$0, it selects a subset $0.33$1 of size $0.33$2 by maximizing

$0.33$3

where $0.33$4 is an additive IRT-based discrimination utility and

$0.33$5

is a facility-location coverage term. The composite objective is monotone submodular, giving a $0.33$6 greedy guarantee for cold starts; warm-start updates are controlled by swap budgets and switching thresholds, with bounded drift

$0.33$7

Across $0.33$8 tasks, POES reports the highest overall average accuracy, a $0.33$9 percent improvement over the best baseline, with approximately $k$00 percent token overhead at the same evaluation budget. It also reports that principled selection at $k$01 matches or exceeds naive evaluation at $k$02–$k$03, reducing token consumption by $k$04–$k$05 percent (Ma et al., 13 Apr 2026).

These two works use different objects—pages in one case, labeled examples in the other—but both treat scale as a prompt- or query-conditioned allocation of limited representational or evaluative capacity. The prompt determines not just content relevance, but the level at which a system should operate.

6. Recurring mechanisms, limitations, and open interpretations

Several mechanisms recur across the literature. First is explicit prompt representation: CLIP or CLAP embeddings with complexity features for CFG selection, nearest-cache retrieval for optimal $k$06, one-sentence task descriptions and task classifiers for routing, and query ITF together with page embeddings for BEAVER. Second is uncertainty control: UPA uses Bayesian path aggregation and LCB filtering; prompt-ranking stability work uses LCB over prompt means and variances; POES modulates swap thresholds and budgets based on optimization progress. Third is decomposition into local and global stages: UPA separates local search from global BTL tournaments, FineRouter separates latent-task discovery from task-aware quality estimation, and POES separates modular discrimination from facility-location coverage (Peng et al., 30 Jan 2026).

The limitations are correspondingly diverse but structurally similar. Prompt-aware CFG depends on proxy metrics such as CLIP, KID, ImageReward, Precision / Recall, and AudioBox, and its selected $k$07 is restricted to a discrete candidate set. SPARKE still requires tuning of $k$08 and kernel bandwidths, and large $k$09 can degrade visual quality. UPA depends on judge-LLM quality, assumes BTL and approximate path-error independence, and incurs $k$10 tournament cost. FineRouter requires offline task discovery and re-running parts of Stage 1 when the model pool changes. BEAVER uses fixed structural hyperparameters such as $k$11, $k$12, and $k$13, and page-level granularity can leave redundant tokens inside retained pages. POES relies on sparse early-round IRT estimates and is derived for binary correctness. Flatness-aware prompt selection requires repeated forward passes under parameter perturbation and was demonstrated on GPT-2 variants and classification tasks rather than broad instruction-tuned LLM settings (Zhang et al., 25 Sep 2025).

A recurrent misconception is that prompt-aware scale selection is a single algorithmic recipe. The evidence instead shows a methodological family. In one setting the selected object is $k$14; in another it is $k$15, a latent BTL score $k$16, a routed model $k$17, a page subset, or an evaluation subset $k$18. What makes these methods comparable is not the type of scale they manipulate, but the shared rejection of prompt-agnostic global defaults. This suggests that the central research question is not “what is the best scale,” but “which scale should be used for this prompt, under this objective, and under these constraints.”