Select Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guarantees

Abstract: Automatic prompt optimization (APO) hinges on the quality of its evaluation signal, yet scoring every prompt candidate on the full training set is prohibitively expensive. Existing methods either fix a single evaluation subset before optimization begins (principled but prompt-agnostic) or adapt it heuristically during optimization (flexible but unstable and lacking formal guarantees). We observe that APO naturally maps to an online adaptive testing problem: prompts are examinees, training examples are test items, and the scheduler should select items that best discriminate among the strongest candidates. This insight motivates Prompt-Aware Online Evaluation Scheduling (POES), which integrates an IRT-based discrimination utility, a facility-location coverage term, and switching-cost-aware warm-start swaps into a unified objective that is provably monotone submodular, yielding a (1-1/e) greedy guarantee for cold starts and bounded drift for warm-start updates. An adaptive controller modulates the exploration-exploitation balance based on optimization progress. Across 36 tasks spanning three benchmark families, POES achieves the highest overall average accuracy (6.2 percent improvement over the best baseline) with negligible token overhead (approximately 4 percent) at the same evaluation budget. Moreover, principled selection at k = 20 examples matches or exceeds the performance of naive evaluation at k = 30-50, reducing token consumption by 35-60 percent, showing that selecting smarter is more effective than selecting more. Our results demonstrate that evaluation scheduling is a first-class component of APO, not an implementation detail.

• The paper introduces POES, a framework that reformulates prompt optimization as a sequential adaptive testing problem with provable submodular guarantees.
• It leverages IRT-based discrimination, facility-location coverage, and adaptive control to robustly evaluate prompt candidates while reducing token overhead.
• Empirical results show POES outperforms baselines by up to +6.2% accuracy and saves 35–60% tokens, demonstrating strong stability across diverse LLM tasks.

Prompt-Aware Evaluation Scheduling for Automatic Prompt Optimization: An Expert Summary

Problem Setting and Limitations of Prior Approaches

Automatic prompt optimization (APO) for LLMs involves iteratively generating, evaluating, and selecting prompt candidates to maximize downstream task performance. Due to the high inference cost, existing APO pipelines typically evaluate each prompt on a small evaluation subset ($k \ll N$), but the methodology for selecting or updating this subset has remained ad hoc. Prior static methods, such as SESS, perform pre-optimization submodular selection, remaining oblivious to the evolving prompt population. Adaptive heuristic schedulers (e.g., IPOMP) modify subsets online but lack theoretical guarantees, often resulting in instability and sub-optimal optimizer feedback.

The POES Framework: Formulation and Algorithmic Design

The core insight is the mapping of APO to a sequential adaptive testing problem: prompts as examinees, training examples as test items, and the evaluation scheduler as an online item selector. The proposed Prompt-Aware Online Evaluation Scheduling (POES) framework formalizes this notion, explicitly integrating prompt-aware discrimination, coverage, and update stability into a provably monotone submodular objective $G_t(S)$:

$$G_t(S) = \sum_{i \in S} u_t(i) + \lambda_t \sum_{j \in V} \max_{i \in S} s_t(i, j)$$

where $u_t(i)$ is the discrimination utility based on an online 1PL IRT fit, $s_t(i, j)$ is TF-IDF-based similarity, and $\lambda_t$ is a coverage weight adaptively decayed as the optimization progresses.

The five algorithmic components (Figure 1) are:

1. Automatic random warmup to prevent use of unstable IRT estimates.
2. IRT-based per-example discrimination, maximizing separation between top-performing prompts.
3. Facility-location coverage, enforcing subset diversity to avoid collapse.
4. Warm-start, bounded swap optimization for stability (constraining per-round subset drift).
5. An adaptive controller for exploration–exploitation: tuning $\tau_t$, $B_t$, and $\lambda_t$ in response to optimization progress.

   Figure 1: POES framework components, with the scheduler integrating discrimination, coverage, warm-start swaps, and adaptive control for principled evaluation subset scheduling.

Theoretical analysis establishes $(1 - 1/e)$ greedy guarantees for cold starts, local optimality with bounded-drift for warm-start updates, and tight conditions for tracking the moving optimum as the APO landscape changes.

Empirical Evaluation and Numerical Findings

Across 36 tasks (BBH, BigBench, Math) and a $G_t(S)$0 optimizer-LLM matrix, POES consistently outperforms all baselines:

• Highest overall average accuracy: POES achieves $G_t(S)$1 improvement over the strongest baseline, with negligible token overhead ($G_t(S)$2).
• Token efficiency: At $G_t(S)$3, POES matches or outperforms naive random or static evaluation at $G_t(S)$4–$G_t(S)$5, leading to $G_t(S)$6–$G_t(S)$7 token savings without sacrificing performance (Figure 2).

  Figure 2: Optimization curves and accuracy–cost tradeoff on BBH Navigate. POES achieves sustained late-stage improvement and higher final accuracy with reduced token consumption.

• Robustness across optimizers/models: POES yields consistent gains irrespective of optimizer or LLM (Table: Cross-optimizer/model results), indicating scheduler-optimizer decoupling.
• Strong per-task results: POES attains highest or tied-highest test accuracy across all top-variance tasks (Figure 3).

  Figure 3: Per-task held-out accuracy (OPRO $G_t(S)$8 Llama-3.1-8B configuration). POES is dominant across most BBH, Math, and BigBench instances.

• Component ablations: Removal of adaptive control or auto warmup sharply degrades performance; coverage plays a critical role in diversity maintenance, and swap-based warm starts ensure stability and monotonic improvement (Figure 4).

  Figure 4: Ablation curves confirm adaptive control and stability mechanisms are essential for best performance.

• Mechanism interpretability: POES achieves controlled, interpretable subset evolution, with mean per-round drift of $G_t(S)$9 (on $$G_t(S) = \sum_{i \in S} u_t(i) + \lambda_t \sum_{j \in V} \max_{i \in S} s_t(i, j)$$0 subsets). The violin rank distribution shows sharp concentration around rank 1, validating reliable superiority (Figure 5).

  Figure 5: Violin plot of per-task scheduler ranking. POES achieves lowest median and concentration at rank 1.

Practical and Theoretical Implications

The findings establish evaluation scheduling not as a secondary engineering artifact, but as a principal factor determining APO efficacy and budget efficiency. Prompt-aware and adaptive subset selection, underpinned by submodular optimization and online discrimination modeling, drastically enhances optimization reliability and sample efficiency. Importantly, benefits are not strictly tied to any particular optimizer or underlying LLM, confirming generalizability.

These results implicate that, for future scalable or federated APO protocols, system designers should treat evaluation scheduling as a first-class, model-agnostic module—separating the prompt proposal mechanism from adaptive, theoretically sound evaluation selection. The monotone submodular structure further admits practical distributed or lazy-greedy variants for larger scale settings.

From a theoretical perspective, this work demonstrates how classical test theory and submodular analysis tightly integrate with LLM-specific optimization constraints (e.g., token cost, shifting prompt landscapes), yielding both practical and provably near-optimal algorithms.

Limitations and Future Directions

While POES delivers substantial gains on classification and structured-output tasks with moderate $$G_t(S) = \sum_{i \in S} u_t(i) + \lambda_t \sum_{j \in V} \max_{i \in S} s_t(i, j)$$1, further work is needed to:

• Extend discrimination utility and coverage terms to sequence generation and multi-class or open-ended tasks.
• Scale scheduler computation and stability mechanisms to $$G_t(S) = \sum_{i \in S} u_t(i) + \lambda_t \sum_{j \in V} \max_{i \in S} s_t(i, j)$$2, possibly leveraging coreset-approximate or GPU-accelerated selection strategies.
• Explore richer IRT variants (2PL/3PL) as pool sizes grow, including robust regularization to avoid early-stage overfitting.

Societal considerations include amplification of APO accessibility and identical application for adversarial prompt search. Direct LLM modifications are not involved, but systems employing POES must deploy the same safety alignments as any strong APO pipeline.

Conclusion

This work rigorously positions evaluation subset scheduling as the central bottleneck and opportunity for improving sample efficiency, stability, and final task accuracy in APO for LLMs. The proposed POES framework unifies online discrimination, coverage, stability, and adaptive exploration into an efficient and theoretically guaranteed scheduler, with consistently strong empirical results and practical generality across optimizers and LLMs. These findings should guide both research and deployment of future APO pipelines, where budget constraints and adaptability are paramount.