MAGE: Understanding Stability-Performance Trade-offs in Multi-component Prompt Optimization

Abstract: How do different components of iterative prompt optimization interact, and what happens when they are combined? We investigate this through MAGE (Memory-Augmented Goal-directed Prompt Evolution), a controlled analysis framework for studying component interaction in prompt optimization. MAGE is not proposed as a superior optimizer in absolute terms; it integrates episodic memory, multi-objective Pareto selection, and adaptive evaluation as a platform for controlled ablation. Our experiments uncover a previously unreported phenomenon, the Prompt Optimization Coupling Effect (POCE): when multiple stochastic optimization signals operate within a closed reflective loop, they interact in ways that simultaneously improve performance and amplify variance, behavior that cannot be predicted by analyzing components in isolation. Three main findings emerge. First, failure-grounded reflection is essential: methods relying only on scores (OPRO) or abstract critique (Self-Refine) fail to improve prompts. Second, MAGE achieves 46.4% versus GEPA's 34.0% on GSM8K-Hard (+12.4%, P(MAGE>GEPA)=0.998, 5 seeds on gpt-4o-mini), with comparable variance (7.3% vs. 7.0%). Third, increasing candidate diversity reveals the clearest POCE signal: expanding the candidate pool from n=3 to n=5 improves mean accuracy by +21.6% while increasing variance by 3.7x. We further validate on Llama 3.1 8B and show POCE is headroom-dependent: when the base model already achieves high accuracy, variance amplification disappears. Finally, in low-data regimes (Ntrain=30), well-designed fixed prompts outperform all reflective optimizers, indicating that scaffold choice dominates optimizer choice. Our results suggest prompt optimization systems behave as coupled stochastic processes and should be evaluated in terms of both performance and stability, not just peak accuracy.

• The paper introduces MAGE as a framework that uncovers the Prompt Optimization Coupling Effect by jointly analyzing episodic memory, Pareto selection, and adaptive evaluation.
• The paper demonstrates how increasing candidate pool diversity triggers a trade-off between high-mean gains and amplified variance in optimization outcomes.
• The paper highlights practical insights, including scaffold ceiling effects and on-device deployment, advocating for multi-seed reporting to capture true system reliability.

MAGE: Stability-Performance Trade-offs in Multi-Component Prompt Optimization

Introduction and Problem Motivation

MAGE (Memory-Augmented Goal-directed Prompt Evolution) is introduced as a controlled analytic framework for investigating the stability–performance trade-offs inherent to multi-component prompt optimization systems. Whereas most literature reports single-seed results and focuses on mean performance, this work foregrounds variance as a critical, underexplored axis, positing that stochastic optimization signals within iterative prompt optimization loops are coupled rather than additive.

A key finding is the Prompt Optimization Coupling Effect (POCE): interacting optimization components—specifically episodic memory, Pareto multi-objective selection, and adaptive evaluation—jointly modulate both the mean and variance of prompt optimization outcomes in non-additive, regime-dependent ways, meaning the behavior of the composite system cannot be predicted by analyzing parts in isolation.

MAGE Architecture

MAGE systematically extends the reflective prompt optimization loop introduced by Gepa (Agrawal et al., 25 Jul 2025) by incorporating three mechanisms:

1. Episodic Memory: Allows cross-task regularization via retrieval and in-context inclusion of previous prompt–reflection–score tuples.
2. Multi-Objective Pareto Selection: Tracks a Pareto front over candidate prompts according to jointly optimized objectives (accuracy, brevity, safety), enabling trade-off navigation rather than scalar maximization.
3. Ensemble-Anchored Adaptive Evaluator: Mitigates evaluation drift over iterations by anchoring an adaptively calibrated judge to an initial fixed reference.

These explicit modular additions enable a rigorous ablation methodology—analyzing interaction effects and driving the discovery of POCE.

Main Findings

Necessity of Failure-Grounded Reflection

Ablations demonstrate that methods without failure-grounded reflection—e.g., OPRO (score-only) and Self-Refine (ungrounded critique)—fail to improve or even collapse performance. The absence of concrete failure signals either traps the system in suboptimal local optima or actively degrades reasoning strategies.

Superiority to Gepa Under Coupled Regime

On GSM8K-Hard (N_train=30, N_test=50, 5 seeds, gpt-4o-mini), MAGE (full) achieves $46.4\% \pm 7.3\%$ accuracy, a +12.4% increase over Gepa ($34.0\% \pm 7.0\%$, $P(\text{Mage} > \text{Gepa}) = 0.998$). In the most informative regime ($n=5$ for Pareto selection), MAGE achieves a $+21.6\%$ mean gain over Gepa, but variance increases by $3.7\times$ (from $\pm 4.7\%$ to $\pm 17.5\%$). These results, detailed in (Figure 1), expose stability–performance trade-offs induced by component interaction.

Figure 1: Stability–performance trade-off across methods on GSM8K-Hard, highlighting the impact of multi-component coupling on mean and variance.

Candidate Pool Diversity as Variance Amplification Trigger

Varying only the candidate pool size from $n=3$ to $n=5$ causally induces a pronounced POCE, where the system abruptly transitions from stable/moderate performance to high-mean/high-variance behavior. The emergence of meaningful Pareto pressure requires sufficient diversity; below threshold, the front degenerates and the optimizer stagnates.

Scaffold Ceiling Effects

A practical finding is that in low-data regimes ($34.0\% \pm 7.0\%$0), strong fixed Chain-of-Thought prompt scaffolds outperform all reflective optimizers (CoT-math achieves $34.0\% \pm 7.0\%$1 with zero optimization cost). Optimizer selection is therefore secondary to scaffold design when training data is scarce—an explicit, reproducible ceiling that is not minimized in the discussion.

Model Headroom and POCE Conditionality

On Llama 3.1 8B, where seed accuracy is near 70%, both reflective optimizers converge to nearly identical optimized prompts, and variance amplification disappears. Thus, POCE is headroom-conditional: coupled variance amplification is observed only when there is substantial optimization potential.

On-Device Deployment

In an applied on-device tests (1.5B deepseek-r1), MAGE converges in $34.0\% \pm 7.0\%$2 iterations with $34.0\% \pm 7.0\%$3 tool selection accuracy, fully resolving adversarial prompt classes. This demonstrates robustness and applicability in locked-weight runtimes (e.g., Google LiteRT, Apple CoreML).

Implications and Future Directions

Theoretical Implications:

The empirical demonstration that prompt optimization systems are coupled stochastic processes, rather than simply additive, motivates new analysis frameworks and reporting conventions. It undermines the sufficiency of mean-only reporting—practitioners and benchmark committees should, at a minimum, report both mean and variance across multiple random seeds to honestly reflect reliability.

Practical Implications:

Scenario-dependent recommendations (as detailed in the paper’s deployment guide) suggest adopting different MAGE module subsets for high-reliability applications (favoring adaptive evaluation) versus maximum-performance settings (accepting higher variance via increased candidate pool and full component stacks). Practitioners should benchmark strong task-specific scaffolds before engaging in expensive optimizer sweeps.

Future Developments:

Open directions include functional characterization of the diversity threshold (pool size $34.0\% \pm 7.0\%$4), structured multi-seed reporting in reflective prompt optimization papers, ablation across a broader suite of models and tasks, advanced semantic retrieval for episodic memory, and extension to multi-tool/partial-credit evaluation in on-device scenarios. These will further illuminate the boundaries of optimization coupling effects and scaffold-dominated regimes.

Conclusion

MAGE establishes a foundation for understanding the complex, regime-dependent trade-offs in multi-component prompt optimization. Through rigorous ablation, the work identifies and characterizes the Prompt Optimization Coupling Effect (POCE), demonstrating non-additive variance amplification and highlighting the crucial role of system coupling in reflective optimization loops. The framework provides actionable guidance for both research methodology and practical deployment, advocating for variance as a critical metric and calling for honest upper-bound benchmarking in low-data settings.