Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs

Abstract: Automated prompt optimization is crucial for eliciting reliable reasoning from LLMs, yet most API-only prompt optimizers iteratively edit monolithic prompts, coupling components and obscuring credit assignment, limiting controllability, and wasting tokens. We propose Adaptive Prompt Structure Factorization (aPSF), an API-only framework (prompt-in/text-out; no access to model internals) that uses an Architect model to discover task-specific prompt structures as semantic factors. aPSF then performs interventional, single-factor updates: interventional factor-level scoring estimates each factor's marginal contribution via validation-performance changes, and error-guided factor selection routes updates to the current dominant failure source for more sample-efficient optimization. Across multiple advanced reasoning benchmarks, aPSF outperforms strong baselines including principle-aware optimizers, improving accuracy by up to +2.16 percentage points on average, and reduces optimization cost by 45--87% tokens on MultiArith while reaching peak validation in 1 step.

• The paper introduces adaptive prompt factorization that decomposes the prompt into semantic factors for targeted and explainable optimization.
• It demonstrates significant gains with up to +2.16pp accuracy improvement and reduces token usage by up to 87% on complex reasoning tasks.
• Methodologically, an Architect LLM discovers task-specific factors and error-guided, single-factor updates enable traceable and stable performance.

Factorized Prompt Optimization for LLMs via Adaptive Prompt Structure Factorization

Motivation and Problem Statement

Prompt optimization under API-only access remains a bottleneck for deploying LLMs in complex reasoning tasks. Conventional API-only optimizers (e.g., OPRO, APE, ProTeGi) perform iterative edits on monolithic prompt strings, treating the prompt as an indivisible unit. This design impedes credit attribution, confounds cause-specific improvements, reduces sample efficiency, and limits controllability, especially when simultaneous edits perturb multiple functional components. The paper introduces Adaptive Prompt Structure Factorization (aPSF) (2604.06699), an API-only framework addressing these limitations by explicit decomposition of prompts into semantic factors, enabling precise, interventional updates and error-guided routing of the optimization budget.

Figure 1: Contrast between monolithic prompt optimization and aPSF’s factorized update paradigm, highlighting selective editing through semantic factorization.

Framework: Adaptive Prompt Structure Factorization

aPSF operates in two sequential phases. The first phase, structure discovery, uses an Architect LLM to induce a task-specific factor schema—identifying roles such as problem analysis, reasoning, formatting, verification—from the task description and optional exemplars. The second phase, interventional factor optimization, iteratively refines a single factor per step, employing error-guided selection to maximize sample efficiency and optimization stability. Each factor is evaluated for its marginal contribution to validation performance by contrastive substitution while freezing all others, thus enabling explicit meta-attribution.

Figure 2: aPSF pipeline: structure discovery yields a compositional prompt schema, followed by error-guided, single-factor optimization.

Empirical Evaluation and Numerical Findings

aPSF was benchmarked against state-of-the-art API-only optimizers and programmatic frameworks across six reasoning datasets, including GSM8K, AQUA-RAT, MultiArith, GSM-Hard, BBH (17 sub-tasks), and MMLU. All methods use identical optimization budgets, validation slices, and initialization. The results show:

• Accuracy gains: aPSF outperforms the strongest baseline by up to +2.16pp, achieving 74.90% test accuracy on BBH and demonstrating exceptional coverage across arithmetic, symbolic, and logical reasoning categories.

  Figure 3: Per-task accuracy on BBH; aPSF (red) dominates arithmetic and symbolic tasks with scores up to 0.96/0.97.

• Optimization efficiency: On MultiArith, aPSF converges to peak validation in a single step using 206K tokens, reducing query cost by 45–87% compared to competitors.

  Figure 4: aPSF optimizes MultiArith with minimal token consumption and rapid convergence.

• Routing interpretability: Error-guided factor selection aligns optimization actions with observed task failures (e.g., prioritizing calculation factors in math, principle application in sciences). Factor-level knockouts and scheduler ablations confirm actionable, interpretable optimization.

  Figure 5: Heatmap of factor selection rates grouped by semantic domain; dominant factors correlate with per-task accuracy.

• Component sensitivity: Ablation studies reveal that removing task-specific factorization causes average drops of 3.35pp, with format-sensitive tasks suffering disproportionately.

  Figure 6: GSM-Hard factor-level ablation: removing Problem Understanding yields a $-2.10$pp drop, quantifying factor importance.

• Generalization and transfer: Prompts optimized on one arithmetic dataset transfer effectively to others, matching or exceeding same-task baselines, although transfer degrades for more distant tasks.

Theoretical and Practical Implications

aPSF’s interventional approach—single-factor freeze-edit—yields traceable credit assignment, interpretable optimization traces, and improved convergence properties under bounded query budgets. Error-guided routing can be interpreted as a contextual bandit policy maximizing the mutual information between factor edits and observed error modes, guaranteeing rapid reduction of critical failure sources. The independence assumption underlying single-factor updates trades off full joint optimization for tractable complexity and stability; future extensions might explore blockwise joint edits for synergistic factors.

The practical implications are substantial: factorization enables selective freezing of output schemas (critical for compliance tasks), reduces confounded validation trials, and supports actionable diagnostics for practitioners. Results on mathematical and logical reasoning further suggest that compositional factorization naturally aligns with LLMs’ structured decision-making capabilities.

Figure 7: Step-wise optimization trajectory on BBH Date Understanding; aPSF maintains high accuracy and convergence stability with minimized volatility.

Limitations and Future Directions

aPSF’s efficacy depends on the quality of the Architect LLM and the adequacy of its error diagnosis. The independence assumption, while empirically effective, may understate complex synergistic dependencies in highly entangled prompts. Current evaluation is limited to text-based reasoning benchmarks. Key avenues for further research include adaptive candidate generation based on factor saturation, hierarchical and multi-level factorization, human-in-the-loop refinement, improved factor-error association models, and extension to multimodal or tool-augmented settings.

Conclusion

Adaptive Prompt Structure Factorization establishes a principled, compositional framework for prompt optimization under API-only access, delivering improved accuracy, efficiency, and interpretability over monolithic prompt editing. The factor-level attribution and error-guided routing represent a methodological advance toward programmatic, diagnostic prompt engineering for LLMs, with both theoretical and practical significance in scalable reasoning applications.