Hierarchical Attribution Prompt Optimization

• HAPO is a hierarchical framework that decomposes language prompts into semantic units for targeted, interpretable optimization.
• It employs an attribution mechanism and iterative edit operations to reduce prompt drift and improve performance in text and multimodal tasks.
• The system uses a multi-agent workflow with rule-based segmentation and bandit-driven edit selection to refine large language model prompts.

Hierarchical Attribution Prompt Optimization (HAPO) is a model-agnostic, semi-automated framework for refining discrete, language-based prompts for LLMs, using an interpretable, iterative, and attribution-driven process. HAPO targets pervasive challenges in prompt engineering—including prompt drift and interpretability—by hierarchically decomposing prompts into functional semantic units, attributing performance gains or failures to these units, and guiding edit operations in a manner transparent both to the practitioner and downstream model (Chen et al., 6 Jan 2026, Liu et al., 2024).

1. Motivation and Design Challenges

Prompt engineering for LLMs is fundamentally an optimization problem over a discrete, combinatorial space of language fragments—words, clauses, instructions, and exemplars. Unlike gradient-based optimization, edits in this space tend to be non-smooth, and naively replacing or regenerating prompts often results in prompt drift: improvements on new failure cases at the expense of degradation on earlier correct instances. Furthermore, end-to-end prompt generation obscures the rationale behind changes, hindering interpretability and making systematic improvement difficult.

HAPO addresses these issues by:

• Treating prompts as hierarchical sequences of semantic units (e.g., bulleted steps, section headers).
• Attributing errors (or gains) to these internal segments.
• Iteratively and selectively applying a controlled set of edit operators to those units.
• Supporting both text-only and multimodal tasks, with a unified workflow.

This design prioritizes modularity, extensibility, and transparency, setting it apart from prior fully-automated or monolithic prompt optimization strategies (Chen et al., 6 Jan 2026).

2. Hierarchical Architecture and Zero-Shot Pipeline

HAPO operationalizes hierarchical multi-agent workflows, inspired by principles laid out in Hierarchical Multi-Agent Workflows for Prompt Optimization (Liu et al., 2024). The architecture is typically instantiated with three LLM "agents" arranged hierarchically:

Agent	Role	Output
Master Planner	Extracts global goals from the query $q$	CEO instruction $q^c$
Sub-Task Generator	Decomposes CEO goals into numbered, actionable sub-tasks	Manager instruction $q^m$
Prompt Refiner	Fuses all context into a finalized, explicit, and detailed prompt	Optimized prompt $p^*$

Skip-connections ensure that the raw query $q$ is always provided to each agent. Each layer receives context-dependent system prompts, enabling strict control over its linguistic and functional scope.

Given a user query $q$, the zero-shot construction pipeline proceeds as: $\text{CEO:}~q^c = \mathcal{M}^c(f^c(c^c, q))$

$$\text{Manager:}~q^m = \mathcal{M}^m(f^m(c^m, q, q^c))$$

$\text{Worker:}~p^* = f^w(c^w, q, q^m)$

An LLM then generates the final response $\hat{y} = \mathcal{M}^w(p^*)$.

This workflow supports untrained, zero-shot operation, and can be extended to multimodal environments where images are base64-encoded into the prompt meta-template (Chen et al., 6 Jan 2026, Liu et al., 2024).

3. Attribution Mechanism and Optimization Objective

Central to HAPO is the attribution mechanism, which tracks the incremental influence of each semantic unit on task performance. For a dataset $q^c$0, prompt optimization is framed as: $q^c$1 where $q^c$2 is a loss function and $q^c$3 is the LLM's output.

Each prompt $q^c$4 is decomposed into $q^c$5 semantic units $q^c$6. Attribution for each unit is dynamically updated using counterfactual occlusion and exponential smoothing: $q^c$7 where $q^c$8 are mispredicted instances at iteration $q^c$9, and $q^m$0 means the prompt excluding $q^m$1.

A history-aware decay term integrates prior improvements: $q^m$2 Top-m units by $q^m$3 are targeted for editing.

The core optimization is thus not global regeneration of prompts, but targeted, interpretable edits on segments whose attribution scores reveal inefficacy or error patterns (Chen et al., 6 Jan 2026, Liu et al., 2024).

4. Semantic-Unit Optimization and Edit Selection

Semantic units are extracted using a two-stage process:

1. Rule-based splitting on discourse markers and list delimiters.
2. Instruction parser ($q^m$4) merges fragments that are too short or splits run-on units to ensure each segment maps to a meaningful function.

The set of candidate edit operators is $q^m$5. Each candidate arm $q^m$6 applies operator $q^m$7 to unit $q^m$8, producing a new prompt $q^m$9.

Edit selection is governed by an upper confidence bound (UCB) multi-armed bandit scheme, balancing exploitation of known high-reward edits and exploration of less-tried actions: $p^*$0 where $p^*$1 is each arm's empirical mean reward and $p^*$2 its selection count.

Key properties:

• Arms with non-positive rewards are pruned.
• Warm start ensures all edits are attempted at least once.
• Early stopping and drift metrics prevent overfitting and regressive editing (Chen et al., 6 Jan 2026).

5. Multimodal Extension and Robustness

HAPO extends natively to multimodal workflows, where both text and image inputs appear as joint meta-prompts. The segmentation, attribution, and editing algorithms remain unchanged; only the input representation differs. In vision-language tasks, images are base64-encoded, with additional meta-instructions e.g., “Multimodal Task: ...” and visual features (e.g., red-box hints) can be referenced.

Empirical studies confirm that enhancements such as structured reasoning, prioritization of weak elements, and enriched visual features contribute 3–7% absolute accuracy improvements in vision-language benchmarks (OCRBench V2, VQA2017). Detailed ablations further show that removing structured meta-prompt components degrades accuracy and increases iteration requirements (Chen et al., 6 Jan 2026).

6. Empirical Validation and Comparative Outcomes

Benchmarked across text-only (BBH, GSM8K) and multimodal (OCRBench V2, VQA2017) datasets, HAPO produces consistent performance improvements. Key highlights include:

Model–Benchmark	Mean Accuracy (%)
Gemini-baseline (Zero-Shot CoT)	BBH: 70.23, GSM8K: 62.45, VQA: 39.68, OCRV2: 50.06
Gemini-HAPO	BBH: 89.76, GSM8K: 84.81, VQA: 48.40, OCRV2: 61.45
GPT-4o-HAPO	BBH: 85.94, GSM8K: 83.41, VQA: 60.17, OCRV2: 48.79
Qwen-HAPO	BBH: 75.70, GSM8K: 80.79, VQA: 45.19, OCRV2: 58.45

• HAPO outperforms Zero-Shot CoT by +13.28% on average and leading automated baselines by +7.21% on mean accuracy (across 11/12 model–task combinations).
• Multimodal prompts gain +2.54% on VQA and +1.80% on OCRV2 over OPRO.
• On general reasoning, HAPO matches or exceeds methods such as RaR, ExpertPrompting, On-Meta, and APE (Liu et al., 2024, Chen et al., 6 Jan 2026).

Efficiency is further demonstrated: HAPO converges in 6.7 iterations (mean 2 080 model calls per branch), as opposed to 453–180 274 for competitive methods.

7. Extensibility and Future Prospects

HAPO’s modular architecture and interpretable workflow make it amenable to domain adaptation (e.g., legal, medical, or scientific prompting), cross-model transfer (including open and proprietary LLMs), and integration with online or real-time adaptive learning via dynamic drifting thresholds and early stopping heuristics.

A plausible implication is that the transparency and segment-level attribution introduced by HAPO will enable future research directions in:

• Robust, deployable prompt adaptation under non-stationary query distributions.
• Automated, human-readable prompt optimization pipelines suitable for mixed-modal, large-scale tasks.
• Transferable templates enabling prompt component reuse across domains and models (Chen et al., 6 Jan 2026).

By overcoming prompt drift and opacity, while delivering state-of-the-art performance, HAPO represents a critical advancement toward scalable, interpretable, and efficient prompt engineering for both current and future LLM-driven systems.