HLPrompt: Structured Hierarchical Prompts

• HLPrompt is a prompt-centered research area that treats prompts as structured control objects to align semantic scale, task structure, and model adaptation.
• It encompasses methods such as hierarchical semantic prompting in vision, retrieval-aware strategies for hybrid text-table reasoning, and prompt-conditioned transformer adaptation.
• The approach enables automated prompt optimization and interactive prompt engineering, improving performance in areas like clinical NLP, code generation, and multimodal tasks.

HLPrompt denotes a prompt-centered line of research whose exact meaning varies across subfields. In current arXiv usage, the term is not confined to a single standardized method: it appears in work on hierarchical semantic prompting for higher-resolution diffusion and multimodal control, retrieval-aware prompting for hybrid reasoning over text and tables, zero-shot clinical NLP, prompt-conditioned transformer adaptation, prompt ensemble transfer, and automated prompt optimization (Liu et al., 2024, Luo et al., 2023, Sivarajkumar et al., 2022, He et al., 2022, Zhang et al., 9 Apr 2025, Chen et al., 6 Jan 2026). Across these settings, prompts are treated not as flat strings alone but as structured control objects that can encode hierarchy, retrieval order, model-layer conditioning, spatial focus, or optimization history.

1. Terminological scope and recurring design principles

The literature uses “hierarchy” in different ways. In some works, it means semantic scale, such as a global prompt paired with local patch prompts for image generation. In others, it means task structure, such as retrieve-then-reason prompting for hybrid table-text QA. Elsewhere it means model-internal organization, such as hypernetwork-generated prompts injected into self-attention, or optimization structure, such as semantic-unit prompt editing guided by attribution and prompt history (Liu et al., 2024, Luo et al., 2023, He et al., 2022, Chen et al., 6 Jan 2026).

This diversity yields a broad but coherent technical landscape. Taken together, these works suggest that HLPrompt-like systems typically pursue at least one of four goals: aligning prompts to multiple semantic scales, turning prompts into lightweight task-conditioning modules, using prompts as structured interfaces for reasoning and retrieval, or optimizing prompts automatically under explicit task objectives. A representative formalization appears in Hierarchical Attribution Prompt Optimization, which defines prompt search as

$$p^* = \arg\min_{p \in \mathcal{P}} \mathbb{E}_{(x,y^*) \sim D} \left[ \mathcal{L}(f(p,x), y^*) \right],$$

and then replaces ordinary parameter gradients with attribution-driven edits over semantic prompt units (Chen et al., 6 Jan 2026).

Mode of hierarchy	Representative mechanism	Representative papers
Semantic scale	Global and local prompts mapped to low/high-frequency or patch-level structure	(Liu et al., 2024)
Task structure	Retrieve-then-reason prompting over text and tables	(Luo et al., 2023)
Model depth	Hypernetwork-generated prompts injected into self-attention layers	(He et al., 2022)
Prompt segmentation	Semantic-unit attribution and UCB-guided editing	(Chen et al., 6 Jan 2026)
Multi-source transfer	Convex prompt ensembles with transferability and stability objectives	(Zhang et al., 9 Apr 2025)
Spatial focus	Foreground-only spatial prompts under domain shift	(Wang et al., 29 Apr 2026)

2. Hierarchical semantic control in vision and multimodal generation

A central HLPrompt formulation appears in higher-resolution image generation. HiPrompt attributes 4K+ generation failures to a semantic mismatch: a single global text prompt is applied uniformly to local patches, which leads to object repetition and structural artifacts. Its solution is a two-level prompt hierarchy consisting of a user-provided global prompt $\mathbf{y}^{\text{global}}$ and MLLM-generated local patch prompts $\{\mathbf{y}^j\}_{j=1}^Q$. During denoising, the latent is decomposed into low- and high-frequency components,

$$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$

with $\sigma = 2$, and the model combines prompt-conditioned noise estimates as

$$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$

On randomly sampled 1k LAION-5B prompts, HiPrompt reported $\text{FID}_r=67.79$ and $\text{FID}_c=35.82$ at $2048 \times 2048$, $\text{FID}_r=73.82$ and $\mathbf{y}^{\text{global}}$0 at $\mathbf{y}^{\text{global}}$1, and $\mathbf{y}^{\text{global}}$2 at $\mathbf{y}^{\text{global}}$3, outperforming several training-free baselines on most reported settings (Liu et al., 2024).

A second visual formulation appears in generalized category discovery under domain shifts. There, HLPrompt is a semantic-aware spatial prompt tuning extension of HiLo. It retains HiLo’s multi-level domain/semantic feature extraction, mutual information minimization, PatchMix augmentation, and curriculum sampling, but adds foreground-focused spatial prompting. Using NCut-derived foreground masks, it applies prompts only to selected patches,

$\mathbf{y}^{\text{global}}$4

and optimizes prompt and model parameters in alternating stages under the same HiLo loss. On DomainNet-GCD, for the Real $\mathbf{y}^{\text{global}}$5 Painting setting, HLPrompt reported Real-All $\mathbf{y}^{\text{global}}$6 and Painting-All $\mathbf{y}^{\text{global}}$7, improving over HiLo’s $\mathbf{y}^{\text{global}}$8 and $\mathbf{y}^{\text{global}}$9; on Scars-C it reported Corrupted-All $\{\mathbf{y}^j\}_{j=1}^Q$0, substantially above several baselines (Wang et al., 29 Apr 2026).

Prompt Highlighter extends the idea of hierarchical or localized control to autoregressive LLMs and VLMs at inference time. It lets a user highlight text spans or image regions, constructs regular and “unconditional” contexts by downscaling highlighted-token embeddings,

$\{\mathbf{y}^j\}_{j=1}^Q$1

combines the resulting conditional and unconditional next-token distributions with a classifier-free-guidance-style rule,

$\{\mathbf{y}^j\}_{j=1}^Q$2

and further activates attention to highlighted tokens by adding $\{\mathbf{y}^j\}_{j=1}^Q$3 to attention logits. Without tuning on LLaVA-v1.5, it reported $\{\mathbf{y}^j\}_{j=1}^Q$4 on the MMBench test and $\{\mathbf{y}^j\}_{j=1}^Q$5 on MME-perception, and on MSCOCO captioning with LLaVA-v1.5 it improved S-CLIP from $\{\mathbf{y}^j\}_{j=1}^Q$6 to $\{\mathbf{y}^j\}_{j=1}^Q$7 (Zhang et al., 2023).

These vision-oriented systems use different backbones and objectives, but they converge on a common HLPrompt principle: prompt hierarchy is mapped onto spatial scope, semantic scale, or attention focus, rather than being confined to a single natural-language string.

3. Structured prompting for reasoning, zero-shot inference, coding, and content analysis

In hybrid question answering, HRoT formulates prompting as a retrieval-first reasoning process over tables and text. On MultiHiertt, it first classifies questions as arithmetic or span selection, retrieves relevant paragraphs and table descriptions using a DeBERTa bi-classifier, reconstructs arithmetic tables with a type-aware table reconstruction algorithm, and then prompts GPT-3.5 with instructions such as “Let’s retrieve above text and table step by step and then think step by step to answer the question.” The retriever uses

$\{\mathbf{y}^j\}_{j=1}^Q$8

and a loss

$\{\mathbf{y}^j\}_{j=1}^Q$9

with $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$0. On MultiHiertt test data, HRoT-fewshot reached EM $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$1 and F1 $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$2, surpassing NAPG at EM $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$3 and F1 $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$4; its DeBERTa(+DSCLoss) retriever achieved text recall $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$5 and table recall $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$6 (Luo et al., 2023).

HealthPrompt applies prompt-based zero-shot learning to clinical NLP in a no-data setting. It chunks long EHR notes, inserts them into prompt templates such as {"text"} : {"mask"} type of disease, and interprets masked-token probabilities through a verbalizer,

$$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$7

On a 347-note MIMIC-III phenotype subset with 10 classes, the best configuration—ClinicalBERT with the {"text"} : {"mask"} type of disease cloze prompt—reported Accuracy $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$8, Precision $$\mathbf{z}_t = \underbrace{\mathbf{z}_t - G_\sigma(\mathbf{z}_t)}_{v_{\text{high}}(\mathbf{z}_t)} + \underbrace{G_\sigma(\mathbf{z}_t)}_{v_{\text{low}}(\mathbf{z}_t)},$$9, Recall $\sigma = 2$0, and macro F1 $\sigma = 2$1. The framework is explicitly zero-shot: no labeled target-task examples are used for fine-tuning (Sivarajkumar et al., 2022).

Prompt structure also appears as a reliability mechanism in automated coding. ADIHQ organizes code-generation prompts into six sections—Analyze, Design, Implement, Handle, Quality, and Redundancy Check—while avoiding explicit chain-of-thought output. On HumanEval with Granite, ADIHQ reported Pass@1 $\sigma = 2$2, Pass@100 $\sigma = 2$3, and $\sigma = 2$4 tokens, compared with Zero-Shot Pass@1 $\sigma = 2$5 and CoT Pass@1 $\sigma = 2$6; with LLAMA Code, ADIHQ reported Pass@1 $\sigma = 2$7, Pass@100 $\sigma = 2$8, and better Pass$\sigma = 2$9@token than CoT despite slightly lower raw Pass@k (Cruz et al., 19 Mar 2025).

HALC applies a similarly systematic prompt-construction logic to computational social-science coding. It defines a prompt search space over components such as role prompting, context information, task specification, coding strategy, chain-of-thought, and justification; evaluates prompt permutations against expert codings with Krippendorff’s alpha; and uses self-consistency via repeated coding and majority voting. Across 1,512 individual prompts and over two million requests to local LLMs, it reported reliable prompts for single variables with $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$0 and $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$1, and across two variables with $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$2 and $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$3, using Mistral NeMo (Reich et al., 29 Jul 2025).

Across these task-centered systems, HLPrompt-like design is less about hierarchy in the geometric sense than about structured decomposition of the task specification: retrieval before reasoning, label verbalization before classification, procedural sections before code synthesis, or codebook translation before automated coding.

4. Prompt-conditioned adaptation and transfer in foundation models

A separate strand turns prompts into learned conditioning modules inside frozen or lightly tuned backbones. HyperPrompt generates task-specific prompts for self-attention using HyperNetworks. For task $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$4 at layer $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$5, it prepends hyper-prompts to the key and value tensors,

$$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$6

where the prompt tensors are themselves generated from a task-global prompt $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$7 and layer-aware task embeddings through shared HyperNetworks. The model reported competitiveness against strong multi-task baselines with as few as $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$8 of additional task-conditioning parameters, and on T5-Large it reported GLUE $$\tilde{\epsilon}_t = \sum_k \epsilon_\theta(\mathbf{z}_t, \mathbf{y}^k, t).$$9 and SuperGLUE $\text{FID}_r=67.79$0, ahead of multi-task T5 and HyperFormer++ under the reported setup (He et al., 2022).

HGPrompt addresses a different adaptation problem: multi-source visual prompt transfer. It treats the target prompt as a convex combination of source prompts,

$\text{FID}_r=67.79$1

and learns $\text{FID}_r=67.79$2 by jointly maximizing an H-score-based transferability criterion,

$\text{FID}_r=67.79$3

while minimizing a Gradient Alignment Regularization term

$\text{FID}_r=67.79$4

On VTAB, HGPrompt reported an average accuracy of $\text{FID}_r=67.79$5, ahead of PANDA at $\text{FID}_r=67.79$6, SPoT at $\text{FID}_r=67.79$7, and VPT at $\text{FID}_r=67.79$8; in its ablation, combining H-score and Gradient Loss produced $\text{FID}_r=67.79$9 average accuracy versus $\text{FID}_c=35.82$0 without both (Zhang et al., 9 Apr 2025).

These methods operationalize prompts as internal control variables rather than external instructions. HyperPrompt uses prompts as task-specific memories in self-attention, whereas HGPrompt treats prompts as reusable assets whose transferability and mutual interference can be optimized explicitly. This suggests that one major meaning of HLPrompt is prompt-based parameter efficiency: the prompt becomes a structured, compositional adaptation surface over a frozen foundation model.

5. Automated prompt optimization and interactive prompt engineering

Another major HLPrompt interpretation treats prompts themselves as optimization objects. AutoHint begins from an initial prompt $\text{FID}_c=35.82$1, runs the LLM on labeled data, gathers the residual set of mistakes, asks the LLM to generate per-example hints $\text{FID}_c=35.82$2, samples representative failures, summarizes them into a global hint $\text{FID}_c=35.82$3, and forms an enriched prompt $\text{FID}_c=35.82$4. On BIG-Bench Instruction Induction tasks, it improved Hyperbaton from $\text{FID}_c=35.82$5 to $\text{FID}_c=35.82$6 in zero-shot accuracy and Implicatures from $\text{FID}_c=35.82$7 to $\text{FID}_c=35.82$8, while also improving several few-shot settings (Sun et al., 2023).

HAPO systematizes this optimization problem around prompt segmentation and attribution. It decomposes prompts into semantic units $\text{FID}_c=35.82$9, estimates counterfactual occlusion scores for each unit, smooths them with prompt-history gains, selects edit arms $2048 \times 2048$0 under a UCB rule, and monitors prompt drift by measuring degradation on previously solved examples. It reported better scores in $2048 \times 2048$1 model-benchmark pairs across BBH, GSM8K, OCRV2, and VQA, an average gain over Zero-Shot CoT of $2048 \times 2048$2 absolute, and mean optimization cost of $2048 \times 2048$3 model calls with $2048 \times 2048$4 iterations (Chen et al., 6 Jan 2026).

Automatic prompt generation can also be driven by task clustering. One 2025 system embeds 23 BBEH task descriptions, clusters them with k-means, assigns each cluster a portfolio of prompting techniques such as Role Playing, Emotion or Stress Prompting, a reasoning method, and possibly an additional technique, then uses a strong LLM to synthesize a prompt template from the task description plus technique descriptions. On BBEH, it reported arithmetic mean $2048 \times 2048$5 and harmonic mean $2048 \times 2048$6, exceeding the Original prompts at $2048 \times 2048$7 and $2048 \times 2048$8, and Anthropic’s prompt generator at $2048 \times 2048$9 and $\text{FID}_r=73.82$0 (Ikenoue et al., 20 Oct 2025).

GREATERPROMPT packages this optimization perspective into a unified toolkit supporting APE, APO, PE2, TextGrad, and GReaTer. On five BBH subtasks with Llama3-8B-Instruct, GReaTer reported an average of $\text{FID}_r=73.82$1 versus Zero-Shot CoT $\text{FID}_r=73.82$2 and TextGrad $\text{FID}_r=73.82$3; on GSM8K, GReaTer reached $\text{FID}_r=73.82$4 versus $\text{FID}_r=73.82$5 for Zero-Shot CoT (Zheng et al., 4 Apr 2025). PromptPilot extends the idea to interactive human-AI collaboration: an LLM-based prompting assistant that diagnoses missing prompt elements, asks guided questions, summarizes changes, and preserves user autonomy. In a randomized controlled experiment with 80 participants, PromptPilot produced a higher median overall score, $\text{FID}_r=73.82$6 versus $\text{FID}_r=73.82$7, with Holm-adjusted $\text{FID}_r=73.82$8 and $\text{FID}_r=73.82$9 (Gutheil et al., 1 Oct 2025).

This optimization-centered literature shifts HLPrompt from prompt design to prompt search, diagnosis, and governance. Prompts are evaluated, attributed, revised, clustered, and interactively refined under explicit metrics rather than being handwritten once.

6. Empirical patterns, limitations, and research directions

Several empirical patterns recur across the literature. First, prompt structure matters most when it aligns with the structure of the underlying problem. In HiPrompt, global prompts align with low-frequency layout and local prompts with high-frequency detail (Liu et al., 2024). In HRoT, retrieve-first prompting is tightly coupled to the retrieval and reconstruction pipeline (Luo et al., 2023). In HALC, acceptable prompts are concentrated around detailed coding strategies, build-up elements, chain-of-thought, and justification, rather than role prompts or generic context descriptions (Reich et al., 29 Jul 2025). In HGPrompt, transferability and stability must be optimized jointly because naive aggregation causes mutual interference (Zhang et al., 9 Apr 2025).

Second, the literature repeatedly reports a trade-off between expressivity and robustness. Richer prompts can improve semantic fidelity, but they also increase sensitivity to prompt wording, sample selection, or optimizer drift. HealthPrompt reports substantial variation across four simple templates, with the best cloze prompt clearly outperforming weaker formulations (Sivarajkumar et al., 2022). AutoHint’s second iteration improves some tasks but degrades others when hints are simply appended (Sun et al., 2023). HAPO introduces explicit drift metrics because prompt edits that fix new errors can break previous successes (Chen et al., 6 Jan 2026). Prompt Highlighter finds that moderate $\mathbf{y}^{\text{global}}$00, $\mathbf{y}^{\text{global}}$01, and $\mathbf{y}^{\text{global}}$02 work best, while overly strong activation or guidance can over-focus the model (Zhang et al., 2023).

Third, many HLPrompt systems are training-free or nearly training-free, but this does not imply low cost. HiPrompt adds MLLM captioning and parallel denoising overhead (Liu et al., 2024). HRoT still relies on a trained DeBERTa retriever and manual correction of few-shot reasoning traces (Luo et al., 2023). HALC required over two million local-model requests to evaluate its prompt grid (Reich et al., 29 Jul 2025). Automated prompt generation and optimization methods reduce manual labor but can require many model calls or strong optimizer models (Ikenoue et al., 20 Oct 2025, Zheng et al., 4 Apr 2025, Chen et al., 6 Jan 2026).

A plausible implication is that HLPrompt is increasingly becoming a systems concept rather than a mere prompting trick. In one branch, it denotes hierarchical semantic control over spatial, frequency, or multimodal structure. In another, it denotes structured task specification and retrieval-aware reasoning. In a third, it denotes prompt-conditioned adaptation modules inside frozen foundation models. In a fourth, it denotes prompt optimization pipelines with attribution, clustering, gradients, or human-in-the-loop guidance. The unifying theme is that prompt design is no longer treated as a flat natural-language preamble; it is treated as an organized, optimizable interface between task structure and model behavior.