Hierarchical Prompting Index (HPI)

• Hierarchical Prompting Index (HPI) is a metric that quantifies task complexity and LLM capability by assessing the minimum prompting level needed for success.
• It utilizes a hierarchical framework with five escalating prompt types, reflecting stages from basic recall to advanced knowledge integration.
• Empirical comparisons between human and model-derived HP-Scores offer actionable insights for dataset evaluation, model benchmarking, and prompt engineering strategies.

The Hierarchical Prompting Index (HPI), designated as the Hierarchical Prompting Score (HP-Score) in its primary reference, is a standardized metric for quantifying the cognitive complexity of tasks and the competency of LLMs in solving them. Derived within the Hierarchical Prompting Taxonomy (HPT) and leveraging the Hierarchical Prompting Framework (HPF), the HPI systematically assesses the minimum level of prompting complexity required for LLMs to achieve task success. It provides both human- and LLM-derived measurements, facilitating a direct comparison of dataset demands, model performance, and prompting strategies. The HPI is grounded in human cognitive principles and supports rigorous evaluation of LLM reasoning, planning, and knowledge integration across diverse datasets (Budagam et al., 2024).

1. Formal Definition of the Hierarchical Prompting Index

The HPI/HP-Score associates each sample or task with a numeric value reflecting the complexity of the prompt needed for solution. Lower values indicate successful completion with minimal prompting (and thereby low cognitive demand), while higher values represent recalcitrant tasks or those only surmountable with highly complex prompting methodologies.

A. Dataset-Level HP-Score (Human Estimation)

Subject-matter experts manually evaluate a representative 5% subsample of a dataset using four cognitive axes:

1. $a_1$ = Basic Recall & Reproduction
2. $a_2$ = Understanding & Interpretation
3. $a_3$ = Analysis & Reasoning
4. $a_4$ = Application of Knowledge & Execution

For item $j$:

$hp_j = (a_1 + a_2 + a_3 + a_4) / 4$

The dataset's human HP-Score is the average over $n$ samples:

$$HP\textrm{-}Score_{dataset} = \frac{1}{n} \sum_{j=1}^{n} hp_j$$

Table: Example HP-Score$_{dataset}$ values

Dataset	HP-Score$_{dataset}$
BoolQ	1.71
CSQA	2.52
IWSLT	1.92
SamSum	2.23

B. Manual HPF HP-Score (LLM Performance)

LLMs are sequentially subjected to each of the $a_2$0 HPF prompt levels:

• If the model first succeeds at level $a_2$1, $a_2$2.
• If it fails across all levels, $a_2$3.

Aggregate model score:

$a_2$4

C. Adaptive HPF HP-Score

A prompt-selector LLM adaptively chooses levels for up to $a_2$5 iterations:

• If success at iteration $a_2$6 using level $a_2$7: $a_2$8
• If unsuccessful after $a_2$9 iterations: $a_3$0
• Mean is taken across samples as $a_3$1.

2. The Hierarchical Prompting Framework (HPF): Levels and Cognitive Requirements

The HPF organizes five distinct prompting strategies in ascending cognitive complexity:

Level	Prompt Type	Cognitive Demand
1	Role Prompting	Basic framing, recall
2	Zero-Shot Chain-of-Thought	Simple internal reasoning, no examples
3	Three-Shot CoT	Guided multi-step inference with exemplars
4	Least-to-Most Prompting	Sequential subproblem decomposition and planning
5	Generated Knowledge Prompt	Knowledge augmentation/integration, highest demand

Each escalation from Level 1 to Level 5 systematically increases requirements in memory, interpretation, decomposition, and external knowledge utilization. For a given sample, the lowest level at which the LLM succeeds is its $a_3$2 score.

3. HPI as a Cognitive Complexity and Competence Metric

The HPI/HP-Score serves as a dual-purpose metric:

• Dataset Complexity: The human HP-Score$a_3$3 quantifies the inherent cognitive challenge posed by a dataset, integrating memory, understanding, reasoning, and knowledge application axes. For example, CSQA's HP-Score$a_3$4 of 2.52 indicates substantially higher task complexity than BoolQ’s 1.71.
• LLM Problem-Solving Competency: Comparing HP-Score$a_3$5 to HP-Score$a_3$6 reveals the model’s efficiency relative to human baseline cognitive effort. Notably, HP-Score$a_3$7 values below HP-Score$a_3$8 signify that the model solves tasks with less complex prompts than typical human cognition would require.

4. Empirical Results Across Datasets and Models

Distinct patterns emerge in manual and adaptive HPI evaluations:

Manual HPF (HP-Score$a_3$9)

Model	BoolQ	CSQA	IWSLT(0.15)	IWSLT(0.20)	SamSum(0.15)	SamSum(0.20)
Llama 3 8B	1.37	2.43	3.40	3.92	1.30	1.72
Mistral 7B	1.41	2.49	3.52	4.15	1.26	1.68
Phi 3 3.8B	1.40	2.17	3.85	4.44	2.09	3.40
Gemma 7B	1.46	2.51	5.39	5.84	3.31	5.03

• All models outperform the human HP-Score$a_4$0 (1.71) on BoolQ, indicating that these tasks generally require only simple prompting (Level 1–2).
• CSQA’s elevated complexity is reflected in higher scores; Phi 3 3.8B achieves 2.17, closely aligning with the human benchmark.
• IWSLT and SamSum generally require prompts at Levels 3–5, indicating greater reasoning, decomposition, or external knowledge demands.

Adaptive HPF (HP-Score$a_4$1)

Model	BoolQ	CSQA	IWSLT(0.15)	IWSLT(0.20)	SamSum(0.15)	SamSum(0.20)
Llama 3 8B	5.22	5.91	6.20	6.28	5.03	5.58
Mistral 7B	5.05	5.91	6.25	6.46	4.74	5.13
Phi 3 3.8B	5.14	5.68	6.40	6.49	5.10	5.78
Gemma 7B	5.15	5.58	6.59	6.66	5.72	6.43

Adaptive HP-Scores are consistently higher, primarily due to prompt-selector hallucinations. Manual HPF is accordingly favored for reliability.

5. Implications for Dataset Evaluation and Benchmarking

• Dataset Assessment: Expert ratings of human HP-Score$a_4$2 serve as a standardized basis for estimating intrinsic difficulty and for calibrating new benchmarks.
• Model Benchmarking: Reporting HP-Score$a_4$3 alongside traditional metrics (e.g., accuracy, BLEU, ROUGE) provides nuanced insight into model competency and prompt efficiency.
• Prompt-Selector Development: Adaptive HPF reliability is contingent on minimizing selector hallucinations; strategies such as selector fine-tuning or ensembling are recommended for research advancement.
• Curricular Design: The five HPF levels constitute a progressive structure for training or evaluating smaller models on reasoning-intensive tasks.
• Best Reporting Practices: Including HP-Score metrics with conventional evaluations enables uniform and cognitively anchored profiling of tasks and model capabilities.

A finding that Llama 3 8B achieves HP-Score$a_4$4 ≈ 1.37 on BoolQ, well below the human score of 1.71, denotes both low dataset complexity and exceptional model capability. Conversely, persistently high manual HP-Scores (e.g., SamSum in the 3.3–5.0 range) point toward architectural or training limitations, especially in abstractive summarization domains.

6. Prospective Use Cases and Recommendations

In practice, the HPI framework supports:

• Validation of dataset complexity prior to adoption in benchmarks.
• Fine-grained model comparison based on required prompting effort.
• Construction of training curricula by sequencing HPF levels.
• Rigorous, multidimensional reporting in model evaluation studies.

A plausible implication is that widespread adoption of the HPI/HP-Score would facilitate deeper integration of cognitive principles into LLM assessment, encourage comparability across future datasets and models, and help systematize the study of prompt engineering strategies in relation to LLM capabilities (Budagam et al., 2024).