5-Level Prompt Intensity Framework

• 5-Level Prompt Intensity Framework is a hierarchical taxonomy that structures prompting strategies by ascending cognitive complexity in LLMs.
• It introduces the Hierarchical Prompting Index (HP-Score) to quantitatively assess task complexity and model proficiency across diverse NLP tasks.
• The framework employs Manual and Adaptive protocols to guide prompt design, dataset profiling, and optimization of LLM performance.

The 5-Level Prompt Intensity Framework is a hierarchical taxonomy for evaluating and engineering prompts in LLMs, structured to reflect human cognitive principles. It provides both a systematic methodology for prompt design and a universal metric—the Hierarchical Prompting Index (HPI or HP-Score)—that quantifies task complexity and model proficiency across diverse NLP tasks. The framework, formalized as the Hierarchical Prompting Framework (HPF) within the Hierarchical Prompting Taxonomy (HPT), organizes prompting strategies by cognitive demand, offering rigorous standards for model evaluation and dataset profiling (Budagam et al., 2024).

1. Structure and Cognitive Rationale of the Five Levels

The HPF comprises five distinct prompting strategies, each mapped to a specific cognitive capability, sequenced in ascending order of complexity:

1. Role Prompting (Level 1): Instructs the LLM to assume a specific role (e.g., "Translator," "Summarizer"), stimulating basic recall and reproduction of surface knowledge.
2. Zero-Shot Chain-of-Thought (Zero-CoT) Prompting (Level 2): Adds a step-by-step reasoning cue (e.g., "Let's think step by step.") to foster interpretation and understanding without providing examples.
3. Three-Shot Chain-of-Thought (3-CoT) Prompting (Level 3): Supplies three solved examples (question, answer, explanation), promoting analysis and inference through logical modeling.
4. Least-to-Most Prompting (Level 4): Decomposes the task into multiple sequential subtasks, requiring multistep reasoning reminiscent of problem-solving decomposition in humans.
5. Generated Knowledge Prompting (GKP, Level 5): Commands the LLM to first generate or retrieve pertinent external knowledge and then apply it to task resolution, emphasizing integration of extrinsic information and execution.

This ordering is grounded in four principal human cognitive functions—recall, comprehension, reasoning, and application—culminating with exogenous knowledge utilization at Level 5.

2. Hierarchical Prompting Index (HPI; HP-Score): Mathematical Specification

The HP-Score quantitatively captures the minimal prompting intensity at which an LLM solves a given sample, aligning prompt success/failure directly to task complexity. The scoring mechanisms are as follows:

• Human HP-Score: For human-annotated samples, experts assign four sub-scores ($a_1, a_2, a_3, a_4$), one per cognitive rule, yielding sample-level HP-Score:

$hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$

Dataset HP-Score aggregates over $n$ samples:

$$\mathrm{HP\text{-}Score}_{\mathrm{Dataset}} = \frac{1}{n}\sum_{j=1}^n hp_j$$

• Manual HPF: For LLMs, if first success occurs at level $x$ (among $m=5$), sample score:

$hp_j = x$

If all levels fail:

$hp_j = m + \mathrm{HP\_Score}_{\mathrm{Dataset}}$

The dataset Manual HPF score:

$$\mathrm{HP\_Score}_{\mathrm{Manual}} = \frac{1}{n}\sum_{j=1}^n hp_j$$

• Adaptive HPF: In $i$ iterative prompt selections, success at level $hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$0 in iteration $hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$1:

$hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$2

If all iterations fail:

$hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$3

Aggregated score:

$hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$4

Performance improvement is measured by the relative reduction in HP-Score:

$hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$5

3. Representative Prompting Templates Across Levels

Templates used for different datasets (BoolQ, CSQA, IWSLT [translation], SamSum [summarization]) exemplify practical instantiation at each HPF level:

Level	Template Example (condensed)	Cognitive Focus
1: Role	"Summarize dialogue '…' as a Summarizer."	Recall
2: Zero-CoT	"… Let's think step by step."	Interpretation
3: Three-CoT	"Example 1: Q1→A1, explanation1; … Solve Q."	Reasoning via analogical examples
4: Least-to-Most	"(1) Identify key points; … (4) Final answer."	Decomposition
5: GKP	"Generate knowledge about X. Answer Q using it."	Application of external knowledge

Empirically, correct prompt selection at lower levels indicates a less cognitively demanding task and greater generality of the model.

4. Algorithmic Procedures for Framework Deployment

HPF evaluation is operationalized through two main protocols:

• Manual HPF: Iteratively apply prompts from Level 1 through Level 5, stopping at the first successful response as determined by pre-set success metrics (accuracy, BLEU, ROUGE); assign corresponding HP-Score.
• Adaptive HPF: Employ a prompt-selector (potentially itself an LLM or ensemble), choosing prompt levels at each iteration up to $hp_j = \frac{1}{4}\sum_{i=1}^4 a_i$6 times; score upon success according to the formula above, or penalize if none succeed.

Both protocols require clear task-specific thresholds for success: accuracy for classification, BLEU (≥0.15 or ≥0.20) for translation, ROUGE-1 (≥0.15 or ≥0.20) for summarization.

5. Experimental Protocol, Benchmarks, and Evaluation Metrics

Experiments are conducted on four datasets—BoolQ, CommonSenseQA, IWSLT-2017 (English→French), and SamSum—using four instruction-tuned LLMs (Llama 3 8B, Phi 3 3.8B, Mistral 7B, Gemma 7B). HP-Scores are recorded on both Manual and Adaptive HPF schemes, with evaluation metrics as follows:

• BoolQ, CSQA: accuracy
• IWSLT: BLEU score (thresholds 0.15, 0.20)
• SamSum: ROUGE-1 (thresholds 0.15, 0.20)

Performance is tracked per HPF level, and improvement is measured by the reduction in HP-Score, with experimental runs demonstrating 2%–63% performance enhancement under HPF compared to baselines (Budagam et al., 2024).

6. Empirical Findings and Model Comparisons

Manual HPF achieves consistently lower HP-Scores, denoting that simpler prompting suffices for correct LLM inference in most cases; Adaptive HPF underperforms due to prompt-selector hallucination. On BoolQ, all models surpass human experts (human HP-Score ≈ 1.71) with near-perfect accuracy (≈0.99) at Level 1. On CSQA, Phi 3 3.8B exhibits maximal accuracy (≈0.88) but requires higher-level prompting. Translation and summarization tasks demonstrate higher complexity, as BLEU and ROUGE thresholds are infrequently met at low HP levels; LLMs yield HP-Scores exceeding human baselines (1.92 for IWSLT, 2.23 for SamSum).

Ordered by overall HP-Score (from strongest to weakest): Llama 3 8B < Mistral 7B < Phi 3 3.8B < Gemma 7B.

Adaptive HPF’s HP-Scores are 5–10% worse than Manual HPF’s, underscoring the importance of mitigating prompt-selector failures.

7. Guidelines for Effective HPF Application

• Dataset Profiling: Annotate 5% of samples with human HPF criteria to benchmark dataset complexity, establishing a reference scale.
• Prompt Design: Begin with clear, role-specific Level 1 templates, advancing incrementally to more complex strategies as required.
• Threshold Calibration: Standardize thresholds for success across levels (e.g., BLEU, ROUGE cutoffs) for fair evaluation.
• Model Selection: Deploy Manual HPF to ascertain model capacity at low complexity; persistent failure suggests the need for model tuning or data augmentation.
• Adaptive Automation: Counter hallucinations in automated prompt-selectors by ensembling approaches, restricting search space, or self-critiquing after each trial.
• Interpretive Use: Compare HP-Scores across models and datasets—lower scores indicate resolution with simpler prompts and higher generality; disparities between human and model HP-Scores reveal underrepresented domains.

The HPF and HP-Score index collectively furnish a rigorous, fine-grained approach to prompt engineering and universal evaluation of LLM task complexity, enabling systematic diagnosis of both dataset hardness and modeling capability (Budagam et al., 2024).