Hierarchical Few-Shot Example Selection

• The paper introduces TreePrompt, a tree-based algorithm that leverages LLM-provided labels to iteratively select high-quality examples for few-shot prompting.
• It combines hierarchical selection with flat retrieval methods like KNN and AFSP, resulting in measurable improvements in translation quality metrics such as COMET scores.
• Ablation studies indicate that increasing positive leaf samples boosts performance but also raises computational costs, highlighting a trade-off between quality and efficiency.

Hierarchical few-shot example selection is an approach for prompt construction in LLMs that leverages tree-based, quality-driven search to identify high-quality and contextually relevant demonstration examples. Departing from flat retrieval strategies—such as K-Nearest Neighbors (KNN) or Adaptive Few-Shot Prompting (AFSP)—that focus solely on similarity, hierarchical selection methods explicitly incorporate the LLM’s own judgment of example helpfulness. This methodology is particularly impactful in machine translation tasks, where careful selection of prompt sentences can drive significant improvements, especially in low-resource language pairs (Kakavand et al., 4 Oct 2025).

1. Motivation and Conceptual Basis

Few-shot prompting in LLMs typically involves presenting $k$ source–target sentence pairs (“demos”) immediately preceding the input to be translated. Traditionally, most example selection techniques are based on query-to-example similarity (e.g., via sentence embeddings), which may overlook whether an example substantively aids the LLM’s output quality. Random selection fails to guarantee relevance, and similarity metrics may surface semantically close but low-utility sentences. Hierarchical selection addresses these limitations by embedding the LLM's “preference signal” directly into the selection pipeline, forming a tree of candidate examples where only the most promising nodes—according to LLM-provided labels—are expanded iteratively (Kakavand et al., 4 Oct 2025).

2. TreePrompt Architecture and Algorithm

The TreePrompt algorithm exemplifies hierarchical few-shot example selection. The procedure commences with an initial random sampling (size $m$) from the prompt corpus $P$, where each example $e$ is labeled by the LLM according to its perceived utility—$l_e \in \{+1, 0, -1\}$, representing improvement, neutrality, or degradation of translation quality, respectively. Each node in the resultant tree corresponds to a single example and its assigned label.

Only leaves with $l_e = +1$ (“promising” examples) are expanded. For each expansion, the node with the highest label is selected and used as a query to retrieve $k$ semantically similar neighbors from $P$ (via RoBERTa-based KNN), which are in turn labeled by the LLM and attached as children. This process continues until a threshold number of positively labeled examples is accrued. Branches with labels $l_e = 0$ or $-1$ are deprioritized and not expanded further. The final output set consists of all examples with $l_e = +1$.

TreePrompt Main Steps:

Step	Operation	Method/Component
Initialization	Randomly sample $m$ examples, LLM-label each	LLM as oracle
Expansion	Select best-labeled leaf, retrieve $k$ KNN neighbors	RoBERTa embeddings
Labeling	LLM labels each new candidate (+1/0/–1)	Discrete labels
Termination	Stop when # positives $\geq$ threshold	Threshold hyperparameter

Unlike approaches that combine similarity and quality via a weighted sum, TreePrompt orders candidates strictly by the LLM’s discrete labels ($+1 > 0 > -1$) and does not employ a loss function or continuous score. The LLM acts as an on-demand oracle rather than being optimized end-to-end; each example can be re-evaluated in variant contexts, so the set of high-quality (“positive”) demos captures diverse contextual nuances (Kakavand et al., 4 Oct 2025).

3. Integration with Flat Selection Methods

TreePrompt is designed to operate in conjunction with standard flat retrieval strategies. After applying TreePrompt, the filtered set $P' \subset P$ contains only high-quality, LLM-selected examples. The final few-shot selection for prompting may then proceed via:

• Random: Uniformly sample from $P'$.
• KNN: Retrieve top $k$ examples from $P'$ by embedding similarity to the translation input.
• AFSP: Apply a hybrid retrieval on $P'$ using multi-embedding configurations and rerank by LLM-assigned quality.
• Reranking: The chosen 5 examples can be further reordered by the LLM for quality maximization.

This joint use of quality and similarity signals attempts to recover both contextual relevance and translation impact in the few-shot prompt selection (Kakavand et al., 4 Oct 2025).

4. Experimental Framework

The evaluation spans two language pairs:

• English–Persian: MIZAN corpus (9000 prompt-source, 520 test pairs)
• English–German: WMT19 (9000 source pairs, 500 test pairs; DeepSeek only)

LLMs evaluated include GPT-4o, GPT-3.5-Turbo, and DeepSeek-V2. For all setups, 5-shot prompting serves as the standard configuration. Performance is assessed using the COMET neural metric (primary), alongside BLEU, CHRF, and BERTScore. The comparative benchmarks include zero-shot, random, KNN, AFSP, and various hybrids incorporating TreePrompt (Kakavand et al., 4 Oct 2025).

5. Results and Ablation Analysis

Key findings include measurable gains in translation quality—particularly COMET scores—when TreePrompt is combined with AFSP or random selection, relative to flat KNN or random baselines. For English–Persian (MIZAN) on GPT-4o, AFSP yields –0.1581 (COMET), while TreePrompt-324 + AFSP achieves –0.1475 ($\Delta$=+0.0106). On DeepSeek, AFSP alone achieves –0.1512, versus TreePrompt-653 + Random+Rerank at –0.1424 ($\Delta$=+0.0088). For English–German (WMT19) with DeepSeek, the approach matches the best COMET (0.9003 vs 0.9004 for KNN), but zero-shot achieves the top BLEU/CHRF.

-Ablation studies show that increasing the number of positive leaves (e.g., from 144 to 324 for GPT-4o) generally improves COMET, albeit at higher computational cost. Hyperparameters—such as the random sample size, neighbor count, and number of expansion iterations—demonstrably impact final quality and resource usage (Kakavand et al., 4 Oct 2025).

6. Discussion, Limitations, and Future Directions

Direct incorporation of LLM preference signals into hierarchical selection yields example sets that the model itself deems beneficial, providing complementary gains relative to strategies that are agnostic to the LLM's internal utility assessment. Hierarchical expansion efficiently focuses computation on the most promising candidate regions. Integrating quality-filtered candidate pools with AFSP or reranking strategies further harmonizes semantic specificity and translation outcome.

Notable limitations include substantial computational overhead due to repeated LLM inferences and KNN operations, as well as uncertainties in automated metric reliability—particularly for small, low-resource test sets (manifested as negative COMET scores). Proposed directions for future work include scaling evaluations to additional language pairs and corpora, incorporating human evaluation for deeper quality assessment, and exploring more cost-efficient variants (e.g., batch LLM labeling, approximate KNN). Extension of the hierarchical selection paradigm to domains such as summarization or question answering is also suggested (Kakavand et al., 4 Oct 2025).