TreePrompt: Leveraging Hierarchical Few-Shot Example Selection for Improved English-Persian and English-German Translation (2510.03748v1)

Abstract: LLMs have consistently demonstrated strong performance in machine translation, especially when guided by high-quality prompts. Few-shot prompting is an effective technique to improve translation quality; however, most existing example selection methods focus solely on query-to-example similarity and do not account for the quality of the examples. In this work, we propose TreePrompt, a novel example selection approach that learns LLM preferences to identify high-quality, contextually relevant examples within a tree-structured framework. To further explore the balance between similarity and quality, we combine TreePrompt with K-Nearest Neighbors (K-NN) and Adaptive Few-Shot Prompting (AFSP). Evaluations on two language pairs - English-Persian (MIZAN) and English-German (WMT19) - show that integrating TreePrompt with AFSP or Random selection leads to improved translation performance.

• The paper introduces a novel tree-based few-shot example selection method that integrates LLM labeling to enhance translation fidelity.
• It employs iterative K-NN expansion on RoBERTa embeddings to select semantically relevant examples for both English-Persian and English-German translations.
• Experimental results on MIZAN and WMT19 datasets reveal significant COMET score improvements, especially beneficial for low-resource translation tasks.

TreePrompt: Leveraging Hierarchical Few-Shot Example Selection

Introduction

Machine translation (MT) has greatly advanced through the use of LLMs, notably improving with architectures such as GPT-3.5 and GPT-4. Yet, these models rely heavily on the quality of few-shot prompts to enhance translation fidelity. Existing example selection strategies for few-shot prompting primarily focus on query-to-example similarity, often neglecting the intrinsic quality of examples. Addressing this limitation, the "TreePrompt: Leveraging Hierarchical Few-Shot Example Selection for Improved English-Persian and English-German Translation" (2510.03748) proposes a novel framework that integrates the translational preferences of LLMs within a hierarchical selection process, emphasizing both quality and contextual relevance.

Figure 1: An overview of the proposed Tree-based example selection approach for few-shot translation prompting.

TreePrompt Methodology

The Tree-Based Approach

TreePrompt innovates by dynamically assessing the quality of translation examples through LLM feedback, contrasting traditional methods that rely solely on similarity metrics. The process begins with a set of randomly selected examples, which are initially labeled by the LLM as positive, neutral, or negative based on their potential to improve translation quality. Subsequently, the algorithm iteratively refines the selected examples. Using a K-NN algorithm applied to RoBERTa embeddings, the most promising examples are expanded into a tree-like structure, fostering the selection of semantically relevant candidates.

This tree-based strategy aims to identify and promote high-quality example prompts that align better with LLM preferences, thereby enhancing translation performance. The approach facilitates a more effective balance between example quality and input similarity, surpassing traditional methods like K-NN and AFSP in generating model-aligned prompts.

Experimental Evaluation

Language Datasets and Models

TreePrompt was rigorously evaluated on two benchmark datasets: MIZAN Persian-English and WMT19 English-German, employing multiple LLMs including DeepSeek, GPT-3.5-Turbo, and GPT-4o. The experiments focused on validating the effectiveness of TreePrompt combined with Random selection, AFSP, and K-NN methods.

Results Overview

The experimental findings reveal significant performance improvements across the tested models and metrics, notably in COMET scores. TreePrompt integrated with AFSP or Random selection consistently yields superior translation outcomes, demonstrating a robust framework for selecting impactful examples, particularly beneficial in low-resource translation settings.

The study highlights the capacity of TreePrompt to achieve comparable translation quality with fewer high-quality examples instead of relying on a multitude of prompts. This efficiency underscores the importance of LLM preference-driven selection strategies, offering valuable insights for optimizing few-shot learning configurations.

Computational Complexity

While TreePrompt involves substantial computational overhead due to its iterative K-NN expansion and LLM scoring processes, its design allows for significant prompt refinement early in the selection process. Proper parameter tuning can balance computational costs with performance, potentially reducing the need for excessive high-quality examples retrieval. The combination of labeling and tree expansion enables the efficient construction of model-aligned prompts.

Conclusion

The integration of LLM preferences into few-shot example selection through TreePrompt represents a substantive methodological advancement for machine translation. By constructing hierarchical and adaptive prompts, the framework improves translation quality and contextual relevance, particularly for low-resource language pairs. While computationally intensive, TreePrompt effectively enhances the selection of high-quality examples, affirming the importance of LLM alignment in prompt design.

Overall, TreePrompt's structured approach to example selection paves the way for future enhancements in LLM-driven machine translation tasks, promising greater efficacy in diverse linguistic contexts. Further exploration on larger and more varied datasets, alongside human evaluations, will be instrumental in bolstering its impact and applicability.