Multilingual Prompt Construction

• Multilingual prompt construction is a set of techniques that design both discrete and soft prompts to enable pre-trained models to perform tasks in multiple languages.
• It employs unified, language-agnostic architectures such as UniPrompt to fuse contextual and prompt representations and bridge cross-lingual gaps.
• Advanced strategies including minimal translation, multimodal integration, and mixture-of-expert frameworks significantly enhance performance in low-resource and cross-lingual settings.

Multilingual prompt construction refers to the set of techniques, design frameworks, and empirical strategies for leveraging or synthesizing prompts—natural language or continuous vectors—that enable pre-trained or parameter-efficient models to perform tasks across multiple languages, with or without further supervised training. Its goal is not only to bridge the gap between language-specific abilities, but also to unlock the cross-lingual generalization potential of pre-trained LLMs, particularly in low-resource, cross-lingual, or transfer learning contexts. The following sections detail core approaches, evaluation findings, architectures, and methodological insights from contemporary research underpinning multilingual prompt construction.

1. Discrete and Soft Prompting Approaches

Discrete prompting (DP) and soft prompting (SP) are foundational paradigms for multilingual prompt construction, exemplified by their deployment in natural language inference (NLI) with XLM-RoBERTa (Zhao et al., 2021):

• Discrete Prompts (DP): Tasks are reformulated as cloze-style “fill-in-the-blank” queries, e.g.,

  Premise. Question: Hypothesis? Answer: ______

  The model’s output token is mapped to class labels via a hand-crafted verbalizer (e.g., “yes”→entailment, “no”→contradiction, “maybe”→neutral). Optimization uses standard cross-entropy loss ($L = -\log p(y)$) with the gold verbalizer token. This method is fully interpretable and maintains strict alignment with the model’s pretraining objective.
• Soft Prompts (SP): A sequence of learnable continuous pseudo-tokens $\mathbf{v}_i \in \mathbb{R}^d$ is inserted before the decision token. To enhance performance, these vectors are processed through a bi-directional LSTM and optionally an MLP before integration; the soft prompt parameters are optimized jointly with cross-entropy loss, while the PLM is kept frozen or semi-frozen.
• Language Adaptation: In cross-lingual transfer, prompts can mix source language cues, code-switched tokens, or be directly translated, e.g., “Question”→“Soru” in Turkish. Performance gains hold for both crosslingual and in-language promptings, even with machine-translated prompts.
• Empirical Performance: In low-resource settings (few-shot, zero-shot), both DP and SP far surpass fine-tuning: for NLI, 4-shot SP yields $41.84\%$ (vs. $33.90\%$ finetune), and with 48-shot cross-lingual English examples, SP achieves $38.79\%$ (fine-tune: $33.74\%$). Discrete prompting, while slightly less performant, provides full interpretability and sometimes superior transfer for certain linguistic setups.

2. Unified and Language-Agnostic Prompting Architectures

To address the combinatorial explosion of language-prompt pairs, unified architectures have been developed (Huang et al., 2022):

• UniPrompt: Employs a two-tower encoder (Template Tower for prompts and Context Tower for task inputs) using duplicated lower layers of a multilingual PLM. Output representations are fused in a top-layer Fusion Tower. Language-agnostic label words (“soft label words”) are initialized as the averaged [MASK] token embeddings from training samples, guaranteeing distributional closeness across languages.
• Optimization: Cross-entropy over label word embeddings ensures the predicted [MASK] token’s hidden vector is close to the label’s averaged embedding. This method avoids manual prompt translation and is robust under zero-shot transfer to unseen languages.
• Empirical Results: On the MARC corpus, UniPrompt consistently outperforms both translation-prompt and standard soft/discrete-prompt approaches, with $2–4\%$ higher accuracy in few-shot settings.
• Implication: Decoupling language-specific and semantic components produces robust language-agnostic representations, reducing the need for language-specific engineering and facilitating efficient cross-lingual generalization.

3. Prompt Design for Multitask and Multilingual Transfer

Polyglot Prompting (Fu et al., 2022) extends prompt construction to unified multitask, multilingual frameworks using the sequence-to-sequence paradigm:

• Prompt Templates: Each task has a predefined template $K_k$, converting $(x_{ij}, y_{ij})$ to $(\hat{x}_{ij}, \hat{y}_{ij})$. Multilingual prompts are either “in-lingual” (template and data in the target language) or “cross-lingual” (e.g., universal English templates).
• Loss and Objective: Sequence-to-sequence modeling maximizes

$$L = \sum_{(\hat{x},\hat{y})\in (X,Y)} \log \left[ \prod_{m=1}^{|\hat{y}|} P(\hat{y}_m|\hat{y}_{<m},\hat{x};\theta) \right]$$

• Unified Template Advantage: Experiments show that using universal prompt templates (e.g., all tasks leveraging similar cue structures) enhances cross-task, cross-lingual transfer, even for low-resource languages and tasks.
• Interpretable Evaluation: Performance is broken down by granular linguistic features (e.g., context/question/answer length, BLEU overlap), facilitating feature-level analysis across language-task pairs.

4. Minimal Translation and Transfer Strategies

Prompt-based methods for multilingual relation classification and extraction demonstrate the efficacy of minimizing translation beyond what is necessary (Chen et al., 2022, Hsu et al., 2023):

• Direct Prompt Derivation: Prompts are constructed directly from relation triples, e.g.,

  x. e_h ____ e_t

  with a verbalizer $\varphi(r)$ mapping relations to natural language labels (typically minimally translated into the target language, or kept in English for code-switch settings).
• Hard/Soft/Hybrid Templates: Experiments with hard (fully natural-language), soft (learned token insertions), and hybrid prompts corroborate that minimal translation—just for label words—is sufficient in most cases to outperform fine-tuning, especially under few-shot or zero-shot settings.
• Empirical Takeaways: In fully supervised and few-shot learning, in-language prompt construction yields the most robust results for low-resource languages, while code-switched prompts dominate in pure zero-shot inference owing to the models’ English-centric pretraining.

5. Advanced Prompt Construction Strategies

Recent research extends prompt construction to tackle LLM limitations, generalization, and cross-lingual fairness/robustness:

• Cross-Lingual-Thought Prompting (XLT): A multi-stage English-anchored template that forces LLMs to “think in English,” analyzing and reasoning in stepwise fashion before outputting in the target language. This reduces performance gaps across high- and low-resource languages, with measurable increases in the “democratization score” (Huang et al., 2023).
• Linguistically Diverse Prompting (LDP): Unsupervised synthetic exemplars from high-resource languages enable LLMs to generalize to low-resource languages without human-crafted examples. In both translation and summarization, LDP matches or outperforms supervised few-shot prompting (Nguyen et al., 2023).
• Dictionary Insertion Prompting (DIP): For non-English prompts, dictionary-based interleaving of English equivalents stimulates more effective pivoting to English-centric LLMs, yielding dramatic accuracy gains (+50 points in GSM8K over standard prompting in some settings) over 200 languages (Lu et al., 2 Nov 2024).
• Multicultural Diversity through Multilingual Prompting: Explicitly encoding cultural or linguistic cues in prompts, and combining outputs from several languages or cultures, robustly increases the diversity and factuality in LLM generations while mitigating hallucinations, proportionally to language/model resource levels (Wang et al., 21 May 2025).
• Automated Mixture-of-Expert Prompt Frameworks: Demo clustering via multilingual semantic embeddings, followed by region-based joint instruction search, produces specialized expert prompts for language or dialect clusters. This mixture approach outperforms single-instruction setups on average benchmarks by 81%. Routing is accomplished via a kernel regression-inspired matching mechanism (Wang et al., 28 Jun 2024).

6. Multimodal and Multilingual Prompt Construction

Expanding to multimodal systems, cross-lingual prompt construction incorporates auxiliary modalities (e.g., vision or audio) for grounding and robust alignment:

• Multimodal Prompting for NMT: Visual tokens (image patches via ViT/CLIP) are concatenated or cross-attended with text tokens, functionally serving as a universal "language" anchor. Contrastive loss (e.g., InfoNCE) further aligns representations across languages and modalities, yielding up to +4 BLEU in low-resource directions (Yang et al., 26 Mar 2024).
• Prompt Selection for Speech Generation: A dedicated multimodal, multilabel prompt database (M³PDB) is indexed using visual/text/speech similarity, with hierarchical, multi-agent annotation of prompts to ensure precise cross-lingual, cross-modal transfer. The selection pipeline uses a two-stage, latency-aware filtering scheme to support real-time, low-resource multilingual speech tasks (Zhu et al., 13 Aug 2025).
• Parameter-Efficient Prompt Tuning in Speech Recognition: Language-Aware Prompt Tuning (LAPT) leverages cross-lingual similarity metrics (e.g., Whisper language IDs) to select and adapt lightweight prompt matrices for language expansion, significantly reducing interference and catastrophic forgetting while preserving parameter efficiency (Yang et al., 16 Jun 2025).

7. Synthesis, Challenges, and Open Directions

• Best Practices:
  • Align with the PLM's pretraining objective (cloze/MLM, translation, denoising).
  • Use language-agnostic or dynamically learned prompt components where possible.
  • Minimal translation (only relation or label words) is usually sufficient; full prompt translation may not confer additional gains and is often unnecessary.
  • For multimodal setups, visual input serves as a robust cross-lingual disambiguation and alignment signal.
  • Prompt mixtures and demo clustering facilitate flexible, dialect/language/feature-specific prompt adaptivity in diverse application spaces.
• Trade-offs:
  • Soft prompts offer higher flexibility and (often) accuracy but lose interpretability compared to discrete prompts.
  • Hybrid and mixture architectures outperform monolithic prompt strategies but introduce complexity in expert selection, routing, and parameter management.
• Limitations & Future Work:
  • Full language coverage, especially in low-resource, morphologically complex, or OOD settings, is still limited.
  • Evaluations consistently reveal persistent biases (e.g., in image generation across gendered languages (Friedrich et al., 29 Jan 2024)), indicating that prompt engineering alone is insufficient to address systemic issues.
  • Automatic prompt translation and selection mechanisms must consider translation quality and pretraining resource imbalances to avoid performance loss.
  • Integrating prompt construction with ongoing developments in parameter-efficient adaptation (e.g., LoRA, PEFT, dynamic architectures) and multimodal fusion is a key avenue.

By systematically leveraging discrete and soft prompt paradigms, dynamic trigger tokens, mixture-of-expert clustering, minimal translation, dictionary insertion, hierarchical annotation, and cross-modal alignment, multilingual prompt construction has become a crucial mechanism for language- and modality-robust adaptation in contemporary NLP and multimodal systems. Continued progress relies on expanding evaluation coverage, refining automated prompt selection, and developing intrinsically fair and steerable frameworks for cross-lingual and cross-domain tasks.