BMAS English: English-First Prompting

• BMAS English is a principle where using English prompts for tasks in Bangla, Hindi, and Urdu yields stronger zero-shot performance and more reliable outputs.
• Empirical evaluations on XNLI and SIB-200 with models like GPT-4, Llama-2-70b-chat, and Gemini Pro show significant macro-F1 gains when prompts are in English.
• The study attributes the English advantage to factors such as dominant pretraining data, better tokenizer coverage, and reduced translation noise, informing practical prompt-engineering strategies.

Searching arXiv for the primary paper and closely related multilingual prompt-evaluation work. “BMAS English” denotes the practice of prompting LLMs in English rather than in low-resource South Asian languages such as Bangla, Hindi, and Urdu, motivated by the empirical observation that English prompts often yield stronger zero-shot performance, cleaner outputs, and more reliable instruction following. In the evaluated setting, the concept is grounded in a comparative study of GPT-4, Llama-2-70b-chat, and Gemini Pro on XNLI and SIB-200 under five prompt settings that vary prompt language and translation origin, with English prompts consistently strongest on XNLI and generally the safest default overall (Dey et al., 2024).

1. Definition and scope

BMAS English is a shorthand for the proposition that, when interacting with LLMs on tasks involving Bangla, Hindi, or Urdu, it is often preferable to “ask in English.” The core research question is whether LLMs perform better when prompted in English compared to low-resource South Asian languages, and how cross-lingual prompt translations affect performance (Dey et al., 2024).

The evaluated scope is narrow and technically specific. The study uses zero-shot prompting only, without few-shot exemplars, and considers two multilingual benchmarks: Natural Language Inference on XNLI and topic classification on SIB-200. The language set is English, Bangla, Hindi, and Urdu; the model set is GPT-4, Llama-2-70b-chat, and Gemini Pro; and the prompting protocol examines five settings, P1 through P5, defined by whether prompts are originally authored in a language or translated from another language with Google Translate (Dey et al., 2024).

Within this scope, BMAS English is not a general theorem about multilingual NLP. It is a prompt-selection principle derived from zero-shot evaluation of specific models, datasets, and scripts. The paper’s own phrasing is pragmatic: “It is generally better to ask in English” (Dey et al., 2024).

2. Experimental formulation

The experimental design compares English prompts against Bangla, Hindi, and Urdu prompts under five cross-lingual prompt settings. P1 uses original prompts authored by native speakers in EN, BN, HI, and UR. P2 uses an English-origin prompt translated to BN, HI, and UR. P3 uses a Bangla-origin prompt translated to EN, HI, and UR. P4 uses a Urdu-origin prompt translated to EN, BN, and HI. P5 uses a Hindi-origin prompt translated to EN, BN, and UR. In all settings, translation is performed with Google Translate, and class labels are kept in English (Dey et al., 2024).

Two task templates are used. For XNLI, the prompt asks the model to classify a premise-hypothesis pair into “Entailment,” “Contradiction,” or “Neutral,” and to “Provide only label as your response.” For SIB-200, the prompt asks the model to classify a text into one of seven English class names: “science/technology,” “travel,” “politics,” “sports,” “health,” “entertainment,” or “geography,” again with a label-only output constraint. GPT-4 alone receives role metadata via a system instruction defining it as an expert data annotator; Llama-2 and Gemini receive plain text prompts without role information (Dey et al., 2024).

The datasets are also explicitly bounded. XNLI contributes EN, HI, and UR test sets, while Bangla XNLI is sourced from a public BN version translated from English using the Hasan et al. (2020) English→Bangla MT model; the BN test set has 4,895 samples. SIB-200 uses EN, BN, HI, and UR subsets, with 701 train, 99 validation, and 204 test instances in the dataset specification reported by the paper (Dey et al., 2024).

Evaluation reports accuracy and macro-F1, with weighted precision and recall also computed. The paper defines accuracy as

$$\text{Acc} = \frac{\sum_{i=1}^{N} \mathbf{1}(\hat{y}_i = y_i)}{N},$$

precision as $\text{Prec} = \frac{TP}{TP + FP}$, recall as $\text{Rec} = \frac{TP}{TP + FN}$, and F1 as

$$\text{F1} = 2 \cdot \frac{\text{Prec} \cdot \text{Rec}}{\text{Prec} + \text{Rec}}.$$

Macro-F1 is the unweighted average of per-class F1 (Dey et al., 2024).

3. Quantitative evidence for asking in English

The central empirical result is that English prompts consistently outperform Bangla, Hindi, and Urdu prompts on XNLI across models and settings, with GPT-4 strongest overall (Dey et al., 2024).

For GPT-4 on XNLI, macro-F1 in English is 86.79 in P1, 87.08 in P3, 81.99 in P4, and 86.81 in P5. The corresponding P1 results for Bangla, Hindi, and Urdu are 69.05, 71.97, and 64.77. The English advantage in P1 is therefore +17.74 over Bangla, +14.82 over Hindi, and +22.02 over Urdu (Dey et al., 2024).

For Llama-2-70b-chat on XNLI, English macro-F1 is 58.76 in P1, while Bangla, Hindi, and Urdu are 36.73, 38.57, and 35.56. The English advantage in P1 is +22.03 over Bangla, +20.19 over Hindi, and +23.20 over Urdu (Dey et al., 2024).

For Gemini Pro on XNLI, English macro-F1 is 73.36 in P1, while Bangla, Hindi, and Urdu are 61.74, 61.72, and 46.11. The English advantage in P1 is +11.62 over Bangla, +11.64 over Hindi, and +27.25 over Urdu (Dey et al., 2024).

On SIB-200, the English advantage is smaller and less uniform. GPT-4 remains strong, with English macro-F1 values of 84.35 in P1, 85.45 in P3, 86.63 in P4, and 85.52 in P5. Bangla, Hindi, and Urdu are often close to English in several settings; for example, GPT-4 achieves 84.35 on BN P1, 86.72 on HI P2, and 84.46 on UR P1. The largest reported gap is in P5 for Urdu, where EN is 85.52 and UR is 68.61 (Dey et al., 2024).

Llama-2-70b-chat shows the sharpest low-resource degradation on SIB-200. Its English P1 macro-F1 is 57.42, whereas Bangla, Hindi, and Urdu are 17.22, 8.20, and 5.14. The corresponding English advantages are +40.20, +49.22, and +52.28 (Dey et al., 2024).

Gemini Pro is intermediate. On SIB-200, its language differences are modest, and some low-resource settings approach or exceed English. A notable example is Urdu in P3, where macro-F1 reaches 84.10, compared with English 81.75 in the same setting (Dey et al., 2024). This indicates that BMAS English is strongest as a default heuristic rather than an exceptionless rule.

4. Mechanisms proposed to explain the English advantage

The study attributes BMAS English primarily to asymmetries in pretraining data, tokenizer coverage, script handling, and instruction-following robustness. English-dominant pretraining is the first explanation. The paper states that LLM training data is dominated by English, and reports that Llama-2 reportedly uses approximately 90% English data. GPT-4 and Gemini are likewise described as heavily English-centric (Dey et al., 2024).

Tokenizer coverage is the second explanation. English subword coverage is strong, whereas Bangla, Hindi, and Urdu scripts are less represented, which can lead to fragmentation, longer sequences, and noisier representations. The paper further notes script complexity and morphological richness as additional burdens in zero-shot settings, with BN, HI, and UR posing more modeling difficulty than English in many contexts (Dey et al., 2024).

A third explanation concerns translation noise. Prompt translation with Google Translate may introduce subtle errors. At the same time, the evidence in the paper suggests that the translation origin of the prompt has limited impact relative to the broader English advantage, because translated prompts sometimes match or exceed original prompts. This suggests that prompt-language quality and model robustness interact in nontrivial ways, but do not overturn the overall English preference (Dey et al., 2024).

A fourth explanation is that zero-shot cross-lingual transfer is imperfect. The paper argues that instruction understanding transfers less well to low-resource scripts, and that models learn task format more reliably in the dominant language. A plausible implication is that BMAS English is partly an instruction-following phenomenon rather than solely a semantic-understanding phenomenon (Dey et al., 2024).

5. Output reliability, failure modes, and model-specific behavior

BMAS English is not only about higher macro-F1; it is also about output regularity. The study reports that LLM outputs often required post-processing to strip extra characters and extract labels via regular expressions. If no label was detected, the output was mapped to “None” and counted as a misclassification (Dey et al., 2024).

The most severe failure mode is instruction misinterpretation in low-resource prompts, especially for Llama-2. On XNLI, Llama-2 exhibits very high invalid label counts in low-resource settings, including 4,147 invalid labels for BN P1 and 4,966 for HI P2. The paper also reports a bias toward “Neutral,” with HI P2 Neutral F1 of 99.61 and HI P1 Neutral F1 of 89.48, suggesting superficial or default predictions rather than stable NLI behavior (Dey et al., 2024).

Gemini’s failure mode is different. The paper states that Gemini Pro frequently blocked prompts containing harmful or inappropriate content. Even after setting all harm category thresholds to BLOCK_NONE, occasional blocked outputs remained. Invalid-label counts are often moderate, but Urdu XNLI P1 shows a spike to 1,330 invalid labels (Dey et al., 2024).

GPT-4 is the most regular generator. The paper states that GPT-4 generally returns only the label and has few invalid labels; for example, XNLI EN P1 has invalid label count 0 (Dey et al., 2024). This supports the stronger practical interpretation of BMAS English: English prompts improve not just task accuracy but also label-format compliance.

These failure patterns matter because the tasks are label-constrained classification tasks. A model that is semantically competent but format-unstable may still underperform after regex-based extraction. This suggests that BMAS English partly reflects reduced decoding pathology under English instructions (Dey et al., 2024).

6. Practical use and prompt-engineering implications

The paper’s practical recommendation is direct: if English instructions and English labels are acceptable, English prompts should be preferred. The expected gains are largest on XNLI, where English prompt advantages in P1 range from +17 to +22 macro-F1 points for GPT-4, +20 to +23 for Llama-2, and +12 to +27 for Gemini (Dey et al., 2024).

When localized prompts are necessary, the paper recommends bilingual prompts, English labels, and stronger output constraints. It specifically suggests bilingual prompts and code-switching, a glossary of key class names and definitions, and explicit output constraints such as “Answer with exactly one of: [labels].” It also recommends role conditioning for GPT-4, noting that the “system: expert annotator” framing improves reliability (Dey et al., 2024).

For strict classification tasks, the paper further recommends reducing verbosity by using lower temperature and narrower top-p when such controls are available, and setting max tokens very small for label-only outputs. Although decoding parameters were not reported in the experiments, this recommendation appears in the practical guidance section (Dey et al., 2024).

The study also proposes a robust cross-lingual pipeline centered on English instruction: write prompts in English; if inputs are in BN, HI, or UR, either keep inputs in the source language while constraining the model to output English labels, or machine-translate inputs to English for comprehension and back-translate outputs for user display if needed. Post-processing via regex extraction remains necessary, and mapping to “None” is treated as a last resort (Dey et al., 2024).

7. Limitations and interpretation

Several limitations constrain the scope of BMAS English. The evaluation is restricted primarily to NLI and one topic-classification dataset. Broader tasks such as question answering, machine translation, or summarization are not covered. The study is also zero-shot only; no few-shot prompting or fine-tuning is attempted, which may understate achievable low-resource performance (Dey et al., 2024).

Dataset composition is another limitation. Bangla XNLI is machine-translated rather than manually curated, and SIB-200 has a small test size. Gemini’s safety filtering may also confound some Urdu results, even after safety thresholds were set to BLOCK_NONE (Dey et al., 2024).

The evidence therefore supports a careful reading. BMAS English is a pragmatic default, not an absolute law. It is strongest for reasoning-heavy tasks such as XNLI, where the English advantage is large and consistent. On SIB-200, differences are smaller, and some low-resource settings approach or exceed English, particularly for Gemini and GPT-4 in selected prompt settings (Dey et al., 2024).

Taken as a whole, BMAS English identifies an English-first prompting regime for low-resource South Asian settings: use English instructions when possible, retain English class labels, add bilingual support only when required, and expect the largest benefits in tasks where instruction following and fine-grained semantic discrimination dominate performance (Dey et al., 2024).