---
title: 'BMAS English: English-First Prompting'
url: https://www.emergentmind.com/topics/bmas-english
type: topic
---

# BMAS English: English-First Prompting

Searching arXiv for the primary paper and closely related multilingual prompt-evaluation work.
“BMAS English” denotes the practice of prompting large language models in English rather than in low-resource South Asian languages such as Bangla, Hindi, and Urdu, motivated by the empirical observation that English prompts often yield stronger zero-shot performance, cleaner outputs, and more reliable instruction following. In the evaluated setting, the concept is grounded in a comparative study of GPT-4, Llama-2-70b-chat, and Gemini Pro on XNLI and SIB-200 under five prompt settings that vary prompt language and translation origin, with English prompts consistently strongest on XNLI and generally the safest default overall [2410.13153].

## 1. Definition and scope

BMAS English is a shorthand for the proposition that, when interacting with large language models on tasks involving Bangla, Hindi, or Urdu, it is often preferable to “ask in English.” The core research question is whether LLMs perform better when prompted in English compared to low-resource South Asian languages, and how cross-lingual prompt translations affect performance [2410.13153].

The evaluated scope is narrow and technically specific. The study uses zero-shot prompting only, without few-shot exemplars, and considers two multilingual benchmarks: Natural Language Inference on XNLI and topic classification on SIB-200. The language set is English, Bangla, Hindi, and Urdu; the model set is GPT-4, Llama-2-70b-chat, and Gemini Pro; and the prompting protocol examines five settings, P1 through P5, defined by whether prompts are originally authored in a language or translated from another language with Google Translate [2410.13153].

Within this scope, BMAS English is not a general theorem about multilingual NLP. It is a prompt-selection principle derived from zero-shot evaluation of specific models, datasets, and scripts. The paper’s own phrasing is pragmatic: “It is generally better to ask in English” [2410.13153].

## 2. Experimental formulation

The experimental design compares English prompts against Bangla, Hindi, and Urdu prompts under five cross-lingual prompt settings. P1 uses original prompts authored by native speakers in EN, BN, HI, and UR. P2 uses an English-origin prompt translated to BN, HI, and UR. P3 uses a Bangla-origin prompt translated to EN, HI, and UR. P4 uses a Urdu-origin prompt translated to EN, BN, and HI. P5 uses a Hindi-origin prompt translated to EN, BN, and UR. In all settings, translation is performed with Google Translate, and class labels are kept in English [2410.13153].

Two task templates are used. For XNLI, the prompt asks the model to classify a premise-hypothesis pair into “Entailment,” “Contradiction,” or “Neutral,” and to “Provide only label as your response.” For SIB-200, the prompt asks the model to classify a text into one of seven English class names: “science/technology,” “travel,” “politics,” “sports,” “health,” “entertainment,” or “geography,” again with a label-only output constraint. GPT-4 alone receives role metadata via a system instruction defining it as an expert data annotator; Llama-2 and Gemini receive plain text prompts without role information [2410.13153].

The datasets are also explicitly bounded. XNLI contributes EN, HI, and UR test sets, while Bangla XNLI is sourced from a public BN version translated from English using the Hasan et al. (2020) English→Bangla MT model; the BN test set has 4,895 samples. SIB-200 uses EN, BN, HI, and UR subsets, with 701 train, 99 validation, and 204 test instances in the dataset specification reported by the paper [2410.13153].

Evaluation reports accuracy and macro-F1, with weighted precision and recall also computed. The paper defines accuracy as
\[
\text{Acc} = \frac{\sum_{i=1}^{N} \mathbf{1}(\hat{y}_i = y_i)}{N},
\]
precision as \(\text{Prec} = \frac{TP}{TP + FP}\), recall as \(\text{Rec} = \frac{TP}{TP + FN}\), and F1 as
\[
\text{F1} = 2 \cdot \frac{\text{Prec} \cdot \text{Rec}}{\text{Prec} + \text{Rec}}.
\]
Macro-F1 is the unweighted average of per-class F1 [2410.13153].

## 3. Quantitative evidence for asking in English

The central empirical result is that English prompts consistently outperform Bangla, Hindi, and Urdu prompts on XNLI across models and settings, with GPT-4 strongest overall [2410.13153].

For GPT-4 on XNLI, macro-F1 in English is 86.79 in P1, 87.08 in P3, 81.99 in P4, and 86.81 in P5. The corresponding P1 results for Bangla, Hindi, and Urdu are 69.05, 71.97, and 64.77. The English advantage in P1 is therefore +17.74 over Bangla, +14.82 over Hindi, and +22.02 over Urdu [2410.13153].

For Llama-2-70b-chat on XNLI, English macro-F1 is 58.76 in P1, while Bangla, Hindi, and Urdu are 36.73, 38.57, and 35.56. The English advantage in P1 is +22.03 over Bangla, +20.19 over Hindi, and +23.20 over Urdu [2410.13153].

For Gemini Pro on XNLI, English macro-F1 is 73.36 in P1, while Bangla, Hindi, and Urdu are 61.74, 61.72, and 46.11. The English advantage in P1 is +11.62 over Bangla, +11.64 over Hindi, and +27.25 over Urdu [2410.13153].

On SIB-200, the English advantage is smaller and less uniform. GPT-4 remains strong, with English macro-F1 values of 84.35 in P1, 85.45 in P3, 86.63 in P4, and 85.52 in P5. Bangla, Hindi, and Urdu are often close to English in several settings; for example, GPT-4 achieves 84.35 on BN P1, 86.72 on HI P2, and 84.46 on UR P1. The largest reported gap is in P5 for Urdu, where EN is 85.52 and UR is 68.61 [2410.13153].

Llama-2-70b-chat shows the sharpest low-resource degradation on SIB-200. Its English P1 macro-F1 is 57.42, whereas Bangla, Hindi, and Urdu are 17.22, 8.20, and 5.14. The corresponding English advantages are +40.20, +49.22, and +52.28 [2410.13153].

Gemini Pro is intermediate. On SIB-200, its language differences are modest, and some low-resource settings approach or exceed English. A notable example is Urdu in P3, where macro-F1 reaches 84.10, compared with English 81.75 in the same setting [2410.13153]. This indicates that BMAS English is strongest as a default heuristic rather than an exceptionless rule.

## 4. Mechanisms proposed to explain the English advantage

The study attributes BMAS English primarily to asymmetries in pretraining data, tokenizer coverage, script handling, and instruction-following robustness. English-dominant pretraining is the first explanation. The paper states that LLM training data is dominated by English, and reports that Llama-2 reportedly uses approximately 90% English data. GPT-4 and Gemini are likewise described as heavily English-centric [2410.13153].

Tokenizer coverage is the second explanation. English subword coverage is strong, whereas Bangla, Hindi, and Urdu scripts are less represented, which can lead to fragmentation, longer sequences, and noisier representations. The paper further notes script complexity and morphological richness as additional burdens in zero-shot settings, with BN, HI, and UR posing more modeling difficulty than English in many contexts [2410.13153].

A third explanation concerns translation noise. Prompt translation with Google Translate may introduce subtle errors. At the same time, the evidence in the paper suggests that the translation origin of the prompt has limited impact relative to the broader English advantage, because translated prompts sometimes match or exceed original prompts. This suggests that prompt-language quality and model robustness interact in nontrivial ways, but do not overturn the overall English preference [2410.13153].

A fourth explanation is that zero-shot cross-lingual transfer is imperfect. The paper argues that instruction understanding transfers less well to low-resource scripts, and that models learn task format more reliably in the dominant language. A plausible implication is that BMAS English is partly an instruction-following phenomenon rather than solely a semantic-understanding phenomenon [2410.13153].

## 5. Output reliability, failure modes, and model-specific behavior

BMAS English is not only about higher macro-F1; it is also about output regularity. The study reports that LLM outputs often required post-processing to strip extra characters and extract labels via regular expressions. If no label was detected, the output was mapped to “None” and counted as a misclassification [2410.13153].

The most severe failure mode is instruction misinterpretation in low-resource prompts, especially for Llama-2. On XNLI, Llama-2 exhibits very high invalid label counts in low-resource settings, including 4,147 invalid labels for BN P1 and 4,966 for HI P2. The paper also reports a bias toward “Neutral,” with HI P2 Neutral F1 of 99.61 and HI P1 Neutral F1 of 89.48, suggesting superficial or default predictions rather than stable NLI behavior [2410.13153].

Gemini’s failure mode is different. The paper states that Gemini Pro frequently blocked prompts containing harmful or inappropriate content. Even after setting all harm category thresholds to BLOCK_NONE, occasional blocked outputs remained. Invalid-label counts are often moderate, but Urdu XNLI P1 shows a spike to 1,330 invalid labels [2410.13153].

GPT-4 is the most regular generator. The paper states that GPT-4 generally returns only the label and has few invalid labels; for example, XNLI EN P1 has invalid label count 0 [2410.13153]. This supports the stronger practical interpretation of BMAS English: English prompts improve not just task accuracy but also label-format compliance.

These failure patterns matter because the tasks are label-constrained classification tasks. A model that is semantically competent but format-unstable may still underperform after regex-based extraction. This suggests that BMAS English partly reflects reduced decoding pathology under English instructions [2410.13153].

## 6. Practical use and prompt-engineering implications

The paper’s practical recommendation is direct: if English instructions and English labels are acceptable, English prompts should be preferred. The expected gains are largest on XNLI, where English prompt advantages in P1 range from +17 to +22 macro-F1 points for GPT-4, +20 to +23 for Llama-2, and +12 to +27 for Gemini [2410.13153].

When localized prompts are necessary, the paper recommends bilingual prompts, English labels, and stronger output constraints. It specifically suggests bilingual prompts and code-switching, a glossary of key class names and definitions, and explicit output constraints such as “Answer with exactly one of: [labels].” It also recommends role conditioning for GPT-4, noting that the “system: expert annotator” framing improves reliability [2410.13153].

For strict classification tasks, the paper further recommends reducing verbosity by using lower temperature and narrower top-p when such controls are available, and setting max tokens very small for label-only outputs. Although decoding parameters were not reported in the experiments, this recommendation appears in the practical guidance section [2410.13153].

The study also proposes a robust cross-lingual pipeline centered on English instruction: write prompts in English; if inputs are in BN, HI, or UR, either keep inputs in the source language while constraining the model to output English labels, or machine-translate inputs to English for comprehension and back-translate outputs for user display if needed. Post-processing via regex extraction remains necessary, and mapping to “None” is treated as a last resort [2410.13153].

## 7. Limitations and interpretation

Several limitations constrain the scope of BMAS English. The evaluation is restricted primarily to NLI and one topic-classification dataset. Broader tasks such as question answering, machine translation, or summarization are not covered. The study is also zero-shot only; no few-shot prompting or fine-tuning is attempted, which may understate achievable low-resource performance [2410.13153].

Dataset composition is another limitation. Bangla XNLI is machine-translated rather than manually curated, and SIB-200 has a small test size. Gemini’s safety filtering may also confound some Urdu results, even after safety thresholds were set to BLOCK_NONE [2410.13153].

The evidence therefore supports a careful reading. BMAS English is a pragmatic default, not an absolute law. It is strongest for reasoning-heavy tasks such as XNLI, where the English advantage is large and consistent. On SIB-200, differences are smaller, and some low-resource settings approach or exceed English, particularly for Gemini and GPT-4 in selected prompt settings [2410.13153].

Taken as a whole, BMAS English identifies an English-first prompting regime for low-resource South Asian settings: use English instructions when possible, retain English class labels, add bilingual support only when required, and expect the largest benefits in tasks where instruction following and fine-grained semantic discrimination dominate performance [2410.13153].

Source: https://www.emergentmind.com/topics/bmas-english