BanglaMedQA: Biomedical MCQ Benchmark

• The paper introduces BanglaMedQA, a benchmark of 1,000 authenticated Bangla medical MCQs that evaluates both answer accuracy and rationale quality in biomedical AI.
• BanglaMedQA is built from real Bangladeshi MBBS, BDS, and AFMC exam questions using advanced OCR and rigorous curation processes.
• The benchmark facilitates testing diverse Retrieval-Augmented Generation strategies, with Agentic RAG achieving 89.54% accuracy and leading rationale metrics.

BanglaMedQA is a Bangla biomedical question answering benchmark consisting of 1,000 unique Multiple-Choice Questions (MCQs) curated from authentic Bangladeshi medical admission examinations and introduced together with BanglaMMedBench to evaluate reasoning and retrieval in medical artificial intelligence for a low-resource language setting. The benchmark is designed around Single Best Answer MCQs with Bangla rationales and is used to study Retrieval-Augmented Generation (RAG) strategies, including Traditional, Zero-Shot Fallback, Agentic, Iterative Feedback, and Aggregate RAG. In the reported experiments, Agentic RAG achieved the highest accuracy, 89.54%, with openai/gpt-oss-120b, while also attaining the strongest rationale-quality metrics among the evaluated configurations (Sultana et al., 6 Nov 2025).

1. Origin, definition, and benchmark role

BanglaMedQA was assembled from authentic Bangladeshi medical admission tests, specifically MBBS, BDS, and AFMC examinations spanning 1990–2024. Its explicit purpose is to support evaluation of Bangla biomedical QA systems under conditions where both domain specificity and language resource scarcity matter. The paper characterizes BanglaMedQA and BanglaMMedBench as the first large-scale Bangla biomedical MCQ datasets designed to evaluate reasoning and retrieval in medical AI (Sultana et al., 6 Nov 2025).

The benchmark is defined at the level of individual MCQ instances. Each item contains a surface-level factual question $q$, a set of four options $O=\{A,B,C,D\}$, one correct answer $O^*\in O$, and a human-written rationale $r$ in Bangla. The question format is Single Best Answer, and the annotation guidelines emphasize uniform option formatting, clarity of the question stem and rationale, and consistency of terminology with Bangla medical curricula.

This structure makes BanglaMedQA simultaneously a classification benchmark and an explanation benchmark. The answer label supports accuracy-based evaluation, while the Bangla rationale supports semantic and lexical evaluation of generated explanations. A plausible implication is that the dataset was designed not only to measure option selection but also to probe whether retrieval materially improves grounded biomedical reasoning in Bangla.

2. Construction pipeline and curation criteria

The raw collection comprised 1,200 MCQs scanned from printed exam papers. Text digitization began with an OCR pipeline. Tesseract OCR was found to struggle with Bangla conjunct characters, or juktakkhor, and Google Lens OCR was therefore adopted for higher fidelity. Post-OCR manual inspection corrected misalignments, formatting issues, and missing text (Sultana et al., 6 Nov 2025).

The filtering pipeline was explicit and conservative. Five ambiguous questions with multiple correct answers were removed. Eleven items with nonstandard option labels, such as Bangla letters or numeric labels, were discarded. A further 184 repeated items were removed during deduplication. These steps yielded a final dataset of 1,000 unique MCQs.

The published release is provided as a single 1,000-question split. The paper states that users may adopt standard train/validation/test allocations, for example 80/10/10. This means the release is benchmark-ready but not prescriptive about downstream experimental partitioning. That choice is consistent with its stated use as an evaluation resource rather than as a fixed supervised training corpus.

3. Domain coverage and representational characteristics

BanglaMedQA covers foundational biomedical topics aligned with Higher Secondary and medical admission syllabi. The reported domains are anatomy, physiology, biochemistry, pathology, pharmacology, and microbiology. This coverage locates the benchmark near the interface between preclinical medical education and admission-level screening.

The question formulation is described as surface-level factual. That characterization is important for interpreting the benchmark: the tasks are not presented as long-form clinical case resolution or open-ended diagnosis, but as structured MCQ answering with one correct option and a rationale. At the same time, the benchmark is used to test retrieval and reasoning pipelines, so factuality alone does not eliminate the need for strategy selection, grounding, and answer explanation.

Because the dataset is sourced from admission examinations over more than three decades, it reflects a standardized educational lineage rather than ad hoc synthetic prompting. This suggests a form of curricular authenticity: the benchmark operationalizes biomedical knowledge in the form in which it appears in high-stakes Bangladeshi assessment.

4. Retrieval-augmented generation strategies

All evaluated pipelines use a dense-embedding similarity scorer based on SBERT embeddings:

$$\mathrm{score}(q,d)=\frac{\langle E_q,E_d\rangle}{\|E_q\|\;\|E_d\|},$$

where $E_q$ and $E_d$ are SBERT embeddings for the query and document chunk (Sultana et al., 6 Nov 2025).

The benchmark study evaluates several RAG variants. Traditional Local RAG retrieves top-$k$ passages from a FAISS index of the Bangla Biology textbook, prompts the LLM with $(q,O,C)$, and generates both an answer $\hat O$ and a rationale $O=\{A,B,C,D\}$0. Local RAG with Zero-Shot Fallback introduces a threshold on retrieved character length and falls back to direct zero-shot answering when local context is insufficient. Web Search with Zero-Shot Fallback queries the Serper API for up to 8 links, condenses the top 3 into $O=\{A,B,C,D\}$1, and uses web-grounded answering only if the retrieved material exceeds a character threshold.

Agentic RAG introduces an LLM router that dynamically selects among Local RAG, Web RAG, and zero-shot inference. Its policy is:

$O=\{A,B,C,D\}$2

The router first evaluates whether $O=\{A,B,C,D\}$3 is sufficient; if not, it attempts web retrieval; if neither retrieval source is adequate, it defaults to zero-shot answering. This routing logic is the paper’s central methodological novelty.

Two additional strategies broaden the comparison space. Iterative Feedback RAG operates in four phases: retrieve context, produce a one-shot answer, extract key terms and reformulate the query if the answer is incorrect, and repeat up to 2 iterations while assigning High, Medium, or Low confidence. Aggregate $O=\{A,B,C,D\}$4-Values RAG retrieves with $O=\{A,B,C,D\}$5, predicts an answer for each context size, and returns the majority vote, with ties broken by $O=\{A,B,C,D\}$6. Together these methods operationalize different views of how retrieval should interact with Bangla biomedical answering: fixed retrieval, fallback-based retrieval, routed retrieval, correction-oriented retrieval, and retrieval ensembling.

5. Experimental configuration and evaluation protocol

The evaluated LLMs were llama-3.3-70b-versatile, llama-3.1-8b-instant, openai/gpt-oss-20b, and openai/gpt-oss-120b. Semantic retrieval used BengaliSBERT embeddings. The local knowledge base was the Bangla Higher Secondary Biology textbook, OCR’d and chunked into 1,000-character units with 200-character overlap, then indexed in FAISS. Web retrieval used the Serper API with HTML parsing and summarization (Sultana et al., 6 Nov 2025).

The main hyperparameters were also fixed and reported: retrieval $O=\{A,B,C,D\}$7 for Local RAG, thresholds $O=\{A,B,C,D\}$8 characters for textbook retrieval and $O=\{A,B,C,D\}$9 characters for web retrieval, up to 2 iterations for Iterative Feedback RAG, and aggregate $O^*\in O$0-values of $O^*\in O$1.

The primary evaluation metric for answer correctness was accuracy:

$O^*\in O$2

Rationale quality was evaluated with BERTScore (F1), METEOR, ROUGE-1/2/L, and BLEU. In the reported framing, BERTScore measures contextual semantic overlap, METEOR uses a precision-recall formulation with a fragmentation penalty, ROUGE-1/2/L captures $O^*\in O$3-gram overlap and longest common subsequence, and BLEU is reported via the standard brevity-penalized geometric mean of modified precision terms. This metric suite treats BanglaMedQA as a benchmark for both discrete answer selection and semantically faithful explanation generation.

6. Reported results and analytical findings

For openai/gpt-oss-120b, the paper reports the following selected accuracies on BanglaMedQA:

Strategy	Accuracy
Zero-Shot	82.60%
Local RAG	86.32%
Local + Zero-Shot Fallback	87.22%
Web + Zero-Shot Fallback	88.83%
Agentic RAG	89.54%
Iterative Feedback RAG	88.73%
Aggregate $O^*\in O$4-Values RAG	84.51%

Agentic RAG also achieved the top rationale-quality scores reported in the study: BERTScore 0.7274, ROUGE-1 0.0897, and ROUGE-L 0.0870, alongside competitive METEOR 0.1136 and BLEU-2 0.0565 (Sultana et al., 6 Nov 2025).

The paper’s analysis identifies several patterns. Incorporating retrieval through Local RAG boosts accuracy by approximately 4–8% over zero-shot. Zero-shot fallback reduces null responses and raises coverage. Web retrieval improves factual grounding for generic questions. Agentic RAG’s dynamic strategy produces the highest overall accuracy and the strongest rationale metrics. Iterative Feedback RAG provides interpretable confidence scores, but the gains vary by model capacity. Aggregate $O^*\in O$5-Values RAG stabilizes performance but does not surpass Agentic RAG.

An example rationale is provided for the translated question, “Which vitamin deficiency causes night blindness?” The reported Agentic RAG output selects answer C, Vitamin A, with the Bangla rationale: “রেটিনাল সংশ্লেষণের জন্য ভিটামিন A অপরিহার্য; এর অভাবে রেটিনায় আলোকে রূপান্তর ঘটতে পারে না, ফলে রাতকালে দৃষ্টিশক্তি কমে যায়।” This example is used in the paper to illustrate the explanatory style associated with the strongest-performing configuration.

7. Significance, limitations, and relation to adjacent Bangla medical benchmarks

The study presents BanglaMedQA as the first large-scale Bangla biomedical MCQ benchmark and argues that RAG methods, especially Agentic RAG, substantially improve answer accuracy and explanation quality in a low-resource language setting. Its recommendations for future work are concrete: add validation and test splits, extend the dataset with scenario-based questions and rationales, OCR and curate additional Bangla medical textbooks and guidelines, incorporate multilingual retrieval between Bangla and English, enhance document reranking, use adaptive thresholds, and further fine-tune LLMs on Bangla medical corpora (Sultana et al., 6 Nov 2025).

A useful point of comparison is BanglaMedVQA, which addresses Bangla medical visual question answering rather than text-only MCQ answering. That benchmark contains 2,000 image–question–answer instances and reports substantially lower Bangla performance than comparable English evaluation, especially on specialized diagnostic questions. Even the best model under chain-of-thought prompting reaches only about 23%–25% Accuracy on Condition and Position in Bangla, which the paper describes as effectively random (Ahmed et al., 18 May 2026). Read together, the two benchmarks indicate that Bangla medical QA research is diverging into at least two technically distinct regimes: text-based MCQ answering, where retrieval and routing substantially improve performance, and medical visual question answering, where current models still struggle with clinically fine-grained reasoning.

This comparison also helps delimit what BanglaMedQA measures. It is a benchmark for biomedical factual question answering with controlled answer spaces and textual rationales, not a benchmark for multimodal diagnostic inference. Its reported success with Agentic RAG therefore demonstrates the value of retrieval and strategy selection for Bangla medical text QA, while not implying resolution of the broader challenges documented in Bangla medical vision-language evaluation.