EverydayMMQA: Culturally Grounded Multimodal QA

• EverydayMMQA is a framework that combines speech, text, and visual inputs to provide culturally grounded answers tailored to local contexts.
• It employs a seven-module pipeline including culturally aware query generation, country-localized image retrieval, QA synthesis, and quality control.
• The framework is validated with the OASIS dataset, revealing that image-guided inputs significantly enhance performance across diverse languages and dialects.

Everyday Multimodal and Multilingual QA (EverydayMMQA) is a framework for constructing and evaluating culturally grounded multimodal question answering systems for everyday life, introduced together with the OASIS dataset in "EverydayMMQA: A Multilingual and Multimodal Framework for Culturally Grounded Spoken Visual QA" (Alam et al., 7 Oct 2025). It targets spoken visual QA (SVQA), a setting in which the input may include speech and/or images and the model produces text output. The framework is motivated by the claim that effective multimodal QA in real environments requires reasoning over an object–question–culture triplet in a specific local context, rather than only recognizing visual entities or solving generic VQA-style reasoning problems.

1. Conceptual scope and problem setting

EverydayMMQA is defined around a failure mode of conventional VQA and multimodal QA benchmarks: they mainly test general scene understanding, including object recognition, attributes, and basic visual reasoning, but do not adequately test culturally grounded everyday knowledge (Alam et al., 7 Oct 2025). The framework formalizes the observation that real user interactions often combine visual reference, spoken or written language, and local cultural knowledge. In this setting, a system may need to interpret customs, religious practices, local cuisine, social norms, dialect, and country-specific routines in order to answer correctly.

The paper treats this contextual knowledge as non-universal. Answers that are appropriate in one country or dialectal context may be irrelevant or offensive in another. This emphasis is especially strong for Arabic, where the paper highlights Modern Standard Arabic (MSA), Egyptian Arabic, and Levantine Arabic, together with substantial dialectal diversity and country-level cultural differences across the Arabic-speaking world. A concrete implication is that strong performance on standard VQA does not guarantee reliable performance on questions grounded in region-specific practice.

This problem formulation also distinguishes EverydayMMQA from prior cultural benchmarks that focus on text-only or image-text settings. The paper explicitly identifies the lack of spoken queries as a major gap for real-world multimodal grounding. EverydayMMQA therefore elevates speech to a first-class modality rather than treating it as an auxiliary channel.

2. Framework architecture

EverydayMMQA is presented as a scalable, country-localized framework for building culturally sensitive multimodal and multilingual QA resources (Alam et al., 7 Oct 2025). Its pipeline is organized into seven modules: culturally grounded topic and query generation, country-localized image retrieval, image filtering and metadata generation, QA generation, speech generation and recording, translation, and quality checking.

The framework supports four input configurations:

• text-only
• speech-only
• text+image
• speech+image

The paper also uses ASR transcripts, enabling evaluation not only with raw speech but also with transcription-derived settings. This preserves a distinction between direct spoken input and text obtained after recognition and normalization.

At the task-definition level, EverydayMMQA treats SVQA as a multimodal interaction problem in which a user may ask a question orally while referring to an image, with the model required to output text. The framework is described as language-independent in principle, although the instantiated system in the paper focuses on English and Arabic varieties. The country-localized design is central: prompts are specialized for local context, and the framework is intended to encode cultural sensitivity rather than only linguistic translation.

3. OASIS dataset

Using EverydayMMQA, the authors construct OASIS, a large multimodal and multilingual dataset intended both as a benchmark and as training data for culturally grounded multimodal QA (Alam et al., 7 Oct 2025). OASIS integrates speech, images, and text, and is structured so that each image supports multiple QA formats.

Dimension	Reported value	Notes
Images	approximately 0.92M	OASIS-scale figure
QA pairs	14.8M	multimodal and multilingual
Spoken questions	3.7M	spoken QA coverage
Synthesized audio	~20,279 hours	English and MSA
Human-recorded audio	141 hours	subset of test items

Each image has four QA items: one open-ended, one multiple-choice, and two true/false. The supported modality settings are speech-only $(S)$, text-only $(T)$, speech+image $(S+I)$, and text+image $(T+I)$. In benchmarking, speech is also examined through transcripts, giving $Tr$ and $Tr+I$.

Language coverage spans four varieties: English, MSA, Egyptian Arabic, and Levantine Arabic. Country coverage spans 18 Arab countries: Algeria, Bahrain, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Morocco, Oman, Palestine, Qatar, Saudi Arabia, Sudan, Syria, Tunisia, United Arab Emirates, and Yemen. The paper states that country coverage is balanced but not perfectly uniform, with total samples ranging from about 36K for Qatar to 64K for Morocco and a median of about 53K.

The source also reports that the image-filtering pipeline leaves about 1.30M photographs, whereas OASIS is summarized at approximately 0.92M images. This suggests different accounting stages in the pipeline rather than a single image count.

4. Data construction and quality control

The curation pipeline is heavily automated but includes human review at several stages (Alam et al., 7 Oct 2025). It begins with a culturally grounded taxonomy of 9 categories $\mathcal{G}$ and 31 subcategories. For each country $c$ and each subcategory $s_j$, an LLM generates 10 topics,

$\mathcal{T}_{c,j}=\{t_{j,1},\ldots,t_{j,10}\},$

so that

$(T)$0

These topics are manually reviewed and revised to remove generic or irrelevant items.

Image-search queries are then generated using GPT-4.1, Gemini-2.0-flash-001, and Claude-3.5-Sonnet. The queries are designed to resemble realistic user behavior, including typos, slang, informal phrasing, incomplete or autocomplete-like queries, and country-specific dialectal variation. The merged output yields approximately 6,100 unique queries per country. A cultural relevance filter based on GPT-4o assigns a score in $(T)$1, and only queries with scores at least 80 are retained. The ablation study reports 5,580 total seed topics, 5,445 manually verified topics, 110,126 generated queries, and 97,678 retained queries; manual verification removed 135 topics and relevance filtering discarded about 12.5K queries.

Country-localized image retrieval uses Google Custom Search with locale settings, safe search, and license restrictions cc_publicdomain, cc_attribute, and cc_sharealike. For each query, the top 20–40 results are kept subject to minimum resolution and standard file formats. This phase retrieves about 4.3M candidate images across all countries. After URL deduplication, about 2.4M unique entries remain, of which about 1.4M are successfully downloaded. Exact and near-duplicate filtering is then performed using a fine-tuned ResNet18 with Euclidean distance threshold

$(T)$2

yielding about 1.35M unique images.

The metadata stage removes non-representative content such as text-overlaid graphics, advertisements, charts, screenshots, and memes. Images are labeled with

$(T)$3

and assigned suitability labels

$(T)$4

The labeling function is formalized as

$(T)$5

with

$(T)$6

Only images labeled photograph are retained:

$(T)$7

For each retained image, QA generation produces four items,

$(T)$8

where $(T)$9 is the question, $(S+I)$0 the answer, $(S+I)$1 the rationale, $(S+I)$2 the semantic label, and $(S+I)$3 the cognitive label. The semantic labels are organized into 11 semantic categories, including location, architecture, and cultural heritage. The cognitive labels distinguish knowledge-based from commonsense-based questions. The full dataset is reported as 51.9% knowledge-based and 48.1% commonsense-based; open-ended questions are mostly knowledge-based at 72.9%, while true/false questions are mostly commonsense at 57.5%.

Speech is incorporated through both synthesis and recording. Synthetic speech uses XTTS-v2 for English and MSA. Reference voices are built from 337 English segments from LibriTTS sampled from 35 speakers, and from 389 Arabic segments from QASR together with 69 manually reviewed segments from ADI17. For each question, the system generates three English audio samples and one MSA sample. The paper explicitly notes that dialectal TTS is limited, so speech coverage is stronger for English and MSA than for dialectal Arabic. Human recordings cover a benchmark subset and amount to about 141 hours from 26 speakers, 13 per language, with average duration about 5 seconds.

Translation is performed from English to MSA using an in-house LLM-based system. For dialects, the paper compares direct GPT-4.1 translation against a two-step English $(S+I)$4 MSA $(S+I)$5 dialect pipeline and reports that direct GPT-4.1 translation performs better overall, though some Egyptian test sets benefit from the intermediate MSA step.

Quality control has two layers. Human annotation covers a sample of 3,000 QAs across six countries, yielding about 24K annotations from 3,000 samples, four QAs per image, and two annotators. Questions are rated on a five-point Likert scale, true/false answers on a three-point scale, and rationales on clarity/informativeness and plausibility/faithfulness. LLM-based annotation uses Gemini-2.5-Pro and Llama-4-Scout-17B-16E-Instruct to score the complete test split of 34,930 samples across all 18 countries. Agreement is measured with $(S+I)$6, and the paper reports strong agreement, including near-perfect true/false answer consistency, very strong MCQ agreement, and slightly lower but still high agreement for open-ended items. Reported LLM-based annotation scores include open-ended answer quality 4.680, MCQ answer quality 4.823, true/false answer quality 2.898–2.945, and many $(S+I)$7 values around 0.95–0.99. Audio quality is evaluated with WER, speaker cosine similarity, and NISQA; reported values include synthetic English WER 6.19, synthetic MSA WER 37.18, synthetic English NISQA 4.33, and synthetic MSA NISQA 3.68.

5. Benchmark protocol

The benchmark evaluates both closed and open models under zero-shot conditions (Alam et al., 7 Oct 2025). The closed-source systems are GPT-4.1, GPT-4o-audio, GPT-5, and Gemini-2.5-Pro. The open-source systems are Qwen2.5-7B-Omni, Qwen2.5-3B-Omni, and Phi-4. In addition, Qwen2.5-3B-Omni is fine-tuned with LoRA.

Evaluation covers text $(S+I)$8, speech $(S+I)$9, text+image $(T+I)$0, speech+image $(T+I)$1, and the transcript settings $(T+I)$2 and $(T+I)$3. Each item is evaluated as open-ended generation, multiple-choice, and two true/false variants. Because human recordings cover only a subset of the test set, speech evaluation is correspondingly limited to those items.

The fine-tuning setup for Qwen2.5-3B uses LoRA with $(T+I)$4, $(T+I)$5, learning rate $(T+I)$6, maximum sequence length 3072, and one epoch. Due to compute limits, only 6.67% of the training split is used, amounting to about 2.75M datapoints out of roughly 41.26M possible training datapoints.

The reported metrics are BERTScore F1 for semantic similarity, GPT-4.1 used as LLM-as-judge for open-ended answers on a 1–10 rubric covering helpfulness, relevance, accuracy, and faithfulness, and accuracy for MCQ and true/false. The benchmark design therefore separates constrained-answer evaluation from free-form answer quality and explicitly treats judge-based open-ended assessment as an important measurement dimension.

6. Empirical findings and analytical results

The central empirical result is that visual grounding is the dominant lever across models, languages, and dialects (Alam et al., 7 Oct 2025). Adding images produces large gains in performance, and the effect is especially strong for the weaker modalities. The paper reports that MCQ and true/false tasks often reach near-ceiling accuracy when images are present, while open-ended judge scores improve substantially more than BERTScore. It also reports that cross-lingual and dialectal gaps shrink when images are added, and that fine-tuning substantially improves the smaller omni model, especially for speech inputs.

The text-only versus text+image contrast is illustrated with explicit scores. For GPT-4.1, English MCQ accuracy rises from 0.82 in text-only to 0.98 in text+image, and MSA MCQ accuracy rises from 0.75 to 0.96. Open-ended judge scores for GPT-4.1 English rise from 6.26 in text-only to 8.60 in text+image. The paper states that the same pattern appears for GPT-5, Gemini, Qwen, and Phi-4.

Speech-only performance is substantially weaker than text-only performance. The explanation given is a “two-stage noise stack”: acoustic noise affects raw speech, and ASR plus normalization errors affect transcripts. Accordingly, $(T+I)$7 generally outperforms both $(T+I)$8 and $(T+I)$9. However, adding images recovers much of this gap, and the paper emphasizes that $Tr$0 is often the largest gain for the weakest settings. Its interpretation is that images act as a modality equalizer by anchoring entities and attributes and thereby reducing the “language burden” created by morphology, orthographic variation, and data sparsity in Arabic varieties.

The comparison between model classes shows that closed models generally outperform open models in text-only settings. The paper suggests that this may reflect broader pretraining, stronger priors, and instruction tuning, while also warning that some apparently correct text-only answers may come from priors rather than grounding, which creates hallucination risk. On evaluation methodology, the authors argue that MCQ and true/false saturate quickly, BERTScore is too surface-level, and LLM-as-judge is the most informative metric for open-ended grounded QA.

Several ablations refine this picture. The topic/query-generation ablation quantifies the effect of manual verification and relevance filtering, showing that noisy or weakly grounded material is removed before image retrieval. The semantic–cognitive balance analysis shows that the benchmark is neither purely factual nor purely commonsense-driven. The audio analysis indicates that synthetic English speech is strong while synthetic MSA is weaker, and that natural human recordings produce high WER because of real-world noise and accent variation. The translation analysis finds that direct GPT-4.1 translation is better overall than a two-step English $Tr$1 MSA $Tr$2 dialect pipeline, although some Egyptian subsets benefit from the intermediate MSA step, while Levantine especially prefers direct translation. The fine-tuning analysis shows that Qwen2.5-3B-Omni improves substantially over its base version, particularly in speech-only, transcript-only, and speech+image settings.

7. Limitations, ethics, and research significance

The paper identifies compute constraints as a major limitation: the fine-tuned model is trained on only 6.67% of the training split rather than the full dataset (Alam et al., 7 Oct 2025). It also notes that speech generation is strong only for English and MSA, with limited dialectal TTS coverage; that the benchmark is centered on English and Arabic varieties rather than being globally multilingual in the broadest sense; that coverage across 18 Arab countries still cannot fully represent all local nuance; and that restricting the final image set to photographs may exclude culturally relevant non-photographic material.

The ethical discussion emphasizes licensing and participant treatment. Images were collected under public-use licensing. Annotators and recording participants were compensated at standard hourly rates and were briefed in advance. A broader ethical concern is cultural representation itself: systems that are not grounded in the relevant local context may produce irrelevant or offensive outputs, especially in culturally sensitive domains.

Within multimodal QA research, EverydayMMQA and OASIS position cultural grounding and visual grounding as complementary rather than interchangeable requirements. A plausible implication is that model scale alone is insufficient for reliable everyday QA in multilingual and dialectally diverse settings. The reported results further suggest that images can compensate for language weakness, that localized query generation is necessary for culturally specific benchmarking, that speech should be treated as a realistic interaction modality rather than an afterthought, and that compact multimodal models can become competitive when trained on culturally grounded data. In that sense, EverydayMMQA functions both as a benchmark construction methodology and as a design argument for multimodal systems that must operate in situated cultural environments.