Native Question Curation

• Native Question Curation is a systematic approach for gathering authentic, culturally grounded QA datasets sourced from native speakers using local queries and iterative validation.
• It employs a mix of manual annotation, template-based query generation, search engine expansion, and rigorous deduplication to ensure region-specific relevance.
• This method enhances large language model benchmarking by mitigating cultural bias and supporting multilingual performance through high-fidelity, diverse QA corpora.

Native Question Curation refers to systematic, scalable methods for collecting, validating, and curating question–answer (QA) datasets that authentically capture how real users from diverse regions and linguistic communities naturally pose everyday, culturally specific questions. This paradigm is critical for robust LLM benchmarking, mitigating cultural bias, and enabling fine-tuning in underrepresented, multilingual contexts. Native question curation leverages localized seed queries, search engine expansion, human annotation, and domain-level reliability mechanisms to yield high-fidelity, location- and language-aligned QA corpora at scale (Alam et al., 8 Apr 2025, Hasan et al., 2024, Perevalov et al., 2022, Mustapha et al., 2024, Keraron et al., 2020).

1. Foundations and Scope

Native question curation differentiates itself from direct translation or synthetic data generation by prioritizing source questions that arise organically from native speakers in their daily context and idiom. Its central objective is to produce benchmarks and resources that reflect true-to-life patterns of information seeking, phrasing, and local knowledge—particularly vital for low-resource or geographically isolated languages where conventional datasets are lacking or insufficient (Hasan et al., 2024, Alam et al., 8 Apr 2025).

Key features include:

• Local grounding: Queries are seeded from native speakers—covering region-dependent events, traditions, and local issues.
• Multilingual, cross-cultural reach: Applied to high-, medium-, and extremely low-resource languages.
• Automatic expansion: Search engines (e.g., Google, Bing) provide further related queries and answer snippets via “People also ask” APIs.
• Iterative filtering, validation, and deduplication: Ensures dataset quality, reduces redundancy, and adapts to local content availability.

2. Pipeline Architecture and Methodology

State-of-the-art frameworks, such as NativQA, implement native question curation through a modular pipeline comprising the following stages (Alam et al., 8 Apr 2025):

Query Collection (QC)

Seed queries $Q_0$ are gathered via:

• Manual annotation: Native speakers pose natural, locally relevant questions.
• Template-based generation: E.g., “What are [topic] in [region]?”
• LLM-based expansion: Automated paraphrasing for increased diversity.

Duplicate filtering employs both exact string matching and near-duplicate detection using Sentence-BERT embeddings. Queries with cosine similarity above a threshold $\tau$ are removed:

$$\text{sim}(q_1, q_2) = \cos(\text{emb}(q_1), \text{emb}(q_2))$$

QA Collection (QAC)

An iterative process expands $Q_0$ by:

• Querying search engine APIs with geo- and language-specific codes.
• Extracting $(q', a', \text{URL})$ triples and related queries.
• Deduplicating newly generated entries at every iteration.
• Optionally applying Maximal Marginal Relevance (MMR) to maximize topical diversity:

$$\text{MMR}(q_i) = \lambda \cdot \text{Sim}(q_i, Q_0) - (1-\lambda) \cdot \max_{q_j \in \text{Selected}} \text{Sim}(q_i, q_j)$$

QA Validation (QAV)

• Domain Reliability Checking (DRC): Only QAs from pre-validated “very reliable” domains (by majority vote) are retained.
• QA Annotation (QAA): Manual or LLM-based processes to:
  • Validate question clarity and answerability.
  • Edit answers for factual correctness and completeness.
  • Label QA pairs for location and topic relevance.

Data Aggregation

The finalized dataset $S_{QA}$ is split by location, language, and topic; exported in standardized formats for subsequent model benchmarking and fine-tuning.

3. Sampling, Annotation, and Quality Assurance

Native question curation protocols establish rigorous annotation and validation workflows:

• Recruitment: Native speakers with region-appropriate backgrounds generate original queries or translate/validate existing ones, as in QALD-9-plus (Perevalov et al., 2022).
• Annotation guidelines: Detailed language-specific instructions ensure idiomatic phrasing, semantic fidelity, and syntactic naturalness. Annotators are trained to avoid over-literalization, preserve referential integrity, and maintain question scope.
• Validation pipelines: Frequently dual- or triple-blind assessments, ensuring majority or consensus agreement for translation/adaptation tasks. For example, QALD-9-plus required at least two independent translations per question and validation by a third annotator (Perevalov et al., 2022). AraSTEM required multi-stage proofreading and span agreement (Mustapha et al., 2024).
• Metrics: Precision, recall, F1-score for QA validation; inter-annotator agreement using metrics such as observed agreement, Cohen’s κ, Gwet’s AC1, and Fleiss’ κ.

Explicit dataset curation choices—e.g., topic lists, degree of manual oversight, level-difficulty calibration—are critical to downstream applicability, as demonstrated by AraSTEM (curriculum-aligned MCQ curation) (Mustapha et al., 2024) and Project PIAF (certified vs. open splits, multi-answer validation) (Keraron et al., 2020).

4. Empirical Benchmarks and Datasets

Major native-curated resources exhibit extensive geographic and linguistic range:

Paper/Framework	Languages	QA pairs	Locations/Countries	Notable Features
NativQA (Alam et al., 8 Apr 2025)	Arabic, English, Hindi, Bangla, Turkish, Assamese, Nepali	>300K	39/24	Search-driven, iterative, 18 topics, DRC, LLM QAA
MultiNativQA (Hasan et al., 2024)	7 (see above)	72,022	9/7	Native and LLM-seeded, stratified by topic, region
QALD-9-plus (Perevalov et al., 2022)	9 (German, French, Russian, etc.)	~2500	--	Native-translated KGQA, double annotation, Wikidata SPARQL
AraSTEM (Mustapha et al., 2024)	Arabic	11,637	--	Native MCQ by subject/level, scientific Arabic
PIAF (Keraron et al., 2020)	French	3,835 (certified split)	--	Wikipedia passages, 285 annotators, extractive QA

All frameworks apply stratified sampling by language, region, and topic to maximize representativeness of local knowledge and question style. For instance, in NativQA, the average number of QA pairs per seed is 8–12, with Arabic comprising ~40% of total QAs, and English ~25% (Alam et al., 8 Apr 2025).

5. Evaluation, Metrics, and Model Performance

Comprehensive evaluation protocols benchmark both data quality and model performance:

• QA validation precision/recall: In NativQA, question quality precision is 0.88, answer quality precision is 0.85, yielding $F_1 = 0.86$ on held-out subsets (Alam et al., 8 Apr 2025).
• Closed/open LLM benchmarking: On MultiNativQA, GPT-4o achieves the highest BLEU (avg 0.278); open models like Mistral outperform Llama-3 in low-resource settings (Hasan et al., 2024). Performance strongly correlates with language resource status and digital content richness.
• Inter-annotator reliability: Consistently reported across datasets; e.g., QAA observed agreement 0.842 (NativQA), Fleiss’ κ 0.53–0.66 depending on language in MultiNativQA (Hasan et al., 2024, Alam et al., 8 Apr 2025).
• Model F1 drop: PIAF demonstrates that natively authored questions yield greater challenge for French models (F1 drop ≈ 18 points vs FQUAD) (Keraron et al., 2020).

6. Best Practices, Challenges, and Extensions

Sustained high accuracy, diversity, and cultural alignment depend on:

• Seed query diversity: Mix of manual, template-generated, and LLM-synthesized queries (Alam et al., 8 Apr 2025).
• Caching and cost constraints: Use of query/result caching to minimize API costs and avoid latencies (Alam et al., 8 Apr 2025).
• Low-resource adaptation: Manual review of LLM outputs; bilingual annotator training; back-translation for QA validation (Hasan et al., 2024).
• Domain-reliability lists: Static and evolving inventories of “trusted” answer sources reduce propagation of noisy web content.
• Iterative expansion: Captures the long-tail of location- and culture-specific queries.
• Quality-control tools: Adoption of annotation UIs and consensus-based automated pipelines.

Extensions under exploration include support for multimodal (audio/video/image) native queries, dynamic trust-scoring models, integration of annotation platforms (e.g., Label Studio), cross-lingual QA retrieval, and few-shot fine-tuning of validator LLMs (Alam et al., 8 Apr 2025).

7. Comparative Perspectives and Utility

Native question curation must be distinguished from translation-based approaches and crowd-sourced QA methods that do not center region- and culture-specific knowledge needs. Key distinctions include:

• Authenticity of linguistic and cultural features: QALD-9-plus explicitly prohibits use of machine translation and mandates that annotators preserve named entities, scope, and semantics faithfully (Perevalov et al., 2022).
• Dual/multi-annotation and validation: Ensures both linguistic and factual accuracy as well as pragmatic relevance for target communities.
• Cross-lingual applicability and transfer: Datasets such as QALD-9-plus, AraSTEM, and MultiNativQA facilitate evaluation and fine-tuning of LLMs for natural question style in rarely supported or endangered languages, boosting accessibility and fairness for knowledge tasks.

Native question curation has become foundational for next-generation QA benchmarks, culturally-responsive LLM evaluation, fair model selection, and, ultimately, for democratizing intelligent systems across the world’s linguistic and geographic diversity (Alam et al., 8 Apr 2025, Hasan et al., 2024, Perevalov et al., 2022, Keraron et al., 2020, Mustapha et al., 2024).