Talking-NSQ: Neural Spoken Query Systems

• Talking-NSQ is a domain that utilizes neural approaches to process free-form spoken queries by integrating automatic speech recognition with language comprehension.
• The methodology involves cascaded and cascade-less architectures, leveraging datasets like SpokenNativQA and Non-Native SQA to evaluate performance using metrics such as WER, TOS, and AOS.
• Practical insights include fine-tuning ASR for accent and non-native variability, incorporating disfluency-augmented training, and using rigorous evaluation protocols to enhance system robustness.

Talking-NSQ refers to the domain of Natural Spoken Query (NSQ) systems, which process, understand, and answer free-form spoken questions using neural approaches rooted in large-scale automatic speech recognition (ASR), language modeling, and machine comprehension. Talking-NSQ encompasses dataset construction, end-to-end cascaded and cascade-less model architectures, error analysis, multilingual and non-native settings, and fine-grained evaluation of performance under realistic conditions.

1. Core Concepts and Definitions

In the context of question answering (QA) and conversational AI, Talking-NSQ systems accept spoken language input (queries or responses), aiming to extract or generate relevant answers. The term covers two closely related technical patterns:

• Spoken Question Answering (SQA): Extracting answers from audio queries, optionally requiring passage retrieval.
• Spoken Answer Extraction: Locating spans or generating responses to spoken prompts, typically for language assessment or knowledge retrieval.

Talking-NSQ research addresses the challenges arising from naturalistic speech variability, ASR-induced transcription errors, grammatical disfluency (notably in non-native speech), multilingual and dialectal diversity, and the evaluation and feedback mechanisms necessary for real-world deployment (Alam et al., 25 May 2025, Raina et al., 2021).

2. Dataset Construction and Characteristics

High-quality datasets are foundational for training and evaluating Talking-NSQ systems. Key datasets and procedures include:

SpokenNativQA (Alam et al., 25 May 2025)

• Multilingual dataset comprising ~33,000 naturally spoken question–answer pairs in Arabic (Qatar) and English (Qatar), recorded across 18 topics (e.g., Animals, Food & Drink, Religion).
• Realistic variability: Unconstrained recording platforms, diverse microphones, inclusion of L1 and L2 speakers, and variability in accents and background noise.
• Test set curation ensures topic and speaker balance, capturing over 3.35 s per utterance on average.

Non-Native SQA Corpus (Raina et al., 2021)

• Spoken responses from Linguaskill Business English assessment, covering CEFR A1–C2, with manually-aligned answer spans, explicit error annotation (ERRANT), and non-native speaker diversity.
• Provides 947 prompt–response pairs, with granular grammatical error rates and hand-marked answer locations.

Implication: Real-world dataset collection for Talking-NSQ requires careful recruitment, fully operationalized recording platforms, and explicit handling of linguistic/diversity variables. This enables multi-accent, code-switched, and dialectal scenario evaluation, critical for robust deployment.

3. ASR Integration and Error Analysis

Talking-NSQ systems are heavily influenced by the performance and properties of the front-end ASR module. Key observations include:

• ASR choice and tuning critically influences downstream QA: State-of-the-art models include Google Qatari ASR, Azure Qatar, Whisper large-v3, and Fanar, with WERs for Arabic and English ranging from 5.85% to 33.80% (Alam et al., 25 May 2025).
• WER directly predicts SQA performance degradation: There is an approximately linear relationship between ASR WER and SQA assessment scores. In extractive QA, a 1% absolute increase in WER causes a 0.5% drop in token-overlap metrics (Text Overlap Score, TOS) and 0.25% in Audio Overlap Score (AOS) (Raina et al., 2021).

ASR Model	Arabic WER	English WER	Role
Google Qatari	5.85%	18.02%	Best for Arabic
Whisper v3	12.50%	10.58%	Strong cross-lingual
Azure Qatar	9.50%	21.40%	Cloud-based, regionalized
Fanar	10.40%	33.80%	Multilingual, Arabic-tuned

WER values as reported in (Alam et al., 25 May 2025).

Significance: Accurate ASR, particularly trained or fine-tuned for the demographic in question (non-native, dialectal, noisy-channel), is the primary determinant of Talking-NSQ system accuracy. General-purpose ASR models can outperform region-tuned systems under domain mismatch.

4. Model Architectures and Training Paradigms

Cascaded Pipeline (ASR → LLM-based QA)

• ASR converts speech to text.
• A text-based QA model (e.g., ELECTRA, GPT-4o) is used for answer span extraction or generation, typically via zero-shot or prompt-based conditioning.
• Prompting strategies include disciplined output formatting (e.g., JSON, word count caps) to enforce answer structure and mitigate hallucinations (Alam et al., 25 May 2025).

Cascade-Less and End-to-End Approaches

• Direct speech-to-answer models (example: GPT-4o-Audio, SpeechGPT) bypass explicit ASR, operating on learned audio encodings (Alam et al., 25 May 2025).
• This suggests superior robustness to ASR cascading errors, particularly in heavily accented/noisy scenarios, but requires sufficiently large parallel datasets for training.

Transfer Learning with Disfluency-Augmented Training

• Pretrained MC models (e.g., ELECTRA-Large) are fine-tuned using text corpora (SQuAD 2.0), then adapted to speech via unstructured text augmentations (back-translation, TTS+ASR) that inject realistic disfluencies, which consistently increase robustness by 1–2% TOS across WER ranges (Raina et al., 2021).

Model Evaluation Metrics

• F1 (token-level), Exact Match (EM), and BERTScore F1 (contextualized embedding overlap) are used for answer quality.
• TOS and AOS provide direct correspondences to text and audio domain overlap with gold answers.
• Human TOS/AOS agreement is ~65–68% for reference.

5. Multidimensional Evaluation and Performance

Empirical QA Performance

• On SpokenNativQA, the average F1 scores (BERTScore variant) for Arabic (no-ASR) are ~0.54, dropping by ~0.01–0.02 with ASR input. For English, the drop from 0.62 to ~0.59 is observed across Whisper, Google, and Azure (Alam et al., 25 May 2025).
• Cascade-less LLMs (e.g., GPT-4o-Audio) achieve F1 on par with or exceeding the cascaded setup, notably in Arabic (0.55 vs. 0.536) and English (0.62, matching the baseline).

Error Analysis

• Speaker accent mismatch in L2 English and environmental variability elevate WER and degrade QA performance (Alam et al., 25 May 2025). Data augmentation via noise injection is suggested to increase ASR/QA robustness.
• Grammar errors in non-native speech exhibit minimal detrimental effect on extractive QA, with a 13% grammatical-error corpus showing ΔTOS = +0.2% post-correction (Raina et al., 2021).
• Error propagation is measurable: for every 10% absolute WER increase, there is an approximate 5% TOS loss in SQA, underscoring the importance of ASR optimization (Raina et al., 2021).

6. System Design Recommendations and Best Practices

• Fine-tune ASR and LLM components on region- or dialect-specific spoken data for maximum robustness.
• Employ disciplined, minimal QA prompting (output caps, JSON schema) to enforce structure and prevent drift.
• Leverage cascade-less end-to-end models to minimize compounding error, particularly for spontaneous, noisy, or low-resource languages.
• Continuously incorporate spontaneous, multi-turn, context-dependent queries into the dataset alongside read questions to capture realistic dialogue flows (Alam et al., 25 May 2025).
• Integrate disfluency-robustness via augmentation strategies (back-translation, noisy text, TTS+ASR) during model training (Raina et al., 2021).
• Augment evaluation protocols with user-centric metrics (latency, confidence calibration, fluency ratings), in addition to static F1/BERTScore.

A summary of core Talking-NSQ design elements is given in the following table:

Component	Best Practice	Reference
ASR Front-End	Fine-tune on region/dialect, data augment for noise	(Alam et al., 25 May 2025, Raina et al., 2021)
LLM/MC Backend	Use strong pretrained transformers	(Raina et al., 2021)
Model Training	Add disfluencies via unstructured augmentation	(Raina et al., 2021)
Prompting	Enforce output format, length, language constraints	(Alam et al., 25 May 2025)
End-to-End Modeling	Prefer audio-LMs for error resilience	(Alam et al., 25 May 2025)
Evaluation	Combine BERTScore F1, TOS/AOS, and user metrics	(Alam et al., 25 May 2025, Raina et al., 2021)

7. Challenges and Future Directions

Key ongoing and emergent research challenges include:

• Scaling to more diverse and code-switched conversational scenarios, requiring spontaneous, multi-turn, and contextually anchored speech data.
• Domain adaptation for low-resource languages and dialects, which may require unsupervised or weakly supervised approaches and targeted fine-tuning.
• Robustness to environmental noise and device heterogeneity in open-field deployments, motivating additional data augmentation and online error adaptation.
• Real-time user feedback and interactive dialogue maintenance (turn-taking, backchanneling, clarification requests) to close the conversational loop, especially in assessment and assistive contexts.

A plausible implication is that as end-to-end, cascade-less, multimodal speech–text architectures mature, and as datasets expand in spontaneous and low-resource coverage, Talking-NSQ systems will evolve into robust, evaluation-friendly conversational agents deployable across a wide range of real-world applications.

---

References:

• (Alam et al., 25 May 2025) SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs
• (Raina et al., 2021) An Initial Investigation of Non-Native Spoken Question-Answering