Human-Guided Reasoning with Large Language Models for Vietnamese Speech Emotion Recognition

Abstract: Vietnamese Speech Emotion Recognition (SER) remains challenging due to ambiguous acoustic patterns and the lack of reliable annotated data, especially in real-world conditions where emotional boundaries are not clearly separable. To address this problem, this paper proposes a human-machine collaborative framework that integrates human knowledge into the learning process rather than relying solely on data-driven models. The proposed framework is centered around LLM-based reasoning, where acoustic feature-based models are used to provide auxiliary signals such as confidence and feature-level evidence. A confidence-based routing mechanism is introduced to distinguish between easy and ambiguous samples, allowing uncertain cases to be delegated to LLMs for deeper reasoning guided by structured rules derived from human annotation behavior. In addition, an iterative refinement strategy is employed to continuously improve system performance through error analysis and rule updates. Experiments are conducted on a Vietnamese speech dataset of 2,764 samples across three emotion classes (calm, angry, panic), with high inter-annotator agreement (Fleiss Kappa = 0.8574), ensuring reliable ground truth. The proposed method achieves strong performance, reaching up to 86.59% accuracy and Macro F1 around 0.85-0.86, demonstrating its effectiveness in handling ambiguous and hard-to-classify cases. Overall, this work highlights the importance of combining data-driven models with human reasoning, providing a robust and model-agnostic approach for speech emotion recognition in low-resource settings.

• The paper introduces a hybrid framework that integrates LLM reasoning and acoustic ML models to enhance Vietnamese speech emotion recognition.
• It employs a confidence-aware routing mechanism that directs ambiguous audio samples to LLMs guided by human-derived rules.
• Empirical results show 86.59% accuracy and robust resolution of angry-panic ambiguities in a newly curated Vietnamese SER dataset.

Human-Guided LLM-Based Reasoning for Vietnamese Speech Emotion Recognition

Motivation and Research Context

Vietnamese Speech Emotion Recognition (SER) presents persistent challenges stemming from the ambiguity of acoustic emotional signals and the scarcity of large-scale, high-quality annotated data. Traditional data-driven techniques for SER, primarily relying on ML and deep learning over acoustic features, exhibit performance degradation, especially in real-world and healthcare-critical scenarios where emotional classes (notably angry and panic) display substantial acoustic overlap. Furthermore, annotation subjectivity and low resource availability exacerbate task difficulty, and data-centric approaches lack explicit incorporation of human-like reasoning and interpretability.

This work introduces a hybrid human–machine collaborative framework for Vietnamese SER, where LLM-based reasoning modules are synergized with conventional acoustic feature-driven models. The proposed system leverages human annotation-derived rules for structured reasoning, iteratively refined based on error analysis, and augmented by a confidence-aware routing mechanism that delegates only ambiguous cases to the LLM. The approach is evaluated on a newly curated, high-agreement Vietnamese SER dataset, highlighting both empirical improvements and model-agnostic robustness.

Framework Architecture

The system comprises three principal modules: (1) acoustic feature extraction, (2) an LLM-based reasoning module, and (3) a hybrid inference mechanism anchored by a confidence-aware router. The design enables the system to offload only low-confidence or ambiguous samples to the LLM for comprehensive reasoning based on structured, human-derived rules, while confidently classified samples are handled by the classical ML subsystem.

Figure 1: Overall architecture of the proposed human–machine collaborative framework.

Feature Extraction and Representation

The input waveform is transformed into a structured embedding incorporating pitch, energy, and MFCC vectors. These features are selected for their saliency in emotional classification: pitch and energy encode arousal and intensity, while MFCCs compactly represent the phonetic structure of speech.

LLM-Based Structured Reasoning

For ambiguous samples, acoustic features are mapped to structured textual descriptions, which are input to an LLM accompanied by a ruleset derived from error analysis and human annotation patterns. Notably, reasoning rules formalize patterns such as high variance in pitch and energy mapping to panic, and stable features indicating calm. The LLM thus performs inference guided not only by training data but also by explicit human reasoning strategies, which are iteratively refined throughout system deployment.

Hybrid Confidence-Aware Inference

Predictions from the ML sub-module (SVM over features) generate confidence scores. High-confidence cases are finalized, while those below threshold are routed to the LLM. This hybrid inference pathway maximizes sample-wise reliability and model interpretability—LLM outputs are grounded in rule-based logic, not solely opaque learned distributions.

Iterative Rule Refinement

Prediction errors feed into a continual refinement pipeline. Systematic ambiguities revealed by error analysis lead to revision and expansion of the rules given to the LLM, thus progressively enhancing overall decision robustness and consistency.

Dataset Construction and Annotation

The authors curate a Vietnamese SER dataset with 2,764 audio samples encompassing three affective classes: calm, angry, panic. Diversity is engineered through inclusion of audio from all major Vietnamese regions and a range of media and interview sources.

Figure 2: Distribution of speakers across different age groups in the proposed dataset.

Annotation is carried out by three human raters, following precise guidelines for each class: calm (stable pitch/energy), angry (elevated energy/pitch), and panic (highly variable prosody). Inter-annotator agreement is quantified using Fleiss’ Kappa ($\kappa = 0.8574$), substantiating ground truth reliability and capturing the subjective boundaries critical for downstream reasoning rule derivation.

Experimental Validation

Experimental Setup

The dataset is slotted into train/validation/test and evaluated with multiple LLM backbones (Qwen2.5-7B/14B, LLaMA3.2-3B, Gemma3-4B). Systems are benchmarked against a text-only baseline (audio-to-text transcription via Whisper feeding into an LLM) and ablated pipeline versions.

Core Results

The v4 hybrid system achieves 86.59% accuracy and Macro F1 of 0.86, nearly matching human annotator agreement rates, and substantially surpassing both zero-shot and text-only benchmarks (~38–44% accuracy). Structured reasoning stages (v2, v3) show progressive gain over basic LLM inference, but only the hybrid route, integrating both feature-level ML and human-guided reasoning, resolves ambiguity robustly—most notably in the angry-panic confusion regime.

Performance remains stable across LLM backbones, and the pipeline's efficacy does not depend on architectural size, affirming its model-agnostic claims. Notably, the text-only approach (audio-to-transcript-to-LLM) underperforms drastically, indicating the indispensability of explicit acoustic features over purely lexical information for this emotional classification task.

Implications and Future Research Directions

This framework demonstrates that model performance on ambiguous SER tasks in low-resource languages can be significantly enhanced by integrating explicit, rule-based human reasoning into LLM-driven inference, especially when coupled with acoustic evidence. The approach advances interpretability and offers a systematic pathway for continual system improvement via rule refinement anchored in expert annotation behaviors.

Practical extensions include real-world deployment in telehealth or emergency monitoring, where robust detection of emotionally ambiguous speech has high stakes, necessitating explainable and high-reliability predictions. The framework is inherently extensible: additional emotion classes, larger datasets, or multilingual adaptation are straightforward by updating rules and retraining the acoustic/LLM modules. Moreover, the rule-based paradigm opens future avenues for explainable AI, human-in-the-loop deployment, and adaptation to new settings with minimal labeled data.

Conclusion

The paper establishes a novel, high-performing Vietnamese SER pipeline leveraging human–machine collaboration: ML-based acoustic modeling and LLM-based, human-rule-guided reasoning. The evidence demonstrates performance convergence toward human expertise, robustness across LLM backbones, and strong gains over text-only or naïve LLM approaches. This work substantiates the critical role of explicit human knowledge integration and structured reasoning in speech emotion tasks marked by ambiguity and low data resources, paving the way for broader applications in interpretable, high-risk, and low-resource AI systems.