AudiBERT: Multimodal Audio-Text Fusion Model
- AudiBERT is a multimodal BERT variant that fuses text and audio, combining utterance transcriptions with acoustic-prosodic features to enhance collaborative problem-solving diagnosis.
- It employs a fusion of BERT text embeddings and Wav2Vec 2.0 audio embeddings through a Bidirectional LSTM and self-attention mechanism, distinguishing it from text-only models.
- Empirical findings show that AudiBERT significantly outperforms unimodal BERT for social-cognitive subskill classification, especially for sparse classes, validated via statistical tests.
Searching arXiv for papers on AudiBERT and closely related audio-BERT models. AudiBERT is a multimodal BERT variant used to integrate textual utterance representations with speech or audio-derived information. In the literature supplied here, the term denotes two distinct but conceptually related lines of work. In educational dialogue analysis, AudiBERT is a multimodal transformer-based model for collaborative problem solving (CPS) diagnosis that combines dialogue transcription with speech and acoustic-prosodic audio features (Wong et al., 19 Jul 2025). In auditory commonsense modeling, the closely related “AudioBERT” is a retrieval-augmented LLM that augments BERT with external auditory knowledge for masked-token prediction on auditory reasoning tasks (Ok et al., 2024). Taken together, these usages place AudiBERT within a broader family of audio-grounded BERT adaptations in which text-only language modeling is supplemented either by aligned utterance-level audio or by retrieved audio knowledge.
1. Terminological scope and research context
Within the material considered here, “AudiBERT” refers primarily to a multimodal CPS diagnosis model contrasted against a unimodal BERT baseline (Wong et al., 19 Jul 2025). That model processes transcriptions at the utterance level, incorporates acoustic-prosodic information from audio, and is reported to improve CPS diagnosis relative to text-only BERT in some settings, especially for some sparse classes, some social-cognitive subskills, and some negative affective states (Wong et al., 19 Jul 2025). The same source emphasizes that this workshop paper is not the original model paper for AudiBERT; rather, it is an extension and analysis paper building on prior work by Wong et al. (2025) (Wong et al., 19 Jul 2025).
A separate but relevant line of work is “AudioBERT: Audio Knowledge Augmented LLM” (Ok et al., 2024). That system is not the same model as the CPS-oriented AudiBERT, but it is directly relevant to the general idea of audio-domain BERT variants. AudioBERT is described as a retrieval-augmented LLM designed to give a text-only LM access to auditory commonsense knowledge, using auditory span detection, CLAP-based retrieval, audio embedding injection, and conditional LoRA activation (Ok et al., 2024). This suggests that the label “AudiBERT” is best understood as an umbrella for BERT-derived architectures that incorporate auditory evidence, rather than as the name of a single canonical architecture.
The supplied material also distinguishes these models from other audio-language uses of BERT. “Leveraging Pre-trained BERT for Audio Captioning” adapts BERT into a left-to-right decoder for audio caption generation, pairing a PANNs encoder with a BERT-family decoder modified to use unidirectional self-attention and cross-attention to audio features (Liu et al., 2022). By contrast, the document labeled “Audio ALBERT” does not contain substantive technical content about audio representation learning and therefore cannot support technical claims about an AudiBERT-style system (Chi et al., 2020).
2. AudiBERT for collaborative problem solving diagnosis
In the CPS setting, AudiBERT is a multimodal model for automatic diagnosis from students’ utterances during collaborative problem solving (Wong et al., 19 Jul 2025). The educational setting comprises 78 secondary school students aged 14–15 who worked in triads on a mathematics problem-solving question via remote video conferencing with no teacher intervention (Wong et al., 19 Jul 2025). The unit of analysis is the utterance level; each utterance has a transcription, associated acoustic-prosodic audio features, and one or more CPS labels (Wong et al., 19 Jul 2025).
The target labels are organized into two dimensions. The social-cognitive dimension contains 10 CPS subskills: 8 problem solving subskills, SS1–SS8, and 2 scripting subskills, SC1–SC2 (Wong et al., 19 Jul 2025). Named examples include SS2, “Building shared understanding,” SC1, “Using scripting,” and SC2, “Regulating scripting” (Wong et al., 19 Jul 2025). The affective dimension consists of three affective states: AS1, Neutral affective state; AS2, Negative affective state; and AS3, Positive affective state (Wong et al., 19 Jul 2025).
The paper also examines fine-grained indicator labels within selected classes. For SS2, 11 indicators are listed: PS04, PS05, PS06, PS07, PS08, PS09, PS11, PS12, PS13, PS15, and PS16 (Wong et al., 19 Jul 2025). For scripting, five indicators are listed: S01, S02, S03, S04, and S05 (Wong et al., 19 Jul 2025). For AS2, six indicators are listed: A02, A03, A04, A05, A07, and A08 (Wong et al., 19 Jul 2025). Named examples include PS04, “Asking questions to clarify understanding, ideas or contributions,” S03, “Responding to script components,” S04, “Discussing work status and progress on script components,” A02, “Verbal cues to express bewilderment from observations in the task or of others,” A03, “Use of strong and vulgar expressions,” and A04, “Verbalising difficulty or dislike of work to others” (Wong et al., 19 Jul 2025).
The comparison baseline is a unimodal BERT model operating on transcription data only (Wong et al., 19 Jul 2025). The distinction is operationally simple: BERT uses what was said, while AudiBERT uses what was said together with how it was said. The paper further states that BERT is easier to interpret with existing XAI tools such as SHAP, whereas AudiBERT lacks an established explainability architecture in this setup (Wong et al., 19 Jul 2025).
3. Architecture and multimodal fusion
The CPS-oriented workshop paper does not provide a full architecture diagram or end-to-end implementation recipe for AudiBERT, but it states that the model involves concatenation of a text embedding from the BERT model and an audio embedding from Wav2Vec 2.0, with these embeddings obtained from a complex architecture involving a Bidirectional LSTM and a self-attention layer (Wong et al., 19 Jul 2025). It therefore presents AudiBERT as a deeper multimodal representation model rather than a shallow late-fusion baseline (Wong et al., 19 Jul 2025).
The paper explicitly identifies the textual side as a text embedding from BERT and the audio side as an audio embedding from Wav2Vec2.0 (Wong et al., 19 Jul 2025). Because the dataset is constructed at the utterance level, the implied alignment mechanism is utterance-level pairing between one utterance transcript and one corresponding utterance-level audio segment or representation (Wong et al., 19 Jul 2025). The source does not specify hidden sizes, number of layers, optimizer, dropout, batch size, epoch count, loss function, sequence length, or whether BERT and Wav2Vec2.0 were frozen or fine-tuned (Wong et al., 19 Jul 2025).
The paper is explicit about what is and is not known regarding audio processing. It states that the data consist of “transcription and acoustic-prosodic features extracted from audio data at the utterance level,” and it implies that the broader multimodal audio processing includes an audio embedding from Wav2Vec2.0 (Wong et al., 19 Jul 2025). However, it does not list specific handcrafted acoustic features, frame size, hop size, toolkit, normalization, or exact feature vector dimensionality (Wong et al., 19 Jul 2025). This suggests that AudiBERT, as described here, is defined more by its multimodal integration principle than by a fully specified public reference implementation.
A related architecture appears in the auditory-commonsense AudioBERT system. There, the pipeline detects auditory spans in text, retrieves audio using CLAP, encodes the retrieved signal with AST, adds the audio embedding to the first token of the auditory knowledge span, and activates LoRA only when auditory knowledge is needed (Ok et al., 2024). The fusion mechanism in that model is additive rather than concatenative, and it uses retrieved audio knowledge rather than utterance-synchronous speech (Ok et al., 2024). The contrast between the two systems is structurally important: one is an utterance-level multimodal classifier, while the other is a retrieval-augmented masked LLM.
4. Statistical findings in CPS classification
The main empirical claim in the CPS paper is that AudiBERT significantly outperforms BERT for social-cognitive CPS subskill classification, but not for affective-state classification as a whole (Wong et al., 19 Jul 2025). For the 10 social-cognitive classes, the paper reports a one-tailed Wilcoxon signed-rank result of , with approximate effect size $0.91$ and power $0.84$ (Wong et al., 19 Jul 2025). For the three affective classes, it reports , with approximate effect size $0$ and power $0.05$ (Wong et al., 19 Jul 2025).
The same source emphasizes that AudiBERT improved classification of classes that were sparse in the dataset (Wong et al., 19 Jul 2025). This point is treated as practically important for low-support affective classes and other fine-grained CPS categories with limited training samples (Wong et al., 19 Jul 2025). At the same time, the null affective-state significance result is explicitly linked to an underpowered analysis based on only three classes (Wong et al., 19 Jul 2025).
The paper also investigates associations between model performance, training data size, and human agreement. For both BERT and AudiBERT, recall shows a significant positive association with class training size. For BERT recall, the reported correlations are and ; for AudiBERT recall, they are and (Wong et al., 19 Jul 2025). Precision does not show a meaningful association with class size for either model (Wong et al., 19 Jul 2025).
Human agreement is operationalized using Cohen’s kappa as a proxy for task complexity or coding clarity (Wong et al., 19 Jul 2025). The paper first reports no significant relationship between kappa and testing data size: $0.91$0 and $0.91$1 (Wong et al., 19 Jul 2025). It then finds that BERT precision is significantly associated with high inter-rater agreement under Spearman correlation, with $0.91$2, whereas the analogous relationship for AudiBERT precision is not significant (Wong et al., 19 Jul 2025). This is used to motivate a complementary role for the more interpretable text-only model in classes that humans also code consistently.
5. Indicator-level analysis and human-AI complementarity
A central argument of the CPS paper is that broad class-level performance does not guarantee strong fine-grained indicator-level performance (Wong et al., 19 Jul 2025). To study this, indicator classification is carried out using BERT only, because BERT is explainable and can support reflective coding workflows (Wong et al., 19 Jul 2025). The classes selected as “well-detected” are defined as those with weighted F1 greater than 0.60 for both BERT and AudiBERT in prior work: SS2, SC1, SC2, AS1, AS2, and AS3 (Wong et al., 19 Jul 2025).
The aggregate BERT indicator-level results show strong variability across categories. For SS2, the reported metrics are accuracy 0.480, precision 0.462, recall 0.480, and F1 0.411 (Wong et al., 19 Jul 2025). For SC1 + SC2, they are accuracy 0.663, precision 0.642, recall 0.663, and F1 0.649 (Wong et al., 19 Jul 2025). For AS2, they are accuracy 0.804, precision 0.794, recall 0.804, and F1 0.792 (Wong et al., 19 Jul 2025). The paper interprets this pattern as evidence that negative affect indicators are much easier for BERT than SS2 indicators, despite SS2 being a broader well-detected class (Wong et al., 19 Jul 2025).
The well-detected indicators under BERT, defined as weighted F1 at least 0.60, are PS04 with F1 0.762, S03 with F1 0.726, S04 with F1 0.711, A02 with F1 0.852, A03 with F1 0.643, and A04 with F1 0.887 (Wong et al., 19 Jul 2025). Many sparse indicators have F1 equal to 0, including PS07, PS08, PS09, PS11, PS12, PS13, S02, S05, A07, and A08 (Wong et al., 19 Jul 2025). The paper treats this as further evidence that class-level summaries can conceal extreme heterogeneity at indicator level.
On that basis, a structured human-AI complementarity workflow is proposed (Wong et al., 19 Jul 2025). Human coders first label about 10%–15% of the dataset to establish inter-rater reliability and compute Cohen’s kappa (Wong et al., 19 Jul 2025). AudiBERT then provides the initial class prediction because it performed better for subskills and some affective classes (Wong et al., 19 Jul 2025). A human coder selects an indicator within the suggested class, after which BERT supplies an indicator prediction, the tokenized words driving the classification, and a warning if the suggested class was trained on fewer examples than the median class size (Wong et al., 19 Jul 2025). If necessary, BERT provides an alternative subskill or affective class (Wong et al., 19 Jul 2025). The paper further recommends using BERT more confidently when $0.91$3 (Wong et al., 19 Jul 2025). This suggests a division of labor in which AudiBERT contributes predictive strength at the broader class level, while BERT contributes interpretability and calibration for human oversight.
6. Relation to other audio-grounded BERT variants
The CPS-oriented AudiBERT can be situated relative to two neighboring research directions in the supplied material. The first is BERT-based audio captioning. In “Leveraging Pre-trained BERT for Audio Captioning,” the model pairs a CNN10 encoder from PANNs with a decoder initialized from public BERT-family checkpoints, including BERT_tiny, BERT_mini, BERT_medium, BERT_base, and RoBERTa_base (Liu et al., 2022). To convert BERT into a caption decoder, the paper modifies bidirectional self-attention to unidirectional self-attention and adds cross-attention layers after the self-attention layers to connect the decoder to audio encoder outputs (Liu et al., 2022). The method is evaluated on AudioCaps, where the downloaded version contains 49,274 training clips, 494 validation clips, and 957 test clips (Liu et al., 2022). Its best reported proposed-model results include CNN10 + BERT_mini with BLEU$0.91$4 67.1, BLEU$0.91$5 25.1, METEOR 23.2, CIDEr 66.7, SPICE 17.2, and SPIDEr 41.9, and CNN10 + BERT_medium with BLEU$0.91$6 67.1, BLEU$0.91$7 25.5, METEOR 23.1, CIDEr 65.4, SPICE 16.8, and SPIDEr 41.1 (Liu et al., 2022). This line of work shows BERT being adapted to audio-conditioned generation rather than multimodal utterance classification.
The second direction is auditory commonsense augmentation via retrieval. “AudioBERT: Audio Knowledge Augmented LLM” introduces AuditoryBench, a benchmark built on LAION-Audio-630K with two tasks: animal sound recognition and sound pitch comparison (Ok et al., 2024). Animal sound recognition uses prompts such as “meow is the sound a [MASK] makes,” while pitch comparison uses prompts such as “The sound of a synthesizer typically has a [MASK] pitch than an acoustic bass” (Ok et al., 2024). The benchmark is evaluated with accuracy for the final tasks and F1-score for auditory span detection (Ok et al., 2024). On the test set, BERT-base obtains 13.46 on animal sound recognition, 60.41 on sound pitch comparison, and 44.61 combined, whereas AudioBERT built on BERT-base reaches $0.91$8, $0.91$9, and $0.84$0 respectively; the BERT-large-based variant reaches $0.84$1, $0.84$2, and $0.84$3 (Ok et al., 2024). This system uses CLAP retrieval, AST-based audio embedding injection, and LoRA with rank $0.84$4 and alpha $0.84$5 (Ok et al., 2024).
These neighboring models clarify what the CPS-oriented AudiBERT is not. It is not a jointly pretrained audio-text generative model, and it is not a retrieval-augmented masked LM operating on general auditory commonsense (Liu et al., 2022, Ok et al., 2024). Instead, it is an utterance-level multimodal classifier tailored to educational dialogue, where transcription and audio-prosodic evidence are fused for CPS diagnosis (Wong et al., 19 Jul 2025).
7. Limitations and interpretation
Several limitations are explicit in the supplied literature. For the CPS-oriented AudiBERT, the workshop paper does not provide a full implementation specification: no hidden sizes, number of layers, learning rate, optimizer, dropout, batch size, epoch count, loss function, sequence length, transcript source, exact audio segmentation procedure, or detailed feature-extraction pipeline are reported (Wong et al., 19 Jul 2025). Many exact model-performance values for AudiBERT versus BERT are also said to come from prior work rather than being reproduced in full in the workshop paper (Wong et al., 19 Jul 2025).
The affective-state significance analysis is explicitly underpowered because it is based on only three classes, with reported power 0.05 (Wong et al., 19 Jul 2025). The paper also notes an explainability gap: AudiBERT may perform better, but there is currently no practical explainability architecture for the described multimodal setup combining BERT text embeddings, Wav2Vec2.0 audio embeddings, BiLSTM, self-attention, and concatenation fusion (Wong et al., 19 Jul 2025). As a result, indicator-level evaluation is performed only with BERT, leaving open whether AudiBERT would also improve fine-grained indicator prediction (Wong et al., 19 Jul 2025).
For the retrieval-based AudioBERT, the paper notes narrower task coverage, limited formal detail about retrieval corpus construction and multi-span fusion, and weaker external Wiki generalization on animal sounds than on the benchmark test split (Ok et al., 2024). For the audio-captioning BERT adaptation, the method is constrained by a mismatch between BERT’s bidirectional masked-language pretraining and left-to-right caption generation, which is addressed architecturally rather than through a generation-style pretraining objective (Liu et al., 2022).
A plausible implication is that “AudiBERT” names a design space rather than a fixed architecture. In the supplied research, the common thread is the attempt to compensate for the limitations of text-only BERT by introducing auditory evidence—through utterance-aligned multimodal fusion in CPS diagnosis (Wong et al., 19 Jul 2025), through retrieval-augmented auditory knowledge injection for commonsense reasoning (Ok et al., 2024), or through conversion of BERT into an audio-conditioned decoder for caption generation (Liu et al., 2022). The strongest specifically AudiBERT-related evidence here is that multimodal audio-text fusion can yield statistically significant class-wise gains over unimodal BERT for social-cognitive CPS subskill classification, while still raising open questions about interpretability, reproducibility, and fine-grained generalization (Wong et al., 19 Jul 2025).