---
title: 'MutterMeter: Dual Bioacoustic Detection'
url: https://www.emergentmind.com/topics/muttermeter
type: topic
---

# MutterMeter: Dual Bioacoustic Detection

MutterMeter refers to two distinct but technically related systems—one for automatic self-talk detection via wearable audio (“earables”) and another for state-of-the-art heart rate and murmur detection from phonocardiogram (PCG) signals using deep learning architectures. Each MutterMeter system leverages multi-stage pipelines that integrate acoustic feature engineering, neural embeddings, and hierarchical classification or multi-task architectures to address application-specific challenges involving audible murmurs, acoustic ambiguity, and real-world usability constraints.

## 1. Self-Talk Detection via Earables

MutterMeter, as described by Lee et al., designates a mobile system for automatic self-talk detection from real-world audio captured by in-ear microphones (earables) [2511.07493]. Self-talk—defined as momentary, self-directed, often incomplete speech used for emotion regulation and cognitive processing—remains challenging to observe in situ due to its sporadic occurrence, low amplitude, and syntactic idiosyncrasies.

### 1.1 Technical Challenges

Self-talk detection departs from standard speech understanding along several axes:
- **Acoustic Diversity**: Self-talk varies from nearly inaudible murmurs to emotionally charged exclamations.
- **Linguistic Incompleteness**: Utterances exhibit grammatical fragmentation, repetition, and omission of standard constituents.
- **Temporal Irregularity**: Unlike dialogue, self-talk has no structured turn-taking or predictable timing, often occurring in clusters under emotional stress.

Conventional approaches relying on self-reports or post-hoc human annotation are error-prone and unscalable, motivating an objective, real-time measurement solution.

## 2. System Architecture and Algorithmic Workflow

MutterMeter adopts a three-stage, hierarchical classification pipeline, employing confidence-gated early exits to reduce computational overhead, minimize latency, and adaptively fuse acoustic and linguistic logits [2511.07493].

### 2.1 Pipeline Stages

**Preprocessing**: Continuous 22 050 Hz audio streams are segmented by detecting vocal events (RMS threshold > –20 dB). Utterances longer than 300 ms with inter-frame gaps less than 800 ms are merged, and a rolling context cache (max 30 s) is maintained for context-aware transcription.

**Acoustic Stage**: Each utterance is embedded via an 80-channel log-mel spectrogram front end (STFT), encoded using a fine-tuned Whisper-base Transformer to yield high-dimensional features. The Locality-Aware Embedding Adaptation (LAEA) module smooths embeddings for utterances proximate in time (ΔT ≤ 4 s) using adaptive weighting:
$$
\alpha = \mathrm{MLP}([e_{\text{curr}}; e_{\text{prev}}]), \qquad e_{\text{adapted}} = \alpha e_{\text{curr}} + (1-\alpha) e_{\text{prev}}
$$
Classification proceeds via a three-layer MLP + softmax.

**Linguistic Stage**: Triggered if the acoustic prediction lacks sufficient confidence (margin $< \tau_a$) or for low-confidence ‘positive’ classes. Utterances aggregated from the context cache are transcribed with Whisper-large ASR to minimize WER (0.60 vs. 0.82 for single-utterance), then embedded through a fine-tuned Korean BERT encoder, and classified via a separate three-layer MLP.

**Fusion Stage**: If the linguistic margin is also low ($< \tau_l$), acoustic and linguistic embeddings are projected and merged with gated fusion:
$$
g = \sigma(W_g [e_a; e_t] + b), \quad z = g \odot e_a + (1-g) \odot e_t
$$
Classification is performed over the fused representation with a five-layer MLP.

### 2.2 Confidence-Gated Early-Exit Logic

The system employs a confidence gating strategy to reduce unnecessary computation:
```python
(y_a, m_a) = Acoustic(x)
if (y_a in {neg, other}) and m_a >= 0.92:
    return y_a
else:
    (y_l, m_l) = Linguistic(x)
    if (y_l == neg) and m_l >= 0.80:
        return y_l
    else:
        return Fusion(x)
```

**Thresholds** ($\tau_a = 0.92$, $\tau_l = 0.80$) are derived empirically from the least-margin distributions.

## 3. Dataset, Training, and Empirical Evaluation

### 3.1 Data Collection and Annotation

- 25 recreational/amateur tennis players (22 male, 3 female, mean experience 2.4 y)
- In situ: Three matches each, open-ear wireless earphones, synchronized to video
- 31.1 hours total; 8 900 utterances (3.9 h), subdivided into negative (26%), positive (16%), and other (58%) self-talk, durations mean 2.0 s, 1.6 s, and 1.4 s
- Labels: Assigned through manual triage using timestamped audio-video alignment by two raters

### 3.2 Model Training

- **Acoustic**: Whisper-base, fine-tuned (batch 32, AdamW, LR 2e–5, early stopping)
- **LAEA**: MLP, Adam, LR 1e–4
- **Linguistic**: Whisper-large ASR + fine-tuned Korean BERT (batch 64, LR 2e–5)
- **Fusion**: batch 64, Adam, LR 1e–4
- **Validation**: Leave-One-Subject-Out (LOSO) cross-validation

### 3.3 Performance Metrics and Baselines

**Overall Macro-F₁** (test, LOSO):
| Model            | Macro-F₁ |
|------------------|----------|
| Acoustic only    | 0.80     |
| Linguistic only  | 0.70     |
| Fusion only      | 0.82     |
| MutterMeter      | 0.84     |

- **By class**: Negative: F₁=0.84, Positive: F₁=0.77, Other: F₁=0.91
- **Baseline comparison**: LLM- and SER-based methods (Emotion2Vec, TweetNLP, Gemini) all scored substantially below MutterMeter (≤0.69)
- **Ablations**: LAEA and adaptive fusion yield +5–10% F₁ improvement on positive self-talk and overall

### 3.4 System Latency

- **Preprocessing**: 21 ms per 100 ms audio
- **Acoustic**: 2 015 ms per utterance (majority exit)
- **Linguistic**: 4 298 ms (minority invoked)
- **Fusion**: 0.8 ms
- **Hierarchical early exits**: Reduce average utterance latency from 6 335 ms to 3 713 ms (–41%)

## 4. Limitations and Areas for Improvement

- **Noise Sensitivity**: Errors driven by low SNR or overlapping background audio; ambiguous utterances without strong linguistic or prosodic cues may confound classification.
- **ASR Reliability**: Short or mumbled utterances are transcription-challenged, reducing the effectiveness of the linguistic stage.
- **Computation**: Acoustic stage incurs ~2 s per utterance latency on-device; continuous operation may impact battery life. ASR offload reduces local computation but introduces privacy and connectivity dependencies.

Potential enhancements identified include deployment of pruned or lightweight ASR on-device, user adaptation via speaker-specific fine-tuning, integration of non-audio physiological signals (accelerometer, PPG), support for silent self-talk via bone conduction or EOG, and generalization to non-sports domains [2511.07493].

## 5. Heart Rate and Murmur Detection from PCG: Relation to MutterMeter

A second line of research led by Nie et al. references “MutterMeter” as a framework for combined heart rate estimation and murmur detection in digital stethoscope use cases [2407.18424]. Here, high-fidelity heart sounds (PCG, resampled to 16 kHz, filtered at 2 kHz) are segmented using annotated S1/S2 boundaries. Features extracted per 5 s snippet include Mel spectrogram (40 bands), MFCC (40 coeffs), power spectral density (PSD, Welch), and RMS energy.

### 5.1 2D-CNN and Multi-Task Learning

- **Input**: Four-channel feature “images” (Mel∥MFCC∥PSD∥RMS, shape 4×time×freq)
- **CNN Architecture**: Five convolutional layers (3×3, ReLU, max-pool), followed by a flattening layer and two parallel heads: one for heart rate classification (141-class, 40–180 bpm), one for murmur detection (binary, sigmoid)
- **Training**: Weighted cross-entropy for HR, weighted BCE for murmur, Adam optimizer (initial LR=1e–3, batch 16, 100 epochs, learning-rate schedulers triggered by val MAE < 2 bpm)

### 5.2 Performance and Benchmarks

| Label                | MAE (bpm) / ACC (%) |
|----------------------|--------------------|
| PSD-only baseline    | 8.28 (HR, MAE)     |
| TCNN-LSTM baseline   | 1.63 (HR, MAE)     |
| 2D-CNN (single-task) | 1.312 (HR, MAE)    |
| 2D-CNN-MTL (multi)   | 1.636 (HR, MAE); 97.5 (Murmur ACC) |

Performance for both heart rate and murmur detection surpasses the AAMI requirement (≤10% or ≤5 bpm MAE). All four features yield the best results versus any subset. Notable limitations include lack of explicit noise reduction, drop in performance on extreme HR, requirement for 5 s windows (3 s increases error to ~3.3 bpm), and the treatment of HR as classification rather than regression.

## 6. Broader Significance and Application Domains

The MutterMeter systems collectively establish end-to-end pipelines for automatic physiological and psychological state measurement—either through murmured self-talk or acoustic cardiology. Both approaches emphasize:
- The need for robust acoustic feature engineering coupled with neural representation learning.
- The value of hierarchical or multi-task classification to mediate ambiguous, context-dependent audio.
- Practical trade-offs (on-device versus offloaded computation, latency, and battery demand) that condition real-world feasibility.

A plausible implication is that architectural motifs, such as hierarchical gating and multi-modal fusion, are likely extensible to other in situ bio-acoustic monitoring domains where events are of ambiguous or irregular structure. Additionally, the use of sliding window segmentation, adaptive fusion, and context caches emerges as a recurrent theme in the technical design of both MutterMeter variants.

## 7. References

- [Enabling Automatic Self-Talk Detection via Earables](https://arxiv.org/abs/2511.07493) [2511.07493]
- [Model-driven Heart Rate Estimation and Heart Murmur Detection based on Phonocardiogram](https://arxiv.org/abs/2407.18424) [2407.18424]

Source: https://www.emergentmind.com/topics/muttermeter