Silent Paralinguistics: Expressive Silent Speech

• Silent Paralinguistics is a research field that combines silent speech interfaces with computational paralinguistics to decode both spoken words and expressive traits.
• It employs diverse biosignals such as EEG, EMG, ultrasound, and accelerometers to infer affect, identity, and speaking style from non-audible speech production.
• The field addresses challenges like data scarcity and user-dependency, aiming to enhance silent communication with natural, expressive voice synthesis.

Silent Paralinguistics is a proposed research area at the intersection of silent speech interfaces and computational paralinguistics. It treats speech-related biosignals as carriers not only of linguistic content but also of paralinguistic information, with the aim of perceiving, understanding, and expressing traits, states, and speaking styles from non-audible speech production and, ultimately, reintegrating them into reconstructed audible voice (Ren et al., 25 Aug 2025). In contrast to conventional systems that recover only what was said, this line of work addresses how it was meant, using signals drawn from the neuro-muscular-articulatory continuum rather than from the airborne acoustic stream (Xu et al., 12 Mar 2026).

1. Conceptual definition and scope

Silent Paralinguistics was introduced as a field intended to extend traditional Silent Speech Interfaces (SSIs) beyond two canonical tasks: speech recognition or decoding, and speech synthesis (Ren et al., 25 Aug 2025). The central claim is that silent articulation may preserve not only linguistic content but also paralinguistic information such as affective states, speaker identity or traits, speaking mode, stress, and intention-like prosodic information. The field therefore shifts the target of silent communication from intelligibility alone toward expressivity, personalization, and social naturalness.

A basic distinction within this framework is between Direct SP and Indirect SP. In Direct SP, paralinguistic information is predicted directly from biosignals during silent articulation, alongside or before speech decoding, and can then be fused into speech reconstruction. In Indirect SP, biosignals are first converted into acoustic features; a separate acoustic paralinguistic model then infers traits or states from those generated features before final synthesis (Ren et al., 25 Aug 2025). The same survey organizes target variables into long-term traits such as age, gender, personality, and nativeness or culture; medium-term states such as sleepiness, intoxication, speaking style, and voice quality; and short-term states such as emotions, stress, confidence, pain, and intention-related states.

Adjacent research broadens the interpretive scope of silent paralinguistics beyond biosignal decoding. In digital text, textual paralanguage (TPL) has been defined as “written manifestations of nonverbal audible, tactile, and visual elements that supplement or replace written language and that can be expressed through words, symbols, images, punctuation, demarcations, or any combination of these elements” (Luangrath et al., 2016). In conversation design, silence itself has been treated as a meaningful signal rather than a missing one: context-aware pacing distinguishes Reflective Silence, Facilitative Silence, Empathic Silence, Holding Space, and Immediate Response, each with different relational functions (Jiang et al., 5 Feb 2026). A complementary line of work argues that pauses in conversational English and Korean provide a memoryless information source and act as a time-invariant template for structuring speech sequences (Linke et al., 2021). This suggests that “silent” paralinguistic communication includes not only covert physiological speech production, but also visually rendered and temporally encoded nonverbal structure.

2. Physiological and non-acoustic substrates

The sensing landscape of silent paralinguistics follows the broader SSI taxonomy of four physiological interception points: neural oscillations, neuromuscular activation, articulatory kinematics, and pervasive active probing via acoustic or radio-frequency sensing (Xu et al., 12 Mar 2026). The foundational survey of Silent Paralinguistics reviews EEG, fNIRS, ECoG, and intracortical recordings on the neural side; surface EMG on the muscular side; EMA, PMA, ultrasound tongue imaging, and video on the articulatory side; and EGG for glottal dynamics (Ren et al., 25 Aug 2025). The same review treats facial EMG as the dominant practical biosignal for silent speech, because it captures articulation- and expression-related activation while remaining non-invasive.

Recent systems instantiate these interception points with markedly different hardware. A sentence-level accelerometer SSI uses six MPU6500 sensors connected to an ESP32 via SPI, sampling at 50 Hz, with sensors placed on the jaw, under-jaw, throat, upper lip, lower lip, and cheek (Xie et al., 25 Feb 2025). Each sensor provides three-axis linear acceleration and three-axis gyroscope signals, yielding a 36-dimensional time series after preprocessing. An ultrasound-based interaction system attaches a 3.5-MHz convex ultrasound probe under the jaw, captures 128 × 128 grayscale frames at 30 fps, and uses windows of $K = 13$ frames, corresponding to about 400 ms (Kimura et al., 2023).

Wearable EMG has been pushed toward practical form factors. One neck-worn system integrates 14 fully-differential EMG channels into a textile neckband, uses BioGAP-Ultra for acquisition and BLE transmission, samples at 500 SPS, and consumes 22.2 mW during continuous acquisition and transmission (Meier et al., 26 Sep 2025). Another line of work uses eight facial EMG electrodes for open-vocabulary silent speech voicing (Gaddy et al., 2021). Still another combines silent EMG with facial images, treating EMG as the content source and the face image as a non-audible source of target vocal identity and global pitch (Lee et al., 2 Feb 2026).

Articulatory imaging remains important for understanding what paralinguistic information is physically encoded in silent speech production. A real-time MRI system operating on USC-TIMIT standardizes each utterance to 75 frames at 23.18 fps and performs sentence-level decoding directly from vocal tract shaping (Pandey et al., 2021). The same study analyzes pharyngeal, velar and dorsal, hard palate, and labial constriction regions, showing that sub-region distortion is affected by both emotion and gender. A plausible implication is that silent paralinguistic inference is not restricted to overt facial motion; it may also depend on deeper vocal-tract geometry.

3. Decoding, alignment, and synthesis architectures

The algorithmic core of silent paralinguistics inherits heavily from sequence modeling in SSI. In the accelerometer-based sentence recognizer, the pipeline is end-to-end: a convolutional front-end extracts local motion patterns, a Conformer-based sequence model captures both short-range articulatory detail and long-range sentence context, and Connectionist Temporal Classification (CTC) performs alignment-free decoding (Xie et al., 25 Feb 2025). Because exact frame-to-word boundaries are unavailable, the model minimizes

$$L_{CTC} = -\log \sum_{\pi \in \Pi} P(\pi \mid X), \qquad P(\pi \mid X) = \prod_{t=1}^{T} P(\pi_t \mid X_t).$$

The local-attention Conformer constrains attention to a band of nearby frames, about 0.4 s in this work, while a global token preserves access to whole-sequence context.

Open-vocabulary EMG voicing uses a different but related design. Raw EMG, after minimal preprocessing, is passed through 3 residual 1D convolution blocks, then through 6 bidirectional Transformer encoder layers with relative position embeddings, and finally projected to 26-dimensional MFCC predictions; a WaveNet vocoder then synthesizes audio (Gaddy et al., 2021). Because silent EMG and vocalized audio are recorded separately, training uses dynamic time warping for alignment. An auxiliary phoneme-prediction task modifies the alignment cost and improves intelligibility, indicating that phonetic supervision stabilizes the mapping from muscle activity to speech acoustics.

EEG-based silent paralinguistic decoding introduces yet another modeling regime. The Large Brain LLM (LBLM) is built on Conformer blocks with a layer-gating mechanism and is pretrained with Future Spectro-Temporal Prediction (FSTP), which combines Masked Spectro-Temporal Prediction (MSTP) and Autoregressive Spectro-Temporal Prediction (ASTP) to predict waveform, Fourier amplitude, and Fourier phase from EEG patches (Zhou et al., 29 Apr 2025). The emphasis here is not merely classification but representation learning for temporal and spectral dependencies in covert articulation signals.

Where silent paralinguistics must be rendered back into speech, synthesis architectures become explicitly factorized. In the multi-speaker silent speech voicing framework, the EMG content network estimates a content embedding $c_{emg}$, a face encoder provides the speaker embedding, and a face-based global pitch estimator predicts speaker-wise pitch from the facial image alone (Lee et al., 2 Feb 2026). A pitch-flattening preprocessing module based on Praat/Parselmouth and PSOLA enforces pitch disentanglement during training. This architecture treats paralinguistic cues such as identity, timbre, and global pitch as separate from lexical content.

Spoken-dialogue modeling contributes a higher-level conditioning view. ParalinGPT serializes three tasks in autoregressive order—current paralinguistic attribute prediction, response paralinguistic attribute prediction, and response text generation—conditioning on conversational text, speech embeddings, and paralinguistic attributes (Lin et al., 2023):

$$P(O|H, C) = P(L_n|H, C)\, P(L_{n+1}|H, C, L_n)\, P(T_{n+1}|H, C, L_n, L_{n+1}).$$

This ordering formalizes a conversational logic in which the system first interprets the affective state of the current turn, then predicts the response stance, and only then generates the lexical response.

4. Representative systems and empirical performance

The strongest current silent sentence results in the provided literature come from six-axis facial accelerometers. The Conformer-CTC system reports 97.17% accuracy in sentence recognition, average word classification accuracy of 97.24%, and average sentence recognition accuracy of 97.16% (Xie et al., 25 Feb 2025). For five especially confusing English words, mean accuracy exceeds 99%, with only about 0.2% misclassification. Sentence recognition remains above 95% even as sentence length increases. Sensor ablations show that single-sensor configurations already achieve mean accuracies between 90.22% and 93.19%, that channels 1 and 4 yield the best two-channel result of 97.03%, and that using only two axes achieves about 95.37% accuracy compared with 95.57% using all six axes.

Ultrasound-based silent interaction has demonstrated a different end point: not text decoding but regenerated audio for downstream commercial devices. In SottoVoce, Network 1 alone reaches a 42.5% average recognition success ratio on unchanged Amazon Echo and Echo Show devices, while Network 1 + Network 2 reaches 65.0%; ground-truth audio reaches 90.0% (Kimura et al., 2023). Using Google Cloud Speech-to-Text, word error rate is 41.03% for Network 1, 33.56% for Network 2, and 20.61% for ground truth. The system also reports that true silent mouthing initially degrades output quality, but users can listen to the generated output and adapt their oral movements, improving performance over several trials.

Wearable EMG results emphasize robustness and practicality. The fully-dry textile neckband reaches average classification accuracies of $87 \pm 3\%$ for vocalized articulation and $68 \pm 3\%$ for silent articulation with a 5-fold CV approach; under leave-one-session-out evaluation with neckband repositioning, it reaches $64 \pm 18\%$ and $54 \pm 7\%$, respectively (Meier et al., 26 Sep 2025). In open-vocabulary facial EMG voicing, the improved raw-EMG model reduces WER from 68.0% to 42.2%, an absolute improvement of 25.8 percentage points, while human transcription of synthesized samples yields an average WER of 32.3% (Gaddy et al., 2021). These results indicate that silent EMG carries enough structure for large-vocabulary speech reconstruction, but also that generalization remains difficult.

Neural and articulatory modalities remain weaker in absolute recognition terms but important conceptually. The EEG LBLM achieves 47.0% accuracy on semantic-level classification and 39.6% on word-level classification in the difficult cross-session setting, outperforming baseline methods by 5.4% and 7.3%, respectively (Zhou et al., 29 Apr 2025). The real-time MRI recognizer reaches 40.6% PER at sentence level with a LLM, and its articulatory analysis shows that lower-boundary regions exhibit greater deviation from neutral than upper-boundary regions, with high-arousal emotions such as angry and happy producing larger distortions than sad (Pandey et al., 2021). In multi-speaker voicing without any audible input, the pitch-disentangled Flatten model obtains silent-EMG WER/CER of 38.91/24.28 and improves both local and global pitch errors relative to the Base model (Lee et al., 2 Feb 2026).

5. Dialogue modeling, paralinguistic benchmarks, and layer-level analysis

As silent paralinguistics moves from signal decoding toward conversational behavior, benchmarking has shifted from lexical correctness to acoustic-social appropriateness. WavBench defines an Acoustic subset covering 10 attributes—age, gender, accent, language, pitch, speed, volume, emotion, audio events, and music—and splits evaluation into explicit understanding, explicit generation, and implicit dialogue (Li et al., 12 Feb 2026). In explicit understanding, Step-Audio-2-mini achieves the best average at 57.36%; in explicit generation, GPT-4o Audio reaches 79.23%; in implicit dialogue, the best average is 2.78, shared by Qwen3-Omni and GPT-4o Audio. The benchmark repeatedly characterizes the current limitation as a “Cognitive-Acoustic Alignment” gap.

Model analysis corroborates this gap. “Resurfacing Paralinguistic Awareness in Large Audio LLMs” finds that layers 0–6 are paralinguistic-salient, layers 7–14 encode semantic understanding, and later layers are more specialized for generation and next-token prediction (Yang et al., 12 Mar 2026). On that basis it proposes paralinguistic-enhanced fine-tuning (PE-FT), combining selective-layer fine-tuning of layers 0–14 with an auxiliary dual-level classification head. The paper reports that PE-FT raises PA-rate on the child-safety dataset to roughly 97–99%, while vanilla models are near-random on PA-score and PA-rate.

A more adversarial evaluation arrives from VoxParadox, a benchmark of 2,000 verified examples spanning 10 paralinguistic tasks in which transcript claims are intentionally mismatched with speaking style (Pang et al., 26 May 2026). Across evaluated models, GT accuracy ranges from 6.10% to 30.85%, with an average of 15.33%, while adversarial-label agreement averages 64.34%. For Audio Flamingo 3, Prompt-Conditioned Layer Mixer (PCLM) improves VoxParadox performance from 17.40% to 60.00%, and PCLM plus Direct Preference Optimization improves it further to 65.20%; on MMSU’s paralinguistic subset the same model improves from 37.74% to 54.78%. The central diagnosis is twofold: paralinguistic cues can degrade in deeper encoder layers and at the encoder–LLM interface, and the LLM may ignore them even when they remain available in audio tokens.

Dialogue generation has also been made explicitly paralinguistic. ParalinGPT reports relative improvements of 6.7% in current sentiment accuracy, 12.0% in response sentiment accuracy, and 3.5% in response text BLEU score by conditioning on conversational context, speech embeddings, and serialized sentiment prediction (Lin et al., 2023). At the interaction-design level, context-aware pacing improves perceived human-likeness, smoothness, and interactivity in a between-subjects study with $N = 50$, and in the career-support scenario improves perceived listening quality and affective trust (Jiang et al., 5 Feb 2026). These results indicate that silent paralinguistic competence in dialogue is not reducible to emotion classification alone; it also includes timing, turn management, and delivery.

6. Corpora, applications, and unresolved challenges

The application space repeatedly centers on assistive and private communication. Silent speech systems are motivated by communication impairments, including laryngectomy and other speech impairments, by public or noisy environments, and by privacy-sensitive interaction in which users prefer not to speak audibly (Xie et al., 25 Feb 2025, Kimura et al., 2023, Ren et al., 25 Aug 2025). The SSI review generalizes this to a shift from traditional transducer-centric analysis to intent-to-execution systems integrated into “invisible interfaces” such as earables and smart glasses, while also noting a practical usability threshold of WER below 15% for real-world deployment (Xu et al., 12 Mar 2026). A plausible implication is that silent paralinguistics will mature only when lexical usability and expressive usability converge.

Data scarcity remains the central bottleneck. The introductory survey emphasizes very limited silent-paralinguistic data, poor generalization, the invasiveness of some modalities, weak realism of current datasets, cultural variability, and inadequate objective evaluation (Ren et al., 25 Aug 2025). The SSI review reframes cross-user instability as the user-dependency paradox and links future progress to self-supervised foundation models, transfer learning, and Latent Semantic Alignment (Xu et al., 12 Mar 2026). Privacy and ethics complicate the matter further. Work on non-verbal vocalisations argues that strongly affective forms are hard to record in real-life private scenarios and that isolated, prompted, or acted exemplars fail to model them in context, leading to a preference for corpus-based approaches despite privacy and sparse-data problems (Batliner et al., 3 Aug 2025).

Output-side supervision is developing faster than silent input-side supervision. SynParaSpeech provides 118.75 hours of Chinese data across six paralinguistic categories with precise timestamps and improves both speech generation and paralinguistic event detection (Bai et al., 18 Sep 2025). NVSpeech contributes a manually annotated dataset of 48,430 human-spoken utterances with 18 word-level paralinguistic categories and an automatically annotated corpus of 174,179 utterances totaling 573 hours, enabling inline recognition and controllable synthesis of paralinguistic vocalizations (Liao et al., 6 Aug 2025). These corpora are not silent-speech corpora, but they provide the aligned paralinguistic targets that Direct SP and Indirect SP would need for expressive reconstruction.

Taken together, the literature portrays silent paralinguistics as an emerging but technically heterogeneous field. Its strongest empirical systems already decode words and sentences from accelerometers, EMG, ultrasound, EEG, and rtMRI; its synthesis systems increasingly separate content from identity, pitch, and style; and its dialogue models and benchmarks show that current audio-language systems still struggle to hear, preserve, and act on non-lexical cues (Xie et al., 25 Feb 2025, Li et al., 12 Feb 2026). The unresolved problem is therefore not simply silent speech recognition. It is the construction of systems that can decode unspoken linguistic intent, retain the speaker’s paralinguistic state, and render the result as socially appropriate, temporally calibrated, and expressive communication.