Linguistic–Paralinguistic Dual Thinking

• Linguistic–paralinguistic dual thinking is a design that separates and couples lexical content with non-verbal cues like emotion, age, and background sounds.
• Key architectures include shared backbones with specialized heads, explicit chain-of-thought reasoning, and dual adapters that integrate both information streams.
• Advanced training approaches such as on-policy self-distillation and selective fine-tuning help models align verbal understanding with paralinguistic signals to improve response safety and expressivity.

Searching arXiv for papers on linguistic–paralinguistic dual mechanisms and closely related speech-LLM work. Speech-language systems increasingly treat spoken interaction as a joint inference problem over lexical-semantic content and non-lexical acoustic evidence. Within this setting, a Linguistic–Paralinguistic Dual Thinking Mechanism denotes an architectural or training principle in which a model separately preserves, explicitly reasons over, or structurally couples two information streams: the linguistic stream, concerned with propositional content, intent, and discourse; and the paralinguistic stream, concerned with age, emotion, prosody, speaker characteristics, background sounds, and related non-lexical cues. Recent work frames this mechanism not as a new sensory frontend but as a way of turning latent paralinguistic perception into response behavior, often by explicit chain-of-thought, dual heads, selective fine-tuning, or on-policy self-distillation (Wang et al., 9 Jun 2026, Geng et al., 13 Aug 2025, Yang et al., 12 Mar 2026).

1. Conceptual scope and definitions

In the contemporary speech-language literature, linguistic information is the verbal content conveyed by words and syntax, whereas paralinguistic information is the non-lexical information carried by speech beyond literal words. In "ParaBridge" the latter includes speaker identity and age markers, emotional tone and prosody, background audio context, and impaired cognitive capacity (Wang et al., 9 Jun 2026). In "OSUM-EChat" paralinguistic labels are explicitly organized as emotion, gender, age, and sound events such as laugh, cough, cry, scream, sigh, throat clearing, sneeze, and other (Geng et al., 13 Aug 2025). In "GOAT-SLM" the modeled dimensions include emotion, non-speech vocalizations, age, and language/dialect (Chen et al., 24 Jul 2025).

Across these systems, the dual mechanism is not reducible to speech emotion recognition. Its defining property is the coupling of non-lexical perception to downstream dialogue behavior. "ParaBridge" formulates the central problem as a perception–behavior gap: current speech LLMs can recognize paralinguistic cues but rarely use them to shape open-ended dialogue (Wang et al., 9 Jun 2026). "CP-Bench" describes the corresponding evaluation target as contextual paralinguistic reasoning, namely the integration of verbal content with non-verbal cues such as emotion and prosody to resolve meanings not recoverable from transcripts alone (Wang et al., 20 Sep 2025).

A second distinction concerns the meaning of dual thinking. In some work, it denotes a structural split between content and delivery, as in GOAT-SLM’s separation of “what to say” from “how to say it” through a shared semantic core and dual heads (Chen et al., 24 Jul 2025). In other work, it denotes explicit two-stage reasoning, as in OSUM-EChat’s chain-of-thought segment bounded by $\langle\text{think}\rangle$ and $\langle\text{think end}\rangle$, where the model first derives linguistic understanding and then paralinguistic understanding before response generation (Geng et al., 13 Aug 2025). A related but distinct lineage appears in hallucination mitigation, where HaluSearch implements a purely linguistic fast-versus-slow dual process and does not model paralinguistic signals; this is relevant mainly as a contrast case (Cheng et al., 2 Jan 2025).

2. Architectural realizations

Several architectural patterns recur in the literature. One pattern is a shared backbone with specialized heads. GOAT-SLM uses Whisper-small plus a projector for speech-to-LLM alignment, TeleChat2-7B as a shared semantic core, and separate Write and Speak heads. The bottom 15 layers serve as Think, the top 15 layers generate text in Write, and a speech head initialized from the write head performs expressive acoustic realization in Speak (Chen et al., 24 Jul 2025). This creates an explicit factorization between semantic intent and speaker-aware realization.

A second pattern is an end-to-end decoder with intertwined linguistic and paralinguistic control. LUCY uses a decoder-only Transformer backbone, Qwen2-7B-Instruct, conditioned on both textual tokens and audio-derived features, and emits one text token plus seven parallel streams of audio codec tokens at each step (Gao et al., 27 Jan 2025). Speaker and emotion tokens are inserted after the user input and before the reply, so that linguistic control signals determine subsequent paralinguistic realization. This suggests a tight rather than modular integration: the two streams are not separate subnetworks, but are intertwined in a shared autoregressive process.

A third pattern is dual adapters feeding a frozen LLM. "Dual Information Speech LLMs for Emotional Conversations" introduces two heterogeneous adapters between Whisper-large-v3 and a frozen LLM. The paralinguistic adapter produces fixed-length soft prompts, while the linguistic adapter produces time-varying content embeddings (Wang et al., 11 Aug 2025). The frozen LLM then arbitrates between them through attention over a structured concatenation of context embeddings, paralinguistic prompts, linguistic embeddings, and textual tokens. This preserves the base LLM’s contextual reasoning while avoiding early entanglement.

A fourth pattern is intermediate thought decoding. MeTHanol’s TaS model adds a language head at an intermediate transformer layer so that the model explicitly generates a thought sequence before the final response (Xi et al., 2024). The paper frames this primarily as reasoning and planning rather than speech paralinguistics, but it is frequently interpretable as a precursor to dual-mechanism designs because it separates pre-verbal cognitive content from overt linguistic realization.

A fifth pattern is layer-selective adaptation of shared representations. "Resurfacing Paralinguistic Awareness in Large Audio LLMs" identifies paralinguistic-salient layers and semantic-understanding layers in Qwen2.5-Omni and Kimi-Audio, then fine-tunes only those layers jointly (Yang et al., 12 Mar 2026). Here the dual mechanism is not expressed as two visible heads at inference; instead, it is a training-time intervention that forces semantic understanding to remain conditioned on early paralinguistic evidence.

3. Training objectives and coupling mechanisms

The central methodological question is how to force a model to use paralinguistic information when generating responses. ParaBridge answers this with on-policy self-distillation (Wang et al., 9 Jun 2026). Let $c$ denote the scaffold-free audio dialogue context and $S(c)$ a scaffolded context formed by prepending a paralinguistic instruction. The student policy is

$$p_\theta(y_t \mid h_t) = \pi_\theta(\cdot \mid c, y_{<t}),$$

while the teacher’s privileged distribution is

$$q_\phi(y_t \mid h_t, S(c)) = \operatorname{sg}\!\left[\pi_\theta(\cdot \mid S(c), y_{<t})\right].$$

The student rolls out its own tokens under the scaffold-free context, and the scaffolded teacher provides dense full-vocabulary next-token targets along the student trajectory. The main loss is the mean per-token Jensen–Shannon divergence,

$$L_{\text{ParaBridge}}(\theta)=\mathbb{E}_{a,y\sim\pi_\theta(\cdot\mid c)}\left[\frac{1}{T}\sum_{t=1}^T \operatorname{JSD}(p_t \,\|\, q_t)\right],$$

with softened logits using $\tau = 1.2$ and no auxiliary supervised loss, i.e. SFT auxiliary weight $=0$ (Wang et al., 9 Jun 2026). The mechanism is explicitly described as coupling the student’s linguistic generation to a paralinguistically aware privileged view token by token.

OSUM-EChat instead uses an explicit chain-of-thought bridge. In Stage 3, the model first outputs a supervised $$\langle\text{think}\rangle \ldots \langle\text{think end}\rangle$$ segment and then generates the dialogue response (Geng et al., 13 Aug 2025). Its Stage 3 objective is

$\langle\text{think end}\rangle$0

with Structured Sparse Learning masking positions whose labels are "unk" so that only known label slots contribute gradients. This makes the linguistic and paralinguistic reasoning trace observable and directly optimizable.

LUCY uses a more conventional joint next-token objective over text and audio streams. Its loss is written as

$\langle\text{think end}\rangle$1

which is approximately decomposed into one text head and seven audio-codebook heads (Gao et al., 27 Jan 2025). Emotion control is not supervised by a separate classifier; it is learned because emotion tokens are part of the text stream and condition later codec-token prediction.

GOAT-SLM uses a staged training strategy rather than a single closed-form objective. Stage 1 instruction-tunes the text model to respond to explicit vocal attributes; Stage 2 aligns speech input to both content and attributes; Stage 3 trains the speech head for high-fidelity expressive generation while freezing Listen and Think (Chen et al., 24 Jul 2025). The paper also provides formalizations of a possible fusion controller and decorrelation regularizer, but explicitly notes that these are compatible formalizations rather than implemented modules.

The dual-information SLM paper uses Equivalence Replacement Regularization (ERR), a weakly supervised invariance scheme in which the complementary stream is randomly replaced with speech-derived embeddings, text-derived embeddings, or none, while the active stream is trained for the task at hand (Wang et al., 11 Aug 2025). This prevents the model from collapsing to brittle task-specific vectors and preserves dialogue-level contextual understanding.

4. Mechanistic interpretations

Mechanistic analyses increasingly localize where dual integration occurs. ParaBridge reports that the backbone SLM already contains latent audio representations sufficient for paralinguistic perception, and that the main behavioral change is in how late-layer read-out maps these representations into output tokens (Wang et al., 9 Jun 2026). CKA and activation-patching analyses place the change in the final two layers before the LM head, implying that ParaBridge preserves the perceptual stack and retunes output use of existing representations rather than learning a new detector.

The PE-FT work reaches a complementary conclusion through five layer-wise analyses on 28-layer LALMs (Yang et al., 12 Mar 2026). Linear probes identify layers 0–6 as paralinguistic-salient; intent classification and cosine-similarity analyses identify layers 7–14 as semantic-understanding layers; logit-lens analysis places strongly generative behavior in deeper layers, especially from 19–21 onward. This yields a concrete dual decomposition:

Functional span	Layers	Reported role
Paralinguistic-salient	0–6	Age, gender, emotion signals
Semantic-understanding	7–14	Intent and semantic differentiation
Generative	>14, especially 19–21+	Token prediction and contextual generation

On the basis of this decomposition, PE-FT trains only layers 0–14 and adds an auxiliary dual-level classification head at layer 14 (Yang et al., 12 Mar 2026). The paper’s interpretation is that early paralinguistic signals are normally suppressed by content-centered training; selective adaptation reintroduces them into semantic formation without perturbing generative fluency.

GOAT-SLM offers a more macro-architectural mechanism: the shared representation $\langle\text{think end}\rangle$2 is produced by Think, after which Write derives linguistic content and Speak derives paralinguistic realization (Chen et al., 24 Jul 2025). An important empirical observation is that dialectal traits can propagate into Speak even when Write produces standard text, indicating that the shared core transmits non-lexical social information independently of overt textual form.

LUCY’s mechanism is similar in spirit but more tightly coupled. Audio encoder features derived from 80-bin mel spectrograms flow through a convolutional adapter into the LM token space, and emotion/speaker tokens predicted in the text stream condition subsequent audio-token generation (Gao et al., 27 Jan 2025). Because one text token and seven audio tokens are generated in parallel each step, prosodic realization is inseparable from the hidden-state dynamics of textual planning.

OSUM-EChat’s mechanism is the most explicit in terms of reasoning format: linguistic thinking first captures utterance semantics and intent, then paralinguistic thinking enumerates emotion, gender, age, and sound events, after which generation proceeds (Geng et al., 13 Aug 2025). No separate gating module is used; fusion occurs inside the LLM’s supervised chain-of-thought tokens.

5. Empirical performance and evaluation regimes

Empirical evidence for dual mechanisms comes from both task-specific gains and reduced mismatch between perception and behavior. ParaBridge reports large improvements on scaffold-free dialogue while preserving general ability (Wang et al., 9 Jun 2026). On Qwen3-Omni-thinking, VoxSafeBench SAR rises from 14.64\% to 40.33\%, and EchoMind average rating rises from 3.27 to 3.92. General ability remains close to the original model: MMAU-Pro changes by −0.22, VoiceBench by −0.35, and GPQA by +0.09. The same paper reports that JSD performs best, that an audio teacher outperforms a text-only teacher, and that false-alarm rate drops to 3.36\%, below scaffolded baseline 6.05\% and RFT 6.42\%.

OSUM-EChat evaluates empathetic spoken dialogue on EChat-eval (Geng et al., 13 Aug 2025). Its automatic scores are Emotion 58.0, Age 63.1, Gender 62.7, Event 87.1, and Multi-label 72.0. Human evaluation reports Emotion 72.0, Age 68.9, Gender 65.8, Event 88.0, and Multi 73.3. Ablations show that removing either the understanding-driven strategy or dual think degrades performance, and removing both yields particularly severe collapse on the Event dimension, down to 10.4, indicating that explicit dual reasoning is especially important for non-speech acoustic events.

LUCY provides evidence that dual linguistic–paralinguistic control can coexist with low-latency speech interaction (Gao et al., 27 Jan 2025). On Linguistic Emotion Control, Chinese accuracies reach 99.45 for Neutral and remain high across Joy, Anger, Fear, Sadness, Surprise, and Sorry; Acoustic Emotion Control is lower but nontrivial, for example 79.54 for Neutral and 66.67 for Joy in Chinese. In web deployment, average latency is 0.4652 s end-to-end, with first audio chunk at 0.3590 s and per-step decoding at 0.0227 s.

GOAT-SLM emphasizes balanced semantic and non-semantic evaluation through TELEVAL (Chen et al., 24 Jul 2025). It reports Dialect Following average 50.73, far above the next-best 18.91, NSV performance 40.91, and Age-aware performance 72.13. On spoken response quality for ESD-zh, GOAT-SLM achieves CER 1.57, DNSMOS 3.46, and Emotion 61.48. These results support the view that decoupling text reasoning and expressive delivery can preserve broad conversational competence while materially improving socially aware response modulation.

CP-Bench serves as an external benchmark for whether such systems truly perform contextual paralinguistic reasoning (Wang et al., 20 Sep 2025). It divides questions into Content-only, Direct Paralinguistic, and Contextual Paralinguistic categories, and evaluates with Unweighted Average Score (UAS):

$\langle\text{think end}\rangle$3

Its results show that strong content models can remain weak on direct paralinguistic tasks, and that multimodally balanced systems such as MERaLiON and Gemini are often better calibrated across content and paralinguistic reasoning types (Wang et al., 20 Sep 2025). This benchmark is significant because it distinguishes simple affect recognition from dialogue-level reasoning that jointly depends on lexical content and audio cues.

6. Variants, misconceptions, and limitations

A common misconception is that a dual thinking mechanism necessarily requires two separate networks. This is contradicted by multiple systems. LUCY uses a single decoder-only Transformer with special control tokens and parallel heads rather than two isolated models (Gao et al., 27 Jan 2025). ParaBridge uses the same SLM backbone for both student and teacher, differing only by whether a temporary paralinguistic scaffold is present (Wang et al., 9 Jun 2026). OSUM-EChat fuses both reasoning modes within one chain-of-thought segment (Geng et al., 13 Aug 2025).

A second misconception is that paralinguistic modeling is equivalent to adding an emotion classifier. The stronger recent systems do not merely classify emotion; they connect non-lexical perception to response strategy. ParaBridge changes unsafe child-directed answers into safer redirections, notices child presence in the background when recommending media, and reacts to affective incongruity such as laughter when describing a fatal disaster (Wang et al., 9 Jun 2026). GOAT-SLM similarly responds to non-speech vocalizations and age cues rather than only labeling them (Chen et al., 24 Jul 2025).

A third misconception is that inference-time prompting is sufficient. ParaBridge explicitly shows that scaffold prompts reveal latent ability but remain fragile under multi-turn context, competing instructions, prompt injection, indirect prompt attacks, and audio adversarial jailbreaks (Wang et al., 9 Jun 2026). Its central claim is therefore not that prompting discovers a new capability, but that training is needed to internalize it.

The literature also identifies substantial limitations. ParaBridge notes that gains depend on a sizable scaffolded–scaffold-free gap, that cue coverage remains limited, and that multi-turn improvements are still modest in absolute terms (Wang et al., 9 Jun 2026). PE-FT notes ethical and representational issues in treating gender as binary from vocal characteristics, as well as privacy risks in inferring age, gender, and emotion (Yang et al., 12 Mar 2026). LUCY reports lower Acoustic Emotion Control on underrepresented emotions such as Disgust, for example 28.57 in Chinese and 12.50 in English (Gao et al., 27 Jan 2025). GOAT-SLM reports uneven dialect coverage, with Northeastern Mandarin subjective dialect generation consistency at 57.2, well below other dialects (Chen et al., 24 Jul 2025). OSUM-EChat notes that dynamic paralinguistic scenarios, such as emotion transitions and multi-speaker interactions, remain underexplored (Geng et al., 13 Aug 2025).

A plausible implication is that current dual mechanisms are strongest when the relevant paralinguistic variables are relatively coarse and enumerable—age category, broad emotion class, dialect label, or obvious sound event—but remain incomplete for subtler social-pragmatic phenomena such as sarcasm, politeness, fatigue, intoxication, or culturally specific prosodic norms.

7. Research trajectory and broader significance

The recent trajectory of the field suggests a shift from paralinguistic perception as an auxiliary task to paralinguistic behavior alignment as a primary objective. Earlier or parallel systems such as TaS and HaluSearch explore general dual-process or think-then-speak designs at the level of internal reasoning, but they do not directly solve speech-conditioned social interaction (Xi et al., 2024, Cheng et al., 2 Jan 2025). In contrast, ParaBridge, OSUM-EChat, GOAT-SLM, LUCY, PE-FT, and dual-information SLMs all attempt to make non-lexical auditory evidence actionable in generation.

Methodologically, three directions are especially visible. The first is explicit reasoning traces, as in OSUM-EChat’s supervised dual-think format (Geng et al., 13 Aug 2025). The second is structural decoupling with shared semantics, as in GOAT-SLM’s Think–Write–Speak architecture and the heterogeneous-adapter design of dual-information SLMs (Chen et al., 24 Jul 2025, Wang et al., 11 Aug 2025). The third is behavioral distillation or selective adaptation, exemplified by ParaBridge’s on-policy JSD distillation and PE-FT’s layer-targeted fine-tuning (Wang et al., 9 Jun 2026, Yang et al., 12 Mar 2026).

These developments matter because spoken dialogue safety, empathy, and social appropriateness often depend on cues that transcripts omit. Child voice, frightened tone, incongruous laughter, background hazards, coughs, or dialectal identity can all change what counts as an appropriate response. The dual mechanism is therefore best understood not as an aesthetic addition for expressive speech generation, but as an attempt to align spoken LLMs with the full informational content of speech.

This suggests a broader redefinition of spoken intelligence. A speech agent that only extracts transcripts is functionally incomplete; a speech agent that perceives paralinguistic cues but fails to let them influence behavior remains misaligned; and a speech agent with a mature linguistic–paralinguistic dual thinking mechanism is one in which semantic understanding and user-context inference are jointly represented, behaviorally coupled, and robust under open-ended dialogue (Wang et al., 9 Jun 2026, Wang et al., 20 Sep 2025).