Beyond-Semantic Speech (BoSS)

• Beyond-Semantic Speech (BoSS) is a framework that extends traditional speech models by incorporating explicit semantics along with affective cues, contextual dynamics, and implicit meanings.
• It employs relevance theory and neural scoring functions to balance cognitive effects with processing effort, thereby enriching the interpretation of spoken interactions.
• Empirical evaluations reveal that while models show strength in context memory, they struggle with emotion recognition, dialect generation, and non-verbal cue preservation.

Beyond-Semantic Speech (BoSS) denotes “the set of information in speech communication that encompasses but transcends explicit semantics.” In this formulation, speech conveys not only lexical, syntactic, and logical content, but also emotions, contexts, and meanings modified or extended through vocal characteristics, dynamic contexts, and implicit semantics. The concept was introduced together with Spoken Interaction System Capability Levels (L1–L5) to explain why modern speech technologies, including ASR, TTS, and spoken LLMs, often remain weaker at human-like spoken interaction than their text-centric performance suggests (Wang et al., 23 Jul 2025).

1. Scope and relation to adjacent fields

BoSS integrates four primary dimensions: Explicit Semantics, Affective Cues, Contextual Dynamics, and Implicit Semantics. Explicit semantics is the “foundational layer comprising lexical, syntactic, and logical structures that convey direct meaning through factual statements or clear commands.” Affective cues include “the emotional subtext carried through vocal characteristics (pitch, rhythm, volume) and non-verbal vocalizations (laughter, sighs).” Contextual dynamics covers environmental and interactional context, including background sounds, acoustic properties of space, conversational rhythm, and shared interaction history. Implicit semantics includes inferred meanings such as semantic modifications, indirect intent, sarcasm, irony, and identity cues that allow listeners to deduce social roles and relationships (Wang et al., 23 Jul 2025).

Within this scope, BoSS is broader than paralinguistics, broader than emotion recognition, and broader than prosody modeling. It overlaps strongly with pragmatics, especially in sarcasm, indirectness, relevance, and inference, but is framed specifically as a speech-centered account of communicative meaning. The framework repeatedly emphasizes that listeners draw on tone, intonation, rhythm, volume, pauses, turn-taking, background or environmental sound, speaker traits and social roles, conversational history, indirectness, sarcasm, irony, and implication when interpreting spoken interaction (Wang et al., 23 Jul 2025).

Adjacent research helps locate BoSS historically but does not exhaust it. Visually grounded semantic speech retrieval showed that untranscribed speech paired with images can support “semantic speech retrieval,” including non-verbatim semantic matches, but that work remained on the semantic and grounded side rather than the broader beyond-semantic side of speech (Kamper et al., 2017). Likewise, semi-supervised spoken language understanding from speech demonstrated direct prediction of intent classification and slot labeling from speech, yet it still treated semantics primarily as intent-plus-slot structure rather than the wider BoSS space of affect, context, and implicit meaning (Lai et al., 2020).

2. Formal framework and theoretical basis

The theoretical basis of BoSS is explicitly tied to Sperber and Wilson’s Relevance Theory and to pragmatic inference. The paper gives the relevance formula

$R = \frac{E}{P},$

where $R$ is relevance, $E$ is cognitive effect, and $P$ is processing effort. Communication is therefore modeled as selecting the most plausible interpretation of speech by balancing informativeness against interpretive effort (Wang et al., 23 Jul 2025).

The framework represents a spoken signal at time $t$ through the observation vector

$O_t = [V_{L,t}, V_{AC,t}, V_{CD,t}, V_{IS,t}],$

where $V_{L,t}$ is explicit semantics, $V_{AC,t}$ affective cues, $V_{CD,t}$ contextual dynamics, and $V_{IS,t}$ implicit semantics. External context is represented as

$R$0

with conversational history, environmental factors, speaker or listener characteristics, and task or domain knowledge as the four contextual components (Wang et al., 23 Jul 2025).

BoSS then seeks an optimal meaning hypothesis $R$1 by maximizing cognitive effect relative to processing effort over a hypothesis space. The paper also proposes neural scoring functions for effect and effort, and an HMM-style temporal model whose emission distribution is defined through relevance-based scoring. The printed equations contain typographic issues in the manuscript, but the intended structure is explicit: BoSS is formulated as time-varying inference over hypotheses conditioned on both signal-side and context-side variables (Wang et al., 23 Jul 2025).

A second formal strand is an end-to-end spoken LLM view based on mutual information. In that view, a speech encoder $R$2, an LLM $R$3, and a TTS decoder $R$4 should preserve not only explicit semantics but the broader information carried by $R$5, $R$6, $R$7, and $R$8. This places BoSS within a general information-preservation perspective rather than a purely symbolic or purely acoustic one (Wang et al., 23 Jul 2025).

3. Capability levels and empirical probes

The Spoken Interaction System Capability Levels organize spoken systems into five levels. Level 1 is speech command execution; Level 2 is task-oriented dialogue; Level 3 is contextual understanding dialogue; Level 4 is emotion-aware dialogue; and Level 5 is human-level conversational intelligence. The hierarchy moves from basic keyword-based or predefined command execution toward open-domain interaction that perceives implicit intentions, responds to emotion and social context, self-adjusts under conflict or ambiguity, and maintains consistency over long histories. In the framework, BoSS becomes increasingly necessary at Levels 4 and 5 (Wang et al., 23 Jul 2025).

The empirical side of the original BoSS paper evaluates five BoSS-related dimensions rather than only the four formal dimensions. These five are Chinese dialect comprehension and generation, context memory, emotion perception and response, age perception and response, and non-verbal information. Results show that current spoken LLMs are relatively stronger on context memory than on the other dimensions: in the reported benchmark, context-memory accuracy ranged from 20.00 for SpeechGPT-2.0-preview to 88.67 for Qwen2.5-Omni. Emotion perception and response remained limited, with textual-response scores ranging from 13.55 to 53.17 and audio-response scores from 24.74 to 52.59. Age perception and response was weaker still, ranging from 12.24 to 42.51. Non-verbal information was the weakest dimension by far, with scores from 1.52 to 9.19 (Wang et al., 23 Jul 2025).

The same evaluation also reports that dialect comprehension can be reasonably strong while dialect generation and following remain poor. The paper’s own summary is that many spoken LLMs can extract key information from dialogue history, but remain weak on dialect following, age-aware adaptation, and especially non-verbal cue response. The resulting picture is not that speech models fail uniformly, but that their gains are uneven and still concentrated on text-like or memory-heavy capabilities rather than on rich beyond-semantic interpretation (Wang et al., 23 Jul 2025).

4. Representations, control interfaces, and transmission schemes

A major line of BoSS-related work recasts speech systems around richer control interfaces. “Borderless Long Speech Synthesis” proposes a “Borderless Long Audio Synthesis” framework with a Global-Sentence-Token annotation schema. Its data philosophy is “Labeling over filtering/cleaning,” with reported data utilization above 90%, and its representational aim is to make show format, style tags, speaker profiles, emotional trajectory, acoustic environment, sound events, tone, intonation, speed, volume, speaking intent, background state, and token-level pronunciation details first-class inputs. The same work frames the hierarchy as a structured semantic interface between an LLM agent and an overview engine, and adds a two-stage “think, then speak” process with Chain-of-Thought and Dimension Dropout to improve instruction following under complex conditions (Song et al., 20 Mar 2026).

A related tokenizer-centered line addresses what one paper calls acoustic blindness in semantic speech tokenizers. UniAudio-Token retains a single codebook, with vocabulary size 8,192 and token frame rate 25 Hz, but augments semantic tokenization with Semantic-Acoustic Primitives (SAP) and Semantic-Acoustic Equilibrium (SAE). SAP decomposes audio into Linguistic Content, Vocal Attributes—Age, Gender, Emotion, Accent, Prosody, Timbre—and Auditory Scene. SAE injects shallow acoustic information into deep semantic representations with a content-aware gate, aiming to make a semantic tokenizer more suitable as a unified audio interface rather than a speech-only interface (Song et al., 29 May 2026).

Another BoSS-relevant strategy is to stop before waveform synthesis and operate directly on speech-semantic latents. Speechless trains a model that maps text to discrete semantic tokens obtained by quantizing Whisper encoder representations with a Residual Vector Quantizer, then uses those tokens to fine-tune an LLM for spoken instruction following without generating waveforms. The paper describes this as “halting synthesis at the semantic representation level,” and shows that synthetic semantic token sequences can stand in for spoken instruction data in low-resource settings (Dao et al., 23 May 2025).

Communication-oriented work makes a similar split between semantic and beyond-semantic information. “Semantic-preserved Communication System for Highly Efficient Speech Transmission” explicitly separates semantic-relevant information from semantic-irrelevant but speech-related information. In its speech-to-speech mode, the additional side information includes duration, pitch, and power, represented as

$R$9

showing that rhythm, intonation contour, and loudness are treated as minimal beyond-semantic structure needed for more faithful reconstruction (Han et al., 2022). LargeSC extends the transmission perspective through Mimi tokens, adaptive transmission, in-band unequal error protection, and LoRA-finetuned Moshi recovery, with reported operating bandwidths from 550 bps to 2.06 kbps and end-to-end latency of approximately 460 ms, illustrating a tokenized, generative speech-transmission regime that preserves more than transcript content alone (Tian et al., 4 Dec 2025).

5. Evaluation and benchmarking beyond text

Benchmarking is one of the clearest sites where BoSS becomes operational. STEB is a 32.6-hour Chinese–English benchmark for speech-to-speech translation expressiveness that evaluates not only translation fidelity, speaker similarity, and duration alignment, but also emotion consistency, scenario style consistency, and NV preservation. Its results show a gap between semantic transfer and expressive transfer: many systems achieve strong translation fidelity, but the reported best emotion-preservation score is 3.82/5, while the best NV-preservation score is only 2.31/5. The benchmark therefore identifies expressiveness preservation as an open challenge rather than a solved auxiliary problem (Cheng et al., 24 Jun 2026).

OpenSTBench generalizes this multidimensional stance across S2TT, S2ST, offline, and streaming settings. It jointly evaluates translation quality, speech quality, speaker preservation, emotion and paralinguistic fidelity, temporal consistency, and latency. Its empirical finding is that systems with strong translation quality can still differ substantially in speech quality and temporal quality. For BoSS, the crucial point is methodological: evaluation is no longer centered on BLEU-like semantic correctness alone, but on a profile of semantic and beyond-semantic behavior (An et al., 29 May 2026).

Task-specific beyond-semantic evaluation has also expanded outside translation. In the EMotion Share track, emotion understanding is treated as a perception problem with a 9-dimensional target of emotion shares rather than a single categorical label. In that setting, HuBERT-Large with a CNN + 2-layer LSTM + self-attention + FFNN reached average Spearman $E$0, a 4.6% relative improvement over the challenge baseline average of 0.500, showing that beyond-semantic tasks can require different representation choices than ASR-centered ones (Mohapatra et al., 2023). Unsupervised speech segmentation has likewise been reframed around “acoustic-semantic styles” rather than phones or single-task diarization: using a pretrained HuBERT tokenizer and a 350M TWIST speech LLM, PMI-based boundary scoring with 0.5-second “acoustic-sentences” outperformed unsupervised baselines on emotion- and gender-change segmentation (Elmakies et al., 7 Jan 2025).

6. Limitations, unresolved issues, and likely directions

The central limitation identified by the original BoSS paper is that current spoken LLMs still “treat speech predominantly as a sequence of textual tokens.” Empirically, this shows up in the sharp weakness of current systems on non-verbal information, weak age-aware response, and poor dialect generation or following, even when context memory and some emotion perception have improved (Wang et al., 23 Jul 2025).

Systems papers around BoSS expose additional gaps. Borderless long speech synthesis is rich in representation and interface design, but its manuscript does not provide formal serialization standards, detailed label-generation procedures, enough architectural detail about the continuous tokenizer or acoustic space to fully reproduce the system from that paper alone, or strong quantitative evidence for improved control over emotion, overlap, interruption, discourse coherence, or environment. It is also optimized for offline content creation rather than real-time interaction (Song et al., 20 Mar 2026). UniAudio-Token shows that single-codebook tokenizers can be made more general-audio-capable, but the paper explicitly states that waveform-level reconstruction quality for complex non-speech audio still trails specialized high-bitrate acoustic codecs, and training and evaluation are mainly in English and Chinese (Song et al., 29 May 2026).

Benchmarking remains imperfect as well. STEB is limited to Chinese↔English and depends on captioning, NV detection, and LLM-based judging, while OpenSTBench relies on automatic proxies such as UTMOS, Whisper-based CER/WER, Resemblyzer, WavLM, Emotion2Vec, and CLAP. Both papers treat these choices as practical necessities, but they also make clear that human-aligned beyond-semantic evaluation is still methodologically open (Cheng et al., 24 Jun 2026, An et al., 29 May 2026).

Finally, some communication and reconstruction work shows that going beyond semantics is possible in a narrow sense without yet reaching full BoSS. The semantic-preserved communication system adds duration, pitch, and power as a compact side channel, but does not directly evaluate speaker identity preservation, emotion preservation, or richer paralinguistic structure in a controlled way. This suggests that a minimal beyond-semantic augmentation and a mature BoSS representation are not the same thing (Han et al., 2022).

A plausible implication is that future BoSS systems will need three elements at once: explicit multi-level representations that separate semantics, affect, context, and implicit meaning; training and annotation procedures that preserve real-world interactional and environmental structure instead of filtering it away; and evaluation protocols that measure expressive, paralinguistic, temporal, and social fidelity alongside semantic correctness. Across the current literature, the idea of speech as a multidimensional communicative signal is now clear; what remains unresolved is how to formalize, scale, and validate that signal with the rigor already customary for text-centered language modeling.