Sylber: Syllable-Level Speech Model

• Sylber is a self-supervised speech representation framework that organizes raw audio into syllable-level tokens for efficient, low-rate processing.
• It employs a student–teacher model with self-segmentation distillation to achieve nearly piecewise-constant embeddings within syllables.
• Sylber 2.0 extends the approach universally, improving reconstruction fidelity and supporting ASR, TTS, and articulatory control across 102 languages.

Sylber is a self-supervised speech representation framework that organizes raw audio at the syllable level rather than the conventional frame level. In its original form, it was introduced as “Syllabic Embedding Representation of Speech from Raw Audio,” with the explicit aim of producing framewise embeddings that become nearly piecewise constant within syllables, support linear-time segmentation, and yield low-rate syllabic token streams averaging 4.27 tokens per second (Cho et al., 2024). Subsequent work expanded the framework into Sylber 2.0, a universal syllable embedding and coding system operating at roughly 5 Hz across 102 languages, while related papers have used Sylber as an external syllable-level acoustic evaluator for articulatory control, or as a baseline in later work on syllable tokenization and spoken language modeling (Cho et al., 29 Jan 2026, Anand et al., 7 Oct 2025).

1. Original formulation and training objective

The original Sylber addresses a specific bottleneck in self-supervised speech modeling: standard SSL features such as HuBERT and WavLM are useful but remain dense, frame-level, and weakly structured, so their discretizations typically produce 25–75 Hz token streams that are expensive for downstream sequence modeling (Cho et al., 2024). Sylber’s design premise is that syllables are a more appropriate temporal unit because they are central to speech perception and production, align with a natural speaking rate of around 4–5 syllables/s in English, and provide a more economical substrate for lexical and syntactic modeling.

Architecturally, Sylber uses the same backbone family as HuBERT: a CNN feature extractor over raw waveform followed by a Transformer encoder. The model uses a 9-layer Transformer, initialized from SDHuBERT up to that layer. Training follows a student–teacher scheme in which the student model $M_S$ predicts framewise embeddings and the teacher $M_T$ is an exponential moving average (EMA) of the student. The key novelty is self-segmentation distillation: the teacher’s frame sequence is first segmented into pseudo-syllables by an unsupervised segmentation algorithm $\mathrm{Useg}$, those segments are average-pooled into segment targets, and the student is regressed toward the corresponding segment-level target for each frame. In the notation given in the paper,

$$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$

with frame-to-segment assignment

$$A(i)=j \quad \text{such that } s_{j,0}\le i < s_{j,1},$$

and teacher segment averages

$v_j = \frac{1}{p-q}\sum_{k\in [p, q]} z_k.$

The training loss is the framewise MSE objective

$$\mathcal{L}_{\text{SegDistill}} := \sum_i \|v^T_{A(i)}-z^S_i\|_2^2.$$

Operationally, this collapses intra-syllable variation while preserving inter-syllable contrast.

Training proceeds in two stages. Stage 1 uses fixed segment boundaries extracted once from SDHuBERT at initialization, with 1.15M steps, batch size 64, random 5s crops, learning rate $1\times 10^{-4}$, 500 warmup steps, and EMA decay 0.9995. Stage 2 switches to online segmentation using Sylber’s own greedy segmenter, adds 500k further steps, uses learning rate $5\times 10^{-5}$ and EMA decay 0.9999, and samples the merge threshold from $[0.8,0.9]$ during training. After training, thresholds are fixed to norm threshold 3.09 and merge threshold 0.8. A secondary denoising term is applied to 20% of student inputs, mixing in environmental noise with probability 75% or another speech sample with probability 25%; the paper states that this is not the primary source of syllabic structure, but substantially improves robustness to noisy audio.

2. Segmentation algorithm, tokenization, and coding efficiency

A central claim of Sylber is that the learned representation is sufficiently structured to permit a simple linear-time greedy segmentation algorithm rather than an $M_T$0 similarity-matrix procedure (Cho et al., 2024). The segmentation pipeline has three passes. First, speech versus non-speech is detected by thresholding frame embedding norms: $M_T$1 Second, a left-to-right monotonic agglomeration merges adjacent speech frames while their cosine similarity remains above a merge threshold $M_T$2. Third, each provisional boundary is locally refined by maximizing similarity to neighboring segment means: $M_T$3 The paper contrasts this $M_T$4 design with prior segmentation procedures reported as HuBERT: $M_T$5, VGHuBERT: $M_T$6, SDHuBERT: $M_T$7, and Komatsu et al.: $M_T$8.

Discrete tokenization is then obtained by averaging frame embeddings within each detected segment and clustering the segment embeddings with k-means. The reported vocabulary sizes are 5K, 10K, and 20K. Because the units are syllable-like rather than phoneme-like, the paper explicitly treats such vocabulary sizes as analogous to BPE vocabularies rather than alphabet inventories. The resulting average token frequency is 4.27 Tok/s, far below HuBERT-based baselines such as 23.59, 26.68, and 28.97 Tok/s for 50/100/200 clusters, and below SDHuBERT at 5.24 Tok/s.

The paper defines bitrate as

$M_T$9

and coding-rate as

$\mathrm{Useg}$0

For Sylber, the reported bitrates are 52.43, 56.70, and 60.97 at 5K/10K/20K, with corresponding coding-rates 0.0315, 0.0302, and 0.0289. These values outperform the reported SDHuBERT and HuBERT-BPE baselines in coding efficiency, supporting the paper’s claim that syllable-structured units reduce bandwidth without catastrophic information loss.

3. Empirical behavior and linguistic interpretation

On syllable detection and discovery, Sylber reports Precision 76.6, Recall 68.3, F1 72.2, and R-value 75.9, with discovery metrics SP 64.0, CP 43.9, and MI 5.28 (Cho et al., 2024). The paper states that this is the best overall segmentation/discovery performance among the compared systems, except that SDHuBERT attains slightly higher recall and is interpreted as oversegmenting. The stage-2 online self-bootstrapping mainly sharpens precision relative to stage 1.

The original paper also evaluates resynthesis from Sylber tokens. In the quantized 20K setting, the reported result is Articulatory corr 0.924, Loudness corr 0.882, Pitch corr 0.774, WER 7.95, CER 4.06, UTMOS 4.210, at 4.27 Tok/s. In the non-quantized setting, using continuous segment embeddings, performance improves to Articulatory corr 0.957, Loudness corr 0.950, Pitch corr 0.918, WER 4.88, CER 2.42, and UTMOS 4.199. The paper notes that quantization strongly reduces pitch fidelity and tends to flatten intonation.

For spoken language modeling, Sylber tokens improve unit LLMs on sWUGGY and sBLIMP. The reported values include Sylber-20K: 70.27 / 57.67 and Sylber-10K: 68.41 / 58.04, compared with GSLM: 68.70 / 57.06, tGSLM: 68.53 / 55.31, and SDHB-20K: 67.85 / 54.87. This places Sylber ahead of the listed baselines on sBLIMP, and ahead of GSLM and tGSLM on sWUGGY at 20K.

One of the paper’s most distinctive claims is that categorical perception emerges in Sylber without being explicitly trained. On interpolated continua constructed from 52 monosyllabic rhyming word pairs, the reported Discriminability Index (DI) is 0.112 for Sylber, better than SDHuBERT: 0.131, HuBERT: 0.141, WavLM Large: 0.140, MFCC: 0.188, and Mel: 0.196. The paper interprets this as evidence that Sylber’s embedding space is more categorical and sparse than standard SSL features. A cautious reading is that Sylber’s compression is not only temporal; it also appears to reshape representational geometry toward sharper phonological boundaries. At the same time, the appendix reports tradeoffs on general-purpose downstream tasks: speaker-sensitive and frame-sensitive tasks degrade, which the paper presents as a consequence of imposing syllabic parsimony.

The paper’s robustness claim is also nuanced. Its abstract states that the segmentation method is highly robust and generalizes to out-of-domain data and unseen languages without tuning. However, the detailed experimental discussion specifies that the main reported experiments are English-only and do not present a dedicated new unseen-language generalization experiment. This suggests that the original evidence for universality was stronger conceptually than directly documented in the main evaluation.

4. Sylber as an auditory feedback model in articulatory control

In “Teaching Machines to Speak Using Articulatory Control,” Sylber is not the motor controller, not the articulatory-to-audio decoder, and not an ASR system; it is a fixed acoustic feedback/perception module that closes the loop from synthesized audio back to a scalar reward for reinforcement learning (Anand et al., 7 Oct 2025). The overall system is explicitly framed as a closed-loop speech production pipeline: policy $\mathrm{Useg}$1 articulator velocities $\mathrm{Useg}$2 articulator positions/trajectory $\mathrm{Useg}$3 SPARC decoder $\mathrm{Useg}$4 waveform audio $\mathrm{Useg}$5 Sylber syllable detection + embedding $\mathrm{Useg}$6 similarity to target $\mathrm{Useg}$7 reward for PPO.

In that paper’s description, Sylber is a framework for “syllabic embedding representation of speech from raw audio.” It creates embeddings for syllables directly in speech audio, operates at the syllable level, includes an automatic syllable detection mechanism that identifies boundaries, and associates each detected syllable with a learned embedding. The articulatory RL environment uses the embedding of the most recently detected syllable and compares it to a fixed target syllable embedding. The reward is defined as cosine similarity: $\mathrm{Useg}$8 with the additional rule that if Sylber detects no syllable, then $\mathrm{Useg}$9. PPO uses these scalar rewards through its clipped surrogate loss; Sylber itself is not optimized.

The action space controls six articulators plus loudness: tongue dorsum, tongue blade, tongue tip, lower incisor, upper lip, lower lip, and loudness, for a total of 13 continuous action dimensions. The state uses frame stacking over the last 15 frames. The authors train separate PPO policies for six target syllables: please, road, fan, loot, cat, and age, with 25,000+ episodes per syllable, 50 timesteps per episode, and actor/critic MLPs. The abstract reports similarity scores exceeding 0.85, and the final table gives please 0.92, road 0.87, fan 0.79, loot 0.85, cat 0.89, and age 0.92. Human transcription confirms intelligibility for please, loot, and cat, but not perfect perceptual fidelity for all outputs: road $$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$0 roar, fan $$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$1 fand, and age $$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$2 we. This implies that Sylber embedding similarity is sufficient to drive learning of intelligible syllable-like outputs, but does not perfectly track human judgments on fine phonetic detail, especially syllable endings.

5. Sylber 2.0 and universal syllable coding

Sylber 2.0 generalizes the original framework from an English-centered syllabic representation into a universal syllable-level speech representation and coding framework (Cho et al., 29 Jan 2026). Its stated objective is broader than compression alone: the representation should be temporally efficient, multilingual, expressive enough for reconstruction and TTS, and useful for downstream tasks such as low-resource ASR. Relative to the original Sylber, the paper emphasizes six main changes: universality / multilingual training, a learned boundary detector, a separate acoustic encoder, removal of explicit silent masking, a stronger decoder with within-segment positional encoding (wSegPE) and Vocos, and a reframing as a full encoding–decoding speech coding system rather than only a syllable detector.

Each syllable-like segment is represented by three factors: duration $$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$3, content embedding $$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$4, and acoustic embedding $$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$5. Encoding operates directly on waveform through a content encoder, a boundary detector, and an acoustic encoder; frames inside each predicted segment are average-pooled to produce segment-level content and acoustic tokens. Decoding duplicates the segment embeddings back to 50 Hz, adds within-segment positional information, and synthesizes 24 kHz waveform. The content encoder is trained with teacher–student self-distillation using

$$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$6

followed by self-segmentation distillation

$$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$7

and the boundary detector is trained with binary cross-entropy against teacher-derived boundary labels. Short segments under 80 ms can be merged if sufficiently similar to neighbors. The student content model has 9 Transformer layers; the 8th layer of the teacher is used as the target feature source; and segment-averaged content embeddings are projected to 64 dimensions. The acoustic pathway uses a CNN front-end initialized from WavLM-Large followed by 6 Transformer layers, and its segment-level acoustic embeddings are also reduced to 64 dimensions.

A key headline result is the low token rate across languages: on FLEURS-R, Sylber 2.0 spans 3.2 Hz to 6.4 Hz, averaging 4.8 Hz across 102 languages. This is explicitly presented as lower than multilingual tokenizer baselines such as Mimi at 12.5 Hz, Vibe Voice at 7.5 Hz, and CLEAR at 7.7 Hz. On syllable detection, the paper reports improvements over the original Sylber: English F1 73.9 vs 72.2, Spanish 74.2 vs 71.7, and Mandarin 66.7, rising to 75.1 when short segments under 100 ms are filtered. The authors retain all segments for coding because reconstruction fidelity is prioritized over neat linguistic segmentation.

The reconstruction results are substantially stronger than those of the original system. On LibriTTS test-clean, Sylber 2.0 at 5.81 Hz reports WER 3.86, STOI 0.89, PESQ 1.99, UTMOS 3.80, and SSIM 0.92, compared with the original Sylber at 4.22 Hz and WER 5.44, STOI 0.75, PESQ 1.13, UTMOS 4.09, SSIM 0.76. On a 20-language average over FLEURS-R, Sylber 2.0 reports WER 7.57, STOI 0.92, PESQ 2.35, UTMOS 3.09, and SSIM 0.98, close to DAC: WER 6.03 and Mimi: WER 6.35 despite operating at a much lower token frequency. On GTSinger, the paper reports F0-PCC 0.96 and F0-$$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$8 0.88, indicating much better prosodic recovery than the original Sylber.

Sylber 2.0 also supports a TTS model, SylFlow, with 72M parameters, using an autoregressive rectified-flow architecture over continuous syllable tokens. In the controlled LibriTTS 560h setting, SylFlow 72M reports WER 3.10, SIM-o 0.31, and UTMOS 4.27 on LibriSpeech-PC; with an additional 47k hours of Emilia, it reaches WER 2.35 on LibriSpeech-PC and 1.92 on SeedTTS English, while remaining smaller than several cited SOTA systems. For low-resource ASR, Sylber 2.0 reports strong results on Korean, Bemba, and Quechua, including Korean WER 9.4, Bemba WER 47.4, and Quechua WER 66.4, with the paper explicitly noting that Bemba and Quechua are unseen in training. Its stated limitations are equally clear: speaker similarity in TTS still lags behind larger SOTA models, PESQ remains below high-frequency codecs, SUPERB performance is mixed, the discovered syllables are emergent rather than linguistically exact, and the variable-length segmentation pipeline remains more complex than simple feed-forward codecs.

6. Later comparisons, simplifications, and standardization

Later work treats Sylber as a central reference point for syllable-level spoken language modeling, but also as a target for simplification and modular analysis. “ZeroSyl” characterizes Sylber as a strong recent syllable-based tokenizer for pure speech language modeling, introduced to reduce the excessively long token sequences produced by standard SSL discretization, but criticizes it as learned, distilled, fine-tuned, multi-stage, specialized rather than training-free (Visser et al., 17 Feb 2026). In that framing, Sylber uses a sentence-level distillation objective to fine-tune HuBERT, after which syllable boundaries become accessible using cosine similarity between framewise features; it also performs a further distillation of boundary information by predicting piecewise-constant targets derived from mean-pooled embeddings over discovered segments. ZeroSyl replaces this with a frozen WavLM Large pipeline, boundary inference from L2 norms at layer 13, segment embeddings from layer 22, spherical K-means, and a causal OPT-125M LLM.

Empirically, ZeroSyl reports better scores than Sylber on all listed segmentation metrics: boundary F1 72 vs 69, R-value 75 vs 71, token F1 54 vs 51, and lower over-segmentation 10 vs 13. On syllable discovery, ZeroSyl reports SNMI 88.9 versus 83.5 for Sylber, and on spoken language modeling it reports sWUGGY (all) 68.0 vs 66.0, sWUGGY (IV) 78.6 vs 74.7, sBLIMP 60.5 vs 59.1, and tSC 68.1 vs 65.8. A plausible implication is not that syllabic tokenization was misguided, but that the syllabic idea can be recovered from simpler frozen-feature pipelines than Sylber originally employed.

The findsylls toolkit offers a different critique: rather than replacing Sylber, it decomposes it into separable components and benchmarks them under a common interface (Martínez, 27 Mar 2026). In that paper, Sylber is treated as a representation-driven syllabifier with two components: Sylber embeddings as the feature source and a greedy cosine-similarity thresholding algorithm as the default segmenter. This decomposition makes it possible to recombine Sylber features with other segmentation backends. The most important finding is that Sylber’s representation is stronger than its default segmentation algorithm in the matched benchmark. The default configuration, Sylber + Cos. Sim. + Cos. thresh., reports Nuclei F1 93.3, Boundary F1 63.1, Span F1 45.7, 3.4 tok/s, and 36x RTFx. Replacing the threshold-merging stage with peakdetect on the same cosine-similarity cue yields Boundary F1 69.9, Span F1 47.0, 3.9 tok/s, and 84x RTFx. By contrast, Sylber + featSSM + peakdetect performs poorly, with Boundary F1 43.3 and Span F1 13.1, indicating that local cosine similarity, rather than a self-similarity-matrix coherence trace, is the effective segmentation cue for Sylber in that benchmark.

Taken together, these later studies position Sylber as an important reference model in syllable-level speech tokenization: strong enough to serve as a baseline, modular enough to be recombined, but also sufficiently elaborate that later work uses it to motivate training-free alternatives and standardized evaluation pipelines.

The term should be distinguished from an unrelated chemistry usage concerning silabenzene-related motifs in surface-confined covalent organic framework chemistry. In that literature, the relevant topic is not Sylber the speech model, but Br-passivated 1,4-disilabenzene-linked C$$\mathrm{Useg}(z)=\{s\}^N,\qquad s_j=(s_{j,0},s_{j,1}),$$9Si$$A(i)=j \quad \text{such that } s_{j,0}\le i < s_{j,1},$$0 rings and their thermal conversion to C$$A(i)=j \quad \text{such that } s_{j,0}\le i < s_{j,1},$$1Si motifs on Au(111) (Sun et al., 2021).