HubertSoft: Soft-Cluster Speech Encoder

• HubertSoft is a soft-clustering variant of HuBERT that uses posterior probabilities over learned acoustic units instead of hard cluster assignments.
• It generates boundary-sensitive embeddings by preserving sub-segmental phonetic features, including adjacent phoneme identity, order, and coarticulatory details.
• Empirical tests on the CORPRES Russian corpus reveal high ordered and unordered accuracies, underscoring its potential for fine-grained phonological analysis and low-resource applications.

Searching arXiv for HubertSoft and the cited 2025 paper to ground the article. HubertSoft is a soft-clustering variant of HuBERT in which the masked prediction objective is defined over soft posterior probabilities over learned acoustic units rather than over hard cluster assignments. In the analysis reported in "Revealing the Hidden Temporal Structure of HubertSoft Embeddings based on the Russian Phonetic Corpus" (Ananeva et al., 9 Jul 2025), the model is examined not merely as a frame-level phonetic encoder, but as a source of boundary-sensitive embeddings whose short-window representations retain information about adjacent phoneme identity, temporal order, and coarticulatory structure. The resulting picture places HubertSoft within the broader family of self-supervised speech models while emphasizing a specific representational property: embeddings extracted near phoneme boundaries can encode how a short segment is situated relative to neighboring sounds.

1. Model definition and relation to HuBERT

HubertSoft is described as a HuBERT-style model with a transformer encoder operating on raw waveforms, but with a different target representation. Standard HuBERT predicts discrete hidden units obtained from hard clustering, so that each frame is assigned a single cluster ID. HubertSoft instead uses soft cluster assignments, representing each frame by a probability distribution over clusters rather than by a single index (Ananeva et al., 9 Jul 2025). In that sense, HubertSoft remains closer in spirit to HuBERT than to Wav2Vec 2.0, but it preserves a soft mixture of acoustic prototypes where HuBERT commits to one discrete prototype per frame.

The distinction is consequential at the level of representation. The source summarizes HubertSoft as generating “soft posterior probabilities over learned acoustic units instead of hard cluster assignments.” This means that the hidden-unit target is no longer a categorical label alone, but a distribution over a codebook. A standard notation given in the source description is

$$\mathbf{q}_t = (q_{t,1}, \dots, q_{t,K}), \quad \sum_k q_{t,k} = 1, \ q_{t,k} \ge 0,$$

with the model predicting a corresponding distribution $\mathbf{p}_t$ for masked positions. The source further summarizes the training objective as a soft-target variant of cross-entropy or KL divergence, written in standard notation as

$$\mathcal{L}_{\text{HubertSoft}} = - \sum_{t \in \mathcal{M}} \sum_{k=1}^K q_{t,k} \log p_{t,k}.$$

The following comparison condenses the distinction as presented in the source material:

Model	Unit target type	Frame representation
Wav2Vec 2.0	Continuous latent representations	Raw-waveform SSL features
HuBERT	Hard cluster assignments	One discrete cluster ID per frame
HubertSoft	Soft cluster assignments	Posterior distribution over clusters

Within the 2025 study, HubertSoft is not modified architecturally during experimentation. It is treated as a frozen pretrained encoder, and the investigation concerns what information is already present in its embeddings rather than how to improve the pretraining procedure (Ananeva et al., 9 Jul 2025).

2. Acoustic units, embeddings, and the role of soft clustering

The central representational difference in HubertSoft lies on the target side rather than in the general encoder design. Acoustic frames are mapped into a latent space, and a learned codebook or mixture mechanism assigns posterior probabilities over units to each frame. These soft assignments serve both as targets for masked prediction and as a representation of frame content in downstream use (Ananeva et al., 9 Jul 2025).

In the Russian phonetic study, HubertSoft is used only as a source of embeddings. The clustering mechanism is not retrained, and all HubertSoft weights remain frozen. Raw audio waveforms from the CORPRES corpus are passed through the encoder to obtain frame-level embeddings. The study distinguishes two uses of these embeddings. For an initial baseline, embeddings are averaged within each phonetic segment to yield one vector per labeled sound. For the main experiments, however, the analysis uses non-averaged 20 ms resolution embeddings, extracted from consecutive non-overlapping 20 ms windows (Ananeva et al., 9 Jul 2025).

This design choice is fundamental because a 20 ms window may fall entirely within a phoneme or may straddle a phoneme boundary. The resulting embedding is therefore potentially sensitive to both local phonetic content and the internal temporal arrangement of that content. The source explicitly frames these as boundary-sensitive embeddings: each vector corresponds to a short time slice that may contain the end of one phoneme and the beginning of another. A common misconception would be to treat SSL speech embeddings as static phoneme fingerprints. The evidence presented here instead supports a more structured view in which HubertSoft embeddings can retain sub-segmental temporal information (Ananeva et al., 9 Jul 2025).

3. Boundary-sensitive probing on the CORPRES Russian corpus

The empirical study uses CORPRES, the Corpus of Russian professionally read speech, with manual phonetic alignment and phonetic labeling by trained phoneticians. The analysis uses eight speakers, split speaker-independently into four for training and four for testing. The corpus is particularly suited to fine-grained probing because it contains phoneme-level time alignment, although the source notes that boundaries between acoustically similar sounds can be imprecise and are sometimes placed at the midpoint of an acoustic transition (Ananeva et al., 9 Jul 2025).

Each 20 ms window is assigned a triplet label based on the phoneme occupying three positions in time: the start, the centre, and the end of the window. In the notation used in the source,

$$\text{label}(\mathbf{e}) = (\phi_{\text{start}}, \phi_{\text{centre}}, \phi_{\text{end}}).$$

If all three positions fall in the same phoneme, the embedding is a central embedding, for example a_a_a or p_p_p. If a single boundary is crossed, the embedding is a border embedding, such as a_p_p, p_p_a, or a_a_p. Windows spanning two boundaries, such as a_p_a or a_p_s, are designated two-border embeddings and are excluded from the main analysis because such cases are rare (Ananeva et al., 9 Jul 2025).

The downstream probe is a 4-layer feed-forward neural network with three separate output heads, one each for start, centre, and end phoneme prediction. HubertSoft embeddings serve as inputs; the encoder remains frozen. The source describes the architecture schematically as a shared hidden representation feeding three softmax classifiers. Training uses cross-entropy loss summed across the three positions and the AdamW optimizer. Hyperparameters such as learning rate, dropout, and weight decay are tuned empirically (Ananeva et al., 9 Jul 2025).

Only selected phoneme types are used as targets. The study restricts the inventory to six stressed vowels and thirty-five consonants, including palatalized and non-palatalized consonants, in order to control the task and reduce severe class imbalance. This constraint is methodologically important because all reported accuracy values are defined over that selected inventory rather than over the full Russian phoneme set (Ananeva et al., 9 Jul 2025).

4. Evaluation of temporal structure

The study defines three complementary metrics to determine what kind of temporal information is recoverable from HubertSoft embeddings. The distinction among them separates mere phoneme presence from ordered boundary structure.

Ordered accuracy requires exact match of all three predicted positions:

$\hat{s} = s,\quad \hat{c} = c,\quad \hat{e} = e.$

This is the strictest measure, since it requires both correct phoneme identity and correct temporal order within the 20 ms window. In indicator notation the source gives

$$\text{Acc}_{\text{ordered}} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\big[(\hat{s}_i, \hat{c}_i, \hat{e}_i) = (s_i, c_i, e_i)\big].$$

Unordered accuracy ignores within-window order and asks only whether the multiset of predicted phonemes matches the multiset of true phonemes:

$\{\hat{s}, \hat{c}, \hat{e}\} = \{s, c, e\}.$

The source illustrates this with a true label a_p_p, for which predictions such as a_a_p, p_a_p, or p_p_a count as correct under unordered accuracy because they preserve one a and two p’s. Formally,

$$\text{Acc}_{\text{unordered}} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\big[\text{multiset}(\hat{s}_i, \hat{c}_i, \hat{e}_i) = \text{multiset}(s_i, c_i, e_i)\big].$$

Ordered flexible centre accuracy is intermediate. It requires the start and end labels to match exactly, but allows the predicted centre to equal either the true start or the true end:

$$\hat{s} = s,\quad \hat{e} = e,\quad \hat{c} \in \{s,e\}.$$

The source motivates this measure as a probe of boundary localization when the centre of the window is inherently ambiguous:

$$\text{Acc}_{\text{flex-centre}} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\big[\hat{s}_i = s_i,\ \hat{e}_i = e_i,\ \hat{c}_i \in \{s_i, e_i\}\big].$$

The three metrics therefore define a hierarchy. Ordered accuracy targets full temporal precision. Unordered accuracy measures phonetic content irrespective of ordering. Flexible-centre accuracy measures whether the model has correctly localized the boundary, even if the midpoint assignment shifts toward one neighboring phoneme or the other. This evaluative framework is the main methodological contribution of the study’s probing design (Ananeva et al., 9 Jul 2025).

5. Empirical findings on phoneme identity, order, and coarticulation

For the largest sound group examined, consisting of six stressed vowels and thirty-five consonants, the reported values are: ordered accuracy 0.5312, unordered accuracy 0.9069, ordered flexible centre accuracy 0.7645, start accuracy 0.8823, centre accuracy 0.6277, and end accuracy 0.8563 (Ananeva et al., 9 Jul 2025). These figures establish three points simultaneously. First, HubertSoft embeddings preserve the identity of phonemes present in the window to a very high degree. Second, they preserve temporal order substantially beyond what would follow from phoneme presence alone. Third, the start and end of the window are more recoverable than its midpoint.

The contrast between ordered and unordered metrics is particularly informative. The source reports a simulated hypothetical model that knows which phonemes are present with 0.9 probability per phoneme but assigns them in random order, yielding expected ordered accuracy of approximately 0.22. The observed ordered accuracy of 0.5312 is far above that level. This suggests that HubertSoft embeddings encode not only unordered membership but also the directional structure of phoneme transitions (Ananeva et al., 9 Jul 2025).

Per-position performance is asymmetric. Start accuracy is 0.8823 and end accuracy is 0.8563, whereas centre accuracy falls to 0.6277. The interpretation given in the source is that embeddings at segment boundaries are more informative than embeddings at the centre. This is consistent with the view that boundary regions concentrate acoustic evidence about change, while the middle of a 20 ms boundary-straddling window is often genuinely ambiguous because it contains cues from both adjacent phonemes (Ananeva et al., 9 Jul 2025).

Performance also varies by phoneme class. Metrics are highest for voiceless plosives and lowest for sonants and vowels, including stressed and unstressed vowel groups. The source explains this in phonetic terms: voiceless plosives have sharp and discrete boundaries, whereas vowels and sonorants exhibit smoother transitions and stronger coarticulatory influence (Ananeva et al., 9 Jul 2025).

The confusion matrices reported in the study are phonetically structured rather than random. Palatalized plosives such as p' and k' are often confused with p and k at the end position, where the representation captures closure information more strongly than release into the following vowel. The source notes that t' behaves somewhat differently, showing less pronounced confusion because it is acoustically closer to a fricative [ts']. For the affricate /ts/, similarity shifts with temporal position: it is more similar to /s/ in the start position and closer to /t/ at its beginning, reflecting the internal plosive-fricative composition of the affricate (Ananeva et al., 9 Jul 2025).

Vowel confusions likewise reflect known coarticulation. For stressed a0, start-position recognition reaches 0.98, while end-position recognition remains high at 0.91 on the diagonal but shows increased confusion toward i1 and i0, especially after palatalized consonants. The source relates this to fronting of /a/ under palatalization-induced coarticulation. The fact that this pattern appears specifically in the end-position matrix suggests that HubertSoft embeddings encode dynamic contextual modification rather than only invariant segment labels (Ananeva et al., 9 Jul 2025).

A short application example further illustrates this property. On a 350 ms recording containing the sequence i1 l a0, spread over 17 windows of 20 ms, the probe achieves an ordered accuracy of 0.88 and shows clear peaks at the two transition regions. The single mismatch corresponds to a one-frame shift relative to a manual boundary, which the source attributes to ambiguity in human segmentation between acoustically similar categories (Ananeva et al., 9 Jul 2025).

6. Significance, applications, and limitations

The cumulative result is that HubertSoft embeddings are not adequately characterized as static, context-free encodings of isolated phonemes. The study shows that they encode phoneme identity, phoneme order, and boundary structure, while also reflecting articulatory detail and coarticulatory effects. This suggests that the self-supervised objective has induced temporally structured internal representations that mirror aspects of phonetic organization in the acoustic signal (Ananeva et al., 9 Jul 2025).

Several implications follow directly from the reported findings. For phonological analysis, the triplet-label probing framework provides a way to study sub-segmental structure and transitions without manual feature engineering. For fine-grained transcription and segmentation, the frozen HubertSoft encoder plus a lightweight probe can recover boundary-sensitive information at 20 ms resolution. The source further suggests a plausible implication for low-resource settings: because HubertSoft is self-supervised, similar probing strategies could be applied in languages with limited annotation, using a small amount of aligned data to expose boundary structure already present in pretrained embeddings (Ananeva et al., 9 Jul 2025).

The scope of the evidence is nevertheless bounded. All experiments are conducted on Russian, and some observed effects, particularly those involving palatalization and vowel fronting, are language-specific. The corpus consists of professionally read speech from eight speakers rather than spontaneous speech, which may limit generalization. Manual boundaries are acknowledged to be approximate for transitions between acoustically similar sounds, so the notion of an exact ground-truth boundary is itself partly gradient. In addition, rare phoneme classes are excluded from the main experiments because of class imbalance, so the reported metrics are not estimates over the entire Russian inventory (Ananeva et al., 9 Jul 2025).

Within those limits, HubertSoft emerges as a speech representation model whose embeddings preserve more than local phonetic content. They retain recoverable information about where a short window lies relative to phoneme boundaries, which neighboring sounds are implicated, and in what order those sounds unfold. That combination of soft unit modeling and boundary-sensitive representational structure defines its distinctive role in current analysis of self-supervised speech encoders (Ananeva et al., 9 Jul 2025).