Acoustic Similarity Groups for Speech Decoding

• Acoustic Similarity Groups (ASGs) are overlapping sets of acoustic or semantic tokens defined in a learned embedding space, enabling coarse-grained token representation.
• They are constructed using cosine similarity thresholds, allowing tokens to belong to multiple groups and preserving soft neighborhood continuity.
• ASGs enable Principled Coarse-Graining in speculative decoding, enhancing throughput and decoding speed while maintaining acceptable error metrics and intelligibility.

Searching arXiv for the specified papers and closely related ASG-relevant work. Acoustic Similarity Groups (ASGs) are overlapping sets of acoustically or semantically similar items defined in a learned embedding space and used as a coarser random variable than exact symbol identity. In speech token generation, ASGs are the core construct that enables Principled Coarse-Graining (PCG) for speculative decoding in speech LLMs that generate discrete acoustic tokens: they formalize the intuition that many codebook tokens are acoustically or semantically interchangeable and therefore should be verified at a coarser resolution than exact token identity (Yanuka et al., 5 Nov 2025). In related settings, analogous grouping procedures have also been synthesized for acoustic word embeddings, acoustic language similarity, and joint acoustic–semantic music embeddings, although the term itself is not uniformly used across those literatures (Abdullah et al., 2021, Wu et al., 2021, Karamanolakis et al., 2016).

1. Conceptual basis and role in coarse-grained verification

In autoregressive speech generation, different discrete acoustic tokens often realize perceptually similar sounds, such as neighboring codewords in a VQ/FSQ codebook. Exact token matching in standard speculative decoding therefore rejects many draft tokens that would be acceptable substitutes at the perceptual level, suppressing acceptance rates and throughput. ASGs address this by aggregating tokens that the target model itself deems similar according to its learned embedding geometry, inducing a coarse semantic/acoustic variable on which the speculative accept/reject decision can be made (Yanuka et al., 5 Nov 2025).

Conceptually, an ASG is a set of tokens that are acoustically or semantically interchangeable according to the target model’s token embedding space. A central feature is that tokens may belong to multiple groups, preserving “soft” neighborhoods without arbitrary boundaries. This differs from exact matching, which only accepts a draft token if the target’s top token equals it, and from prior speech relaxations such as SSD, which adds a constant bias to the acceptance probability irrespective of acoustic similarity, or top-$k$ restrictions, which shrink the support and reduce diversity. PCG instead defines a proper coarse random variable, the ASG index, and accepts or rejects via rejection sampling on its induced distribution (Yanuka et al., 5 Nov 2025).

This framing is significant because it separates two issues that are often conflated in relaxed speculative decoding: whether two tokens are interchangeable for perceptual purposes, and whether the accept/reject mechanism remains probabilistically well defined. ASGs are intended to handle the former, while overlap-aware coarse distributions and rejection sampling handle the latter.

2. Construction from the target model’s embedding space

Let the target model be $q$, with vocabulary $V$ and learned token embeddings $\mathrm{Emb}_q(t)$ for $t \in V$. ASGs are constructed directly from this target embedding space using cosine similarity,

$$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$

For each token $t$, its neighborhood at threshold $\theta$ is defined as

$$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$

Lower $\theta$ yields larger, more permissive groups. In the reported experiments, $q$0 gave the best quality–speed trade-off, and the set of distinct groups is written as $q$1 (Yanuka et al., 5 Nov 2025).

Overlap is allowed: a token can belong to several groups, which better tracks the continuity of the acoustic space. This overlap is not an incidental implementation choice; it is part of the definition. A plausible implication is that ASGs are intended to preserve local continuity in the target model’s representation geometry rather than enforce a hard partition.

The same section of the PCG work also gives practical construction guidance. Token embeddings should be taken from the target model’s token embeddings $q$2 because they reflect the target model’s internal acoustic or semantic structure and gave the best empirical results. Cosine similarity is the metric used in the experiments, and a comparison against Mel-spectrogram similarity showed cosine in the target space yields better WER at matched speedups. For storage and computation, one may sparsify with top-$q$3 nearest neighbors when constructing $q$4 or compress the indices; this is described as an engineering convenience and does not alter the method’s definition (Yanuka et al., 5 Nov 2025).

3. Overlap-aware membership weights and induced coarse distributions

Because ASGs overlap, naïve probability aggregation is invalid. Let

$q$5

be the set of groups containing token $q$6, and let

$q$7

Nonnegative weights $q$8 are then defined for $q$9 so that each token’s probability mass is split across the groups that contain it:

$V$0

The default choice is equal-split weighting,

$V$1

This ensures every token’s full probability mass is preserved when aggregated over groups (Yanuka et al., 5 Nov 2025).

At a decoding step, let $V$2 and $V$3 denote the draft and target token distributions respectively. The group-level coarse distributions are

$V$4

where $V$5. Because $V$6 for each token $V$7, both $V$8 and $V$9 are proper distributions:

$\mathrm{Emb}_q(t)$0

Under equal-split weights, these reduce to

$\mathrm{Emb}_q(t)$1

The importance of this construction is that ASGs become not merely neighborhoods in embedding space but a legitimate coarse random variable with a well-defined overlap-aware probability distribution (Yanuka et al., 5 Nov 2025).

4. Acceptance, rejection, and exactness in Principled Coarse-Graining

PCG modifies speculative decoding by running acceptance on the group label rather than directly on the token variable. The proposal coupling has two steps: first sample a draft token $\mathrm{Emb}_q(t)$2; then sample a group $\mathrm{Emb}_q(t)$3 that contains $\mathrm{Emb}_q(t)$4 with probability $\mathrm{Emb}_q(t)$5 over $\mathrm{Emb}_q(t)$6. This ensures that $\mathrm{Emb}_q(t)$7 is marginally distributed as $\mathrm{Emb}_q(t)$8, since

$\mathrm{Emb}_q(t)$9

Group-level acceptance then uses

$t \in V$0

If accepted, the group-level draw is exact under $t \in V$1 (Yanuka et al., 5 Nov 2025).

If the proposal is rejected, the group is sampled from the residual distribution

$t \in V$2

where $t \in V$3 denotes $t \in V$4. The paper describes an efficient thinning realization: sample $t \in V$5 and then $t \in V$6 over $t \in V$7, so that $t \in V$8; accept $t \in V$9 with probability

$$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$0

otherwise resample. Upon accepting a group through the residual, realize a token within $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$1 via the target-conditional split

$$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$2

The expected number of thinning trials is $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$3, and was $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$4 in all experiments (Yanuka et al., 5 Nov 2025).

The exactness guarantee is at the group level. Informally, with the coupling $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$5, the acceptance rule $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$6, and reject-branch sampling from $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$7, the emitted group is exactly distributed as $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$8 at every step. The proof sketch is standard rejection-sampling correctness on the group variable: the accepted mass for group $$\cos(u,v)=\frac{u\cdot v}{\lVert u\rVert \lVert v\rVert}.$$9 is $t$0, the reject branch contributes $t$1, and the sum is $t$2 (Yanuka et al., 5 Nov 2025).

A crucial practical distinction follows. To maximize speed, when a group is accepted via $t$3, PCG outputs the draft token $t$4 as the group representative, preserving the KV cache. This keeps the emitted group distribution exact while sacrificing token-level exactness within-group. When a group is obtained via the residual, the token is sampled by the target-conditional distribution above, which is token-exact within that group draw (Yanuka et al., 5 Nov 2025).

A concrete example in the paper uses tokens $t$5 with a group $t$6 and overlap on token $t$7 with another group $t$8. Under equal-split weights, $t$9 and $\theta$0, so

$\theta$1

and similarly for $\theta$2. If the draft samples token $\theta$3 but $\theta$4 places its mass mainly on token $\theta$5, exact matching would reject, whereas PCG accepts or rejects based on $\theta$6 and can emit token $\theta$7 as an acoustically interchangeable representative (Yanuka et al., 5 Nov 2025).

5. Computation, empirical behavior, and tuning

The algorithmic overhead is modest relative to the softmax. For one speculative step, computing $\theta$8 and $\theta$9 for the sampled group $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$0 costs $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$1 operations once $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$2 and $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$3 are available. The paper notes that the $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$4 softmax computation still dominates, and in the reported setup the typical group size is much smaller than $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$5, around $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$6 versus $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$7. ASGs and weights are precomputed once from $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$8 and cached. The memory footprint is $$G(t):=\{t' \in V : \cos(\mathrm{Emb}_q(t), \mathrm{Emb}_q(t'))>\theta\}.$$9 indices; for $\theta$0 and $\theta$1, this is about $\theta$2M indices, roughly $\theta$3 MB with 32-bit integers or about $\theta$4 MB with 16-bit indices (Yanuka et al., 5 Nov 2025).

On LibriTTS, the target model is LLaSA-8B with a $\theta$5-entry FSQ/X-codec2 tokenizer, the draft model is a small 3-layer subset trained on Libri-heavy, and inference uses temperature $\theta$6 and lookahead of $\theta$7 tokens. Relative to standard speculative decoding and SSD, PCG increases acceptance and throughput while better preserving intelligibility and quality. On test-clean, standard SD gives $\theta$8 speedup with WER $\theta$9, CER $q$00, Sim-O $q$01, and NMOS $q$02; SSD gives $q$03 speedup with WER $q$04, CER $q$05, Sim-O $q$06, and NMOS $q$07; PCG gives $q$08 speedup with WER $q$09, CER $q$10, Sim-O $q$11, and NMOS $q$12. The improvements over SSD are reported as statistically significant with $q$13 (Yanuka et al., 5 Nov 2025).

The ablations clarify what ASG construction is buying. Grouping with cosine similarity in $q$14 outperforms grouping by Mel-spectrogram similarity at similar speed, with WER $q$15 versus $q$16, CER $q$17 versus $q$18, and equal Sim-O at $q$19. A stress test replaces each target-generated token by a uniformly sampled alternative from its ASG, ignoring $q$20; even when $q$21 of tokens are swapped and only $q$22 remain unchanged, the degradation is small, with $q$23WER $q$24 and $q$25Sim-O $q$26, supporting the substitutability assumption within ASGs. In the reported setup, a lookahead of $q$27 tokens maximized speedup for SD+PCG (Yanuka et al., 5 Nov 2025).

Practical guidance follows directly from these results. The default recommendation is to use the target model’s embeddings, cosine similarity, overlap, and equal-split weights. Lowering $q$28 enlarges groups and boosts acceptance but risks quality loss; increasing $q$29 does the opposite. Monitoring WER, CER, and NMOS as $q$30 varies is therefore a safeguard. Because the exactness guarantee holds at the group level, the reject branch must be implemented exactly as specified; otherwise the guarantee is void (Yanuka et al., 5 Nov 2025).

6. Related formulations, cautions, and limitations

The broader literature shows that the usefulness of ASGs depends critically on the geometry of the embedding space from which they are derived. In acoustic word embeddings, the distance in the embedding space in the best cases only moderately correlates with phonological distance, with mean Kendall’s $q$31 mostly $q$32, and improving word discrimination does not necessarily yield models that better reflect phonological similarity. In particular, strongly contrastive Siamese models can achieve high mAP while showing weak phonological correspondence. This supports a cautionary principle for ASGs: raw embedding distance should not be used uncritically as a proxy for phonological neighborhoods (Abdullah et al., 2021).

At a different scale, cross-lingual speech transfer work proposes an acoustic language similarity framework that produces language-level embeddings and pairwise distances directly from speech. That paper does not explicitly use the term “Acoustic Similarity Groups,” but its method can be used to form clusters or neighborhoods of acoustically similar languages for ASR and TTS transfer. Across ASR and TTS, acoustic similarity correlates strongly with transfer performance, often outperforming typological or geographic measures, and smaller but acoustically similar sources can outperform much larger but dissimilar ones (Wu et al., 2021). This suggests that ASG-like grouping is not confined to token-level speculative decoding; it can also operate at the level of languages.

In music information retrieval, joint acoustic–semantic embeddings induce sets of clips that are close to each other in a shared space defined over audio-word codebooks and tag embeddings. There, clip embeddings can be acoustic, semantic, or fused, and cosine similarity in the final space supports clustering or graph-based communities interpretable as acoustic similarity groups of clips. The best reported constraint accuracy in that setting is obtained by FUSION with SVD$q$33, reaching $q$34 on MagnaTagATune triplet constraints (Karamanolakis et al., 2016). A plausible implication is that ASGs are most stable when acoustic evidence is supplemented by semantically informative structure rather than relying on a single similarity source.

For speech-token PCG specifically, the limitations are explicit. If tokens are extremely fine-grained with little substitutability, ASGs must be very tight, limiting gains. Domain shifts that misalign $q$35 with current acoustics can degrade grouping quality. The exactness guarantee depends on overlap-aware mass splitting, $q$36, and on a correct residual sampler; deviating from these conditions voids the guarantee. Finally, keeping the draft token as the representative on acceptance preserves group-level exactness but not token-level exactness (Yanuka et al., 5 Nov 2025).

ASGs are therefore best understood not as arbitrary clusters but as a formally defined coarse variable whose validity depends on how similarity is represented, how overlap is handled, and what level of exactness is required. In the PCG formulation, they are an overlap-aware acceptance mechanism for speculative decoding in speech; in related work, they are a more general pattern for organizing acoustically similar words, languages, or clips when the embedding geometry is sufficiently aligned with the task-relevant notion of similarity.