Spectrogram-Guided Phonetic Alignment (SGPA)

• The paper introduces SGPA, a method that transforms dense audio tokens into semantically grounded, word-aligned segments for Shapley value attribution.
• It combines CTC forced alignment with spectrogram-based boundary refinement by minimizing weighted RMS energy and spectral flux to locate stable acoustic pauses.
• Empirical results show SGPA achieves a 43× reduction in model calls and a 28× decrease in runtime compared to native audio tokenization.

Spectrogram-Guided Phonetic Alignment (SGPA) is a four-stage pipeline for end-to-end audio LLM explainability that replaces model-native audio units with word-aligned, acoustically stable audio segments for Shapley value attribution. In the formulation introduced in "SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal LLMs" (Pozorski et al., 25 Feb 2026), the method combines Connectionist Temporal Classification (CTC) forced alignment with spectrogram-based boundary refinement, then aggregates refined character spans into words. SGPA is presented not as a generic forced aligner, but as a feasibility-enabling layer that redefines the players of the Shapley game from dense encoder frames to semantically grounded speech units, thereby addressing the combinatorial, interpretive, and masking-pathology problems of native audio tokenization.

1. Problem formulation and conceptual motivation

The SGPA formulation is motivated by the difficulty of applying Shapley value methods to end-to-end multimodal LLMs that ingest raw audio directly, rather than through a conventional ASR transcript intermediary (Pozorski et al., 25 Feb 2026). In text, token-level attribution is already expensive but remains aligned with discrete semantic units. In audio, the native units are dense temporal frames from an encoder. The paper identifies three obstacles.

The first is dimensionality explosion. Exact Shapley computation over $n$ features requires evaluating all $2^n$ feature coalitions. The paper contrasts a text-like game of $2^{10}$ with a native audio-frame game of $2^{150}$, which it describes as an increase of roughly $10^{42}$ in coalition space. For a representative 50-token prompt, it gives the estimate

$$2^{50} \times 10^{-3}\,\text{s} \approx 3.6 \times 10^{10}\,\text{s} \approx 35 \text{ years},$$

assuming 1 ms per model call (Pozorski et al., 25 Feb 2026).

The second is semantic dilution. A single $\approx 20$ ms audio frame does not correspond to a meaningful standalone speech unit. The third is boundary artifacts: frame boundaries are not chosen to respect phonetic structure, so perturbation-based masking may cut through coarticulated transitions and produce unnatural, out-of-distribution audio. SGPA addresses these issues by redefining the Shapley players as word-aligned waveform spans rather than native audio frames.

In this usage, “spectrogram-guided phonetic alignment” means that temporal boundaries are first estimated from a phonetic or character alignment process driven by a CTC model, and are then adjusted using local spectral evidence so that cuts land in low-energy, low-change acoustic regions rather than within transitions (Pozorski et al., 25 Feb 2026). This suggests that SGPA is best understood as an acoustically constrained coarse-graining of the attribution game rather than as a conventional alignment system whose primary goal is transcript timing.

2. Four-stage pipeline and boundary refinement

SGPA takes a raw audio waveform and its transcript, and outputs word-aligned audio segments for attribution (Pozorski et al., 25 Feb 2026). The stages are transcript decomposition, CTC forced alignment, spectral boundary refinement, and word-level aggregation.

In the first stage, the transcript is decomposed into words and characters, and a character-to-word membership map is recorded. This is a bookkeeping step because the alignment model operates at character level whereas the final Shapley game is defined at word level.

In the second stage, the waveform is passed through Wav2Vec2-XLSR-53. The model outputs an emission matrix

$\mathbf{E} \in \mathbb{R}^{T \times V},$

where $T$ is the number of time frames and $V$ is the vocabulary size. Viterbi decoding computes the most likely character path,

$2^n$0

from which approximate start and end frame indices for each character are extracted (Pozorski et al., 25 Feb 2026). These timings are explicitly described as coarse estimates rather than perturbation-ready boundaries.

The third stage is the distinctive SGPA component. For each estimated boundary $2^n$1, the method defines a local search window

$2^n$2

where $2^n$3 is a tunable half-width. Within this window it computes RMS short-time energy $2^n$4 and spectral flux $2^n$5, the latter defined as frame-to-frame $2^n$6 change in the magnitude spectrum. The refined boundary is then chosen as

$2^n$7

with empirically chosen weights

$2^n$8

This objective favors low-energy, low-change regions such as pauses, closures, or local minima between words. The implementation uses librosa for these signal-level features (Pozorski et al., 25 Feb 2026).

In the fourth stage, refined character-level boundaries are merged into word-level spans using the membership map from stage 1. The result is a sequence of word-aligned segments. The paper characterizes the scale change as from $2^n$9 native tokens to $2^{10}$0 word segments, and reports practical averages of roughly 50 native tokens versus 7 SGPA segments (Pozorski et al., 25 Feb 2026).

For coalition evaluation, omitted players are masked by replacing their corresponding waveform region with silence, specifically zero amplitude, while preserving the absolute timing and ordering of the remaining audio (Pozorski et al., 25 Feb 2026). This preserves temporal layout while muting selected semantic units.

3. SGPA as a redefinition of the Shapley game

A central claim of the paper is that SGPA is not merely a preprocessing convenience. It changes the feature space and therefore changes the cooperative game whose Shapley values are being estimated (Pozorski et al., 25 Feb 2026). Without SGPA, the players are native audio units, tokens, or frames. With SGPA, the players are word-aligned waveform segments.

The estimator used is the Neyman-allocated complementary-contributions estimator with total budget

$2^{10}$1

where $2^{10}$2 is the number of players. The initialization phase uses

$2^{10}$3

samples per stratum, after which the remaining budget is allocated according to estimated standard deviations under the Neyman allocation principle (Pozorski et al., 25 Feb 2026). Because the budget scales quadratically in $2^{10}$4, the reduction from dozens of native audio tokens to single-digit word segments is computationally decisive.

The paper analyzes attribution concentration using normalized absolute Shapley values,

$2^{10}$5

together with Top-20% mass and Gini coefficient: $2^{10}$6 Normalized entropy is also reported as a third concentration metric (Pozorski et al., 25 Feb 2026).

This framing is methodologically important. SGPA deliberately trades fidelity to native model segmentation for interpretability and feasibility. The paper explicitly states that attributions under SGPA and native tokenization are not directly comparable as if they were estimates of the same game (Pozorski et al., 25 Feb 2026).

4. Experimental configuration and empirical findings

The experimental study evaluates SGPA on LFM2-Audio-1.5B, an open-source 1.5B-parameter end-to-end audio foundation model chosen because it processes audio directly, has publicly available weights, and fits in the 16 GB VRAM of an Nvidia RTX 4080 (Pozorski et al., 25 Feb 2026). The dataset is a single-sentence English subset of VoiceBench constructed by removing exact-match duplicates across partitions, retaining only single-sentence prompts using NLTK sentence segmentation, applying stratified sampling over source sub-datasets, reducing the pool by 90%, and obtaining 100 samples. The resulting text statistics are mean 7.19 tokens and mean 37.87 characters. Audio is synthesized with Google Chirp 3 HD TTS in two speech-to-speech modes: SM2S, a male voice, and SF2S, a female voice. Mean durations are 2.69 s for male and 2.48 s for female, with total combined corpus length 1035.5 s (Pozorski et al., 25 Feb 2026).

The controlled comparison is between native audio tokenization and SGPA word-aligned segments, with all other attribution settings held fixed. The reported mean model calls and runtimes are as follows.

Condition	Mean model calls	Mean time
SGPA, SM2S	59.42	67.53 s
SGPA, SF2S	59.32	66.08 s
No SGPA, SM2S	2365.76	1696.49 s
No SGPA, SF2S	2552.14	1819.88 s

The headline reduction is approximately

$2^{10}$7

which the paper reports as a 43$2^{10}$8 reduction in model evaluations. The wall-clock reduction is about 28$2^{10}$9, for example

$2^{150}$0

(Pozorski et al., 25 Feb 2026).

The budget distribution also changes. With SGPA, 37% of budget goes to Phase 1 and 63% to Phase 2; without SGPA, at least 98% goes to Phase 2. The paper interprets this as evidence that native tokenization leaves too little budget for initial variance estimation (Pozorski et al., 25 Feb 2026).

The concentration analysis uses paired $2^{150}$1-tests with Bonferroni correction at $2^{150}$2. Significant results are reported for SM2S Gini, entropy, and Top-20% mass; for SF2S, entropy and Top-20% mass are significant, while SF2S Gini is marked non-significant in the table despite $2^{150}$3. The strongest effect is normalized entropy in SM2S with Cohen’s $2^{150}$4 (Pozorski et al., 25 Feb 2026). The paper interprets the negative sign as indicating that SGPA increases entropy relative to native tokenization, meaning attribution becomes more evenly spread across fewer, larger word-level players.

At the same time, the paper reports preservation of the macro-level cumulative pattern. Both segmentation regimes exhibit an “early mass” cumulative SV profile, with disproportionate attribution accumulation in early positions followed by slower growth. SGPA changes local concentration but preserves the global cumulative attribution profile (Pozorski et al., 25 Feb 2026). This suggests that the coarse-graining alters local partitioning without wholly disrupting broader explanatory structure.

5. Relation to adjacent alignment literature

SGPA sits near several strands of alignment research, but the relationship is not one of equivalence. LCS-CTC introduces a two-stage framework for phoneme-level speech recognition that predicts a frame-phoneme cost matrix, extracts high-confidence local alignment zones with a modified LCS procedure, and constrains CTC training accordingly. It is described as enabling both robust recognition and text-free forced alignment, but it does not use explicit spectrogram guidance in the narrow sense; its guidance comes from frame-level acoustic embeddings derived from pretrained encoders such as Wav2Vec2.0-L, HuBERT-L, and WavLM-L (Ye et al., 5 Aug 2025). Relative to SGPA, it is therefore best characterized as an acoustically guided phonetic alignment method rather than a direct spectrogram-based counterpart.

FALCON, introduced in "Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming" (Rousso et al., 24 Jun 2026), is closer to SGPA in its emphasis on boundary localization. It uses a waveform encoder to produce latent frames, a context encoder to produce framewise phoneme posterior distributions, and a differentiable soft dynamic programming decoder that combines local transition evidence with segment-level phoneme consistency. The model is not based on a handcrafted spectrogram frontend, but the learned latent sequence is explicitly used as boundary-sensitive acoustic evidence. This suggests a strong methodological convergence with SGPA at the level of boundary-aware, monotonic, acoustically guided segmentation.

MAPS, the Mason-Alberta Phonetic Segmenter, is a neural forced alignment system based on MFCCs + deltas + delta-deltas, dynamic-programming decoding, and an interpolation procedure that places boundaries more finely than the common 10 ms frame grid (Kelley et al., 2023). Its direct relevance to SGPA lies in two points: boundary precision can be improved beyond the frame lattice, and one-hot phone targets may misrepresent overlapping acoustic evidence near segment transitions. MAPS is not an SGPA system, but its interpolation technique and its critique of discrete frame labeling are directly pertinent to spectrogram-guided boundary placement.

Taken together, these neighboring systems indicate that the broader alignment literature is converging on three recurrent themes: boundary-aware acoustic evidence, monotonic transcript-constrained decoding, and dissatisfaction with framewise discrete labeling as a faithful account of phonetic transitions. SGPA’s distinctive contribution is to repurpose these ideas for attribution feasibility in audio MLLMs rather than for recognition or standalone forced alignment.

6. Scope, limitations, and documentary status

The SGPA paper states four explicit limitations. First, the spectral refinement parameters are empirically tuned on English, with $2^{150}$5, $2^{150}$6, and tunable $2^{150}$7; robustness to tonal languages or more diverse phonological systems is not established. Second, the diagnostics are performed on a single model, LFM2-Audio-1.5B, due to hardware constraints. Third, SGPA and native tokenization define different Shapley games, so their attributions are not estimates of the same object. Fourth, SGPA requires a transcript, which in audio-only settings would have to come from an upstream ASR system (Pozorski et al., 25 Feb 2026).

Additional limitations are implied in the paper’s discussion: dependence on forced alignment quality from the CTC model, possible ASR or transcript errors, phonetic ambiguity and heavy coarticulation, speaker variation, and sensitivity to silence or noise conditions because the refinement objective explicitly prefers low-energy stable regions (Pozorski et al., 25 Feb 2026). The paper also does not establish unchanged transfer to non-speech audio, multilingual speech, or tasks beyond the speech-to-speech prompting setup used in the experiments.

A separate documentary clarification is necessary because one cited arXiv entry has been misdescribed in some contexts. The supplied document for (Lou et al., 2024) is described as an IEEE conference template with placeholder material rather than a substantive TTS or alignment paper. It does not define an alignment problem, does not describe an aligner or staged training, does not mention Mel-spectrograms, MFCCs, or latent features, and reports no datasets, model configurations, or evaluation metrics relevant to SGPA (Lou et al., 2024). It therefore cannot be used as evidence for Spectrogram-Guided Phonetic Alignment or for an aligner-guided duration-training method.

Within the literature represented here, SGPA refers specifically to the explainability-oriented segmentation procedure of (Pozorski et al., 25 Feb 2026): coarse character timing from CTC forced alignment, local boundary refinement by minimizing a weighted combination of RMS short-time energy and spectral flux, and aggregation into word-level Shapley players. Its significance lies in making attribution computationally feasible while preserving a human-interpretable alignment between spoken words and explanatory units.