CUPE: Contextless Phoneme Encoder

• CUPE is a speech representation model that learns phoneme-discriminative embeddings from fixed 120ms acoustic segments, capturing essential short-term acoustic patterns.
• It employs overlapping window slicing with cosine weighting to fuse per-frame predictions, thereby enhancing phoneme alignment and reducing error rates compared to long-context models.
• Integrating CNNs and windowed Transformer blocks, CUPE supports both supervised and self-supervised training regimes and serves as an effective acoustic front end in multilingual forced alignment systems.

Searching arXiv for papers on CUPE and related forced alignment to ground the article. CUPE, the Contextless Universal Phoneme Encoder, is a speech representation model designed to learn phoneme-discriminative embeddings from short, fixed-width acoustic windows rather than from long contextual spans. Its central premise is that universal phoneme recognition can be approached through approximately phoneme-length segments—specifically about 120 milliseconds—so that the learned representation emphasizes fundamental acoustic patterns shared across languages instead of language-specific sequential regularities. In the original formulation, CUPE operates directly on raw 16 kHz waveforms and targets cross-lingual phoneme recognition, frame alignment, and zero-shot generalization to unseen languages; in subsequent work, it also serves as the acoustic front end of a multilingual forced aligner (Rehman et al., 21 Aug 2025).

1. Concept and problem setting

CUPE was introduced to address a specific limitation of large-context speech encoders such as wav2vec 2.0 and XLS-R: models that consume 300–2500 ms of audio context may improve downstream ASR, but they can also “smear” individual phoneme representations through contextual averaging. CUPE instead imposes a strict short-context regime and asks whether an embedding space can be learned that both discriminates phonemes across languages and generalizes zero-shot to unseen languages under that constraint (Rehman et al., 21 Aug 2025).

The intended use cases are tasks requiring “pure” frame- or phoneme-level embeddings, including precise phoneme alignment in TTS, allophone style encoding, and speaker-specific phonetic analysis. The model therefore treats contextlessness not as a denial of all temporal structure, but as a restriction to a narrow, approximately one-phoneme window. This design is meant to prioritize formant trajectories, voice onset time, and spectral shapes over longer-range language-specific sequencing.

A recurrent source of confusion is terminological rather than methodological: CUPE denotes a phoneme encoder for speech processing, whereas CUPED denotes “Controlled-experiment Using Pre-Experiment Data” in online experimentation. The two are unrelated despite near-identical orthography. This suggests that disambiguation is essential in bibliographic and systems contexts where both terms may appear.

2. Input representation and windowed inference

In its original form, CUPE processes raw 16 kHz waveforms without mel-spectrograms or filter banks. Let the waveform batch be $x \in \mathbb{R}^{B \times T}$. The signal is sliced into overlapping windows of length $W = 1920$ samples, corresponding to 120 ms, with stride $s = 1280$ samples, corresponding to 80 ms:

$$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$

with

$N = \lfloor (T-W)/s \rfloor + 1.$

This windowing strategy yields partially overlapping local views of the signal. Because adjacent windows cover common frames, CUPE fuses overlapping predictions with a cosine weighting scheme. If $y_k(b,t,c)$ denotes the logit for class $c$ at frame $t$ of window $k$, the fused logit is

$$\tilde y(b,t,c) = \frac{\sum_{k} \cos\!\bigl(\tfrac{\pi\,t}{F_w}-\tfrac{\pi}{2}\bigr)\,y_k(b,t,c)} {\sum_{k}\cos\!\bigl(\tfrac{\pi\,t}{F_w}-\tfrac{\pi}{2}\bigr) + \epsilon}, \quad t \in [0,F_w],$$

where $W = 1920$0 is the number of output frames per window. For a 120 ms window, CUPE produces 10 output frames at about 13.1 ms per frame (Rehman et al., 21 Aug 2025).

The local-window formulation is fundamental to the model’s semantics. It makes the representational unit close to a single phoneme rather than a phrase or word fragment. A plausible implication is that CUPE shifts the inductive bias from sequence modeling toward segmental acoustics, which is consistent with its intended role in alignment and phonetic analysis.

3. Neural architecture

The original CUPE architecture has approximately 30 million parameters and is organized as a hierarchical front end followed by within-window temporal modeling. The first stage is a 1D-CNN extractor with four convolutional layers:

• Conv1D-1: kernel = 15, stride = 7, pads = 7, output $W = 1920$1
• Conv1D-2: kernel = 11, stride = 5, pads = 5, output $W = 1920$2
• Conv1D-3: kernel = 7, stride = 3, pads = 3, output $W = 1920$3
• Conv1D-4: kernel = 5, stride = 2, pads = 2, output $W = 1920$4

After each convolution, the network applies BatchNorm, GELU, and dropout$W = 1920$5. This is followed by a frequency-attention block in a “squeeze-and-excitation” style that recalibrates the 2048 channels while preserving channel dependencies (Rehman et al., 21 Aug 2025).

The next stage is a two-stream temporal and spectral CNN. The temporal stream uses two Conv1D layers with group convolution $W = 1920$6 and kernels 7 then 3, padded to preserve the 10-frame length. The spectral stream uses two $W = 1920$7 Conv1D layers. The streams are concatenated and fused through a $W = 1920$8 convolution, BatchNorm, and GELU, producing an output of shape $W = 1920$9.

A windowwise Transformer then models short-range structure within each 120 ms segment. It uses 4 layers, 8 heads, model dimension 512, dropout 0.25, and pre-norm configuration, with about 13 million parameters. It operates on the 10 frame embeddings produced per window. The output is consumed by one of two heads: an FT-Classifier for supervised learning, implemented as a two-layer MLP $s = 1280$0 with $s = 1280$1 phoneme classes, or a PT-Projection for self-supervised learning, implemented as a two-layer MLP $s = 1280$2.

A common misconception is that a “contextless” encoder must be frame-independent. CUPE does not satisfy that stronger condition. It is contextless at the scale of broad utterance context, yet it still performs structured modeling inside the fixed 120 ms window via CNN and Transformer components.

4. Training objectives and optimization

CUPE supports both supervised and self-supervised training regimes. In supervised training, the model predicts frame-level logits and optimizes a CTC objective augmented with a silence-awareness term. The silence term is

$s = 1280$3

where $s = 1280$4 is the blank-token probability, $s = 1280$5 is the ground-truth silence mask, and the associated coefficient is $s = 1280$6 (Rehman et al., 21 Aug 2025).

In self-supervised training, 40% of windows are masked using energy and boundary heuristics. Targets are provided by a 256-entry vector quantizer with EMA decay 0.99. The objective combines an $s = 1280$7 reconstruction term, a contrastive term, a codebook-diversity term, and a similarity-regularization term, with typical weights $s = 1280$8.

Optimization uses AdamW. The reported weight decay is 0.01 for supervised training and 0.05 for self-supervised training. The learning-rate schedule is OneCycleLR with maximum learning rates of $s = 1280$9 for the encoder, $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$0 for the quantizer, and $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$1 for the head. Additional settings include batch size $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$2 words, gradient clipping at norm 1.0, and mixed-precision BF16. Supervised training on MSWC-train runs for 20 epochs in about 7 hours on two A6000 GPUs, while self-supervised pretraining on FLEUR, covering 102 languages at 5 hours per language, takes about 3 days on four GPUs.

These design choices indicate that CUPE was engineered not only as a conceptual alternative to long-context encoders, but also as a practically trainable compact model.

5. Cross-lingual evaluation and acoustic behavior

CUPE was evaluated on MSWC, FLEUR, and the UCLA Phonetic Corpus (UPC). The reported metrics are Phoneme Error Rate (PER), Ground-truth Probability Macro (GPm), Ground-truth Probability Weighted (GPw), and frame-alignment F1-score. In supervised training without pretraining, CUPE at 120 ms outperforms XLS-R at the same context length on both MSWC-eval and zero-shot UPC-eval; with self-supervised pretraining and fine-tuning, CUPE-PT also improves on XLS-R under the same 120 ms constraint (Rehman et al., 21 Aug 2025).

Model	MSWC-eval PER	UPC-eval PER
XLS-R (120 ms)	52.6	66.3
CUPE (120 ms)	45.9	56.9
XLS-R	52.2	63.6
CUPE-PT	45.6	56.2

The same experiments report additional gains beyond PER. In the supervised 120 ms comparison, CUPE improves GPm from 34.0 to 40.0, GPw from 52.1 to 57.5, and F1 from 59.9 to 64.5 relative to XLS-R. In the pretrained comparison, CUPE-PT reaches GPm 41.2, GPw 58.1, and F1 64.0 on MSWC-eval. On unseen UPC languages, zero-shot PER for CUPE-PT is 56.2% versus 63.6% for XLS-R. The paper further states that CUPE-PT matches or outperforms XLS-R despite an order of magnitude fewer parameters, specifically 30 million versus 300 million.

Window-size ablations compare 120 ms, 160 ms, and 360 ms contexts. PER on MSWC-eval is 45.9% at 120 ms, 47.8% at 160 ms, and 44.8% at 360 ms, while 120 ms achieves the best GPm at 40.0. The reported interpretation is that longer windows slightly reduce PER on common long vowels but harm rare phoneme recall. Appendix analyses also show that most confusions occur between phonetically similar pairs, such as /p/ versus /b/ and /i/ versus /ɪ/, and that probability peaks align sharply to ground-truth phoneme boundaries at about 13 ms temporal resolution.

Taken together, these results support the claim that strict local context does not preclude cross-lingual generalization. They also suggest that CUPE’s principal advantage is not merely compactness, but a favorable trade-off between local acoustic specificity and multilingual transfer.

6. CUPE within multilingual forced alignment

CUPE is also the acoustic encoder in Bournemouth Forced Aligner (BFA), where it supplies per-frame phoneme probability distributions to a CTC-based decoder. In that system, CUPE is described as entirely “contextless” in the decoder sense: it produces one phoneme prediction per input frame without requiring left or right context bands in the decoding step. BFA combines three components: CUPE acoustic encoding, multilingual text-to-phoneme conversion, and a CTC dynamic-programming decoder (Rehman et al., 27 Sep 2025).

Within BFA, the encoder typically consumes short-term spectral features such as 80-dimensional log-Mel spectrogram bins computed over 25 ms windows with 10 ms hop, rather than raw waveform windows. The shared trunk maps $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$3 to $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$4, with $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$5 in the original CUPE proposal referenced there. Two classification heads are used: a fine head producing a $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$6-way phoneme distribution and a coarse head producing a $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$7-way phoneme-group distribution, where the extra symbol is the blank or silence token. Non-linearities are ReLU, with dropout around 0.1–0.2 and weight decay $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$8.

The BFA training objective applies CTC separately to both heads and sums the two losses. For a target phoneme sequence $$w_{b,i}(t) = x_b\bigl(t + i\,s\bigr), \quad t = 0,\dots,W-1, \quad i = 0,\dots,N-1,$$9, the fine-head loss is

$N = \lfloor (T-W)/s \rfloor + 1.$0

with an analogous term for the coarse head. Explicit blank modeling is central to the alignment strategy: the decoder path is constructed as

$N = \lfloor (T-W)/s \rfloor + 1.$1

which permits arbitrary durations of silence between phonemes.

End-to-end evaluation of BFA provides an indirect characterization of CUPE in this role. On TIMIT, $N = \lfloor (T-W)/s \rfloor + 1.$2 achieves 71.4% recall at 20 ms versus 71.9% for Montreal Forced Aligner (MFA); on Buckeye, it achieves 63.7% versus 58.1% for MFA. At 60 ms tolerance, recall is about 88% on TIMIT and about 76% on Buckeye. Mean absolute boundary error is about 14 ms, with median about 10 ms, comparable to MFA’s 13 ms mean. BFA runs 45–240 times faster than MFA, with real-time factor 0.05–0.1. The paper also reports that approximately 35% of phonemes in BFA exhibit non-zero gaps, with median gap 40 ms on TIMIT versus 60 ms in MFA, reflecting BFA’s explicit modeling of inter-phoneme gaps.

These results indicate that CUPE’s short-context phoneme encoding is compatible with real-time multilingual alignment, particularly when silence and gap structure are modeled explicitly rather than treated as residual timing noise.

7. Limitations, applications, and disambiguation

The original CUPE study identifies several limitations. Fixed 120 ms windows struggle with very short stops below 60 ms and very long vowels above 200 ms. Rare phoneme classes, including infrequent vowels and affricate variants, continue to exhibit higher PER. The mapping to 65 phoneme classes is described as a trade-off between cross-linguistic generalization and phonemic granularity, and a model capacity of 30 million parameters may limit the capture of very fine phonetic distinctions across all world languages (Rehman et al., 21 Aug 2025).

The same source lists several potential extensions: dynamic window sizing, hybrid input that combines raw waveform CNNs with mel-filter-bank streams, articulatory-attribute conditioning akin to Allophant, and expansion to 50–100 million parameters while preserving contextless purity. Proposed applications include high-precision forced alignment for TTS and corpus annotation, allophone style encoding for expressive TTS and voice conversion, zero-shot phoneme verification for ASR post-processing, and cross-linguistic phonetic research and speech pathology diagnostics.

In the BFA setting, further limitations are also explicit. Predicting both onset and offset boundaries doubles the number of boundary events and reduces precision under conventional onset-only evaluation; tonal or non-Indo-European evaluation remains to be established; and the absence of explicit context modeling may be suboptimal for highly coarticulated events (Rehman et al., 27 Sep 2025).

The term should also be distinguished from CUPED, the variance-reduction method for online A/B testing. CUPED concerns unbiased treatment-effect estimation with pre-experiment covariates in randomized experiments and has no relation to multilingual phoneme encoding or forced alignment (Zhang et al., 17 Jun 2026). This orthographic collision has practical relevance for literature search, indexing, and citation hygiene, especially in technical corpora where acronym expansion may be omitted.