Contextless Universal Phoneme Encoder (CUPE)

• Contextless Universal Phoneme Encoder (CUPE) is a language-agnostic model that processes independent 120ms waveform windows to produce frame-level phoneme embeddings free from longer-span context.
• It employs a hierarchical 1D-CNN stack followed by a lightweight transformer to extract 10 precise 512-dimensional embeddings per window, ensuring fine local acoustic-phonetic representation.
• CUPE demonstrates competitive phoneme recognition accuracy and inference speed, supporting effective forced alignment and multilingual transfer with just 30M parameters.

Contextless Universal Phoneme Encoder (CUPE) denotes a language-agnostic acoustic front-end for phoneme-level speech representation under tightly constrained temporal context. In its explicit 2025 formulation, CUPE is a 30M-parameter encoder that processes independent 120 ms waveform windows and outputs 10 frame-level embeddings of dimension 512, with the stated aim of capturing local acoustic-phonetic structure rather than longer-span lexical or language-specific regularities (Rehman et al., 21 Aug 2025). In subsequent forced-alignment work, the same concept is used as a frame-level phoneme classifier trained in a contextless fashion over a shared multilingual phoneme inventory and coupled to a CTC-based decoder for temporal alignment (Rehman et al., 27 Sep 2025).

1. Definition and conceptual scope

In the primary CUPE formulation, “contextless” means that each short window is processed independently rather than as part of a full word or utterance. The model is defined as

$$f_\theta : \mathbf{w}_{i} \in \mathbb{R}^{1\times W} \mapsto \mathbf{h}_{i,1:F_w} \in \mathbb{R}^{F_w \times d},$$

with $W = 1920$ samples, $F_w = 10$, and $d = 512$. At 16 kHz, this corresponds to a 120 ms input segment and 10 frame-level output embeddings per window. The design goal is “pure phoneme representations free from contextual influence,” motivated by phoneme recognition, alignment precision, and phonetic analysis (Rehman et al., 21 Aug 2025).

In the forced-alignment reinterpretation used by Bournemouth Forced Aligner, “contextless” is defined more operationally: CUPE is trained and used as a frame-wise classifier with no temporal context modeling in the acoustic encoder, so phoneme probabilities at each time step depend only on the current acoustic frame plus any local receptive field of the encoder. Temporal structure is then delegated to CTC/Viterbi decoding rather than to the encoder itself. In that system, “universal” refers to a shared multilingual phoneme representation implemented as a 67-class phoneme set and a 17-class phoneme-group set, both including blank tokens for CTC (Rehman et al., 27 Sep 2025).

The term “universal” is therefore narrower than full IPA completeness and broader than monolingual phoneme recognition. It denotes a shared output or latent space intended to support transfer across languages, including zero-shot or few-shot deployment, while minimizing dependence on language-specific phonotactics, lexical statistics, or long-span contextual cues (Rehman et al., 21 Aug 2025).

2. Antecedents and research lineage

The explicit CUPE label appears after several architectures that already implemented closely related ideas. One immediate precursor is the multilingual articulatory-attribute system built around universal manner and place inventories, supervised attribute recognizers, and deterministic attribute-to-phoneme mapping matrices $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$. That model was evaluated on English, Kyrgyz, Dutch, Russian, Swedish, and Tatar, and reported a 6.85% relative improvement in average PER over a multilingual phoneme baseline while eliminating phoneme predictions inconsistent with detected attributes (Yen et al., 2023).

A second precursor is the language-universal IPA encoder for low-resource ASR. There the encoder of a multilingual IPA model is reused as a phonetic front-end for BPE-based ASR, with the IPA decoder discarded during transfer. On MLS languages, the monolingual baseline averaged 12.45% WER, whereas the proposed system with the MLS-8 IPA encoder averaged 8.09%; the effect was strongest for smaller training languages such as Polish, Portuguese, and Italian (Feng et al., 2023).

Related multilingual G2P work established shared grapheme and phoneme spaces across large language sets. “Massively Multilingual Neural Grapheme-to-Phoneme Conversion” trained a single encoder-decoder with attention on 311 training languages and reported an 11% improvement in PER over adapted monolingual G2P systems (Peters et al., 2017). “Neural Machine Translation for Multilingual Grapheme-to-Phoneme Conversion” used a shared encoder and decoder across 18 languages with System-ID conditioning and reported a 7.2% average improvement in PER over low-resource monolingual baselines, with no degradation over high-resource ones (Sokolov et al., 2020).

A separate lineage comes from unsupervised phoneme recognition. The adversarial cluster-to-phoneme mapping model based on Audio2Vec embeddings, K-means, and a context-independent generator implemented a local mapping from discovered acoustic token clusters to phoneme distributions. In English TIMIT experiments it achieved 36.05% phoneme accuracy with unrelated text and an ensemble of 6 GAN models, but it was neither multilingual nor universal in the later CUPE sense (Liu et al., 2018).

By contrast, Mixed-Phoneme BERT is relevant mainly as a foil. It improves TTS phoneme encoding through contextual mixed phoneme and sup-phoneme representations, raises phoneme MLM accuracy from 45.40% to 70.55%, and improves TTS quality, but it is explicitly sentence-contextual and monolingual English rather than contextless or cross-lingually universal (Zhang et al., 2022).

3. Core architecture of the explicit CUPE model

CUPE operates directly on raw waveform at 16 kHz. For an input $\mathbf{x} \in \mathbb{R}^{B \times T}$, the waveform is segmented into overlapping windows of size $W = 1920$ samples and stride $s = 1280$ samples, giving

$N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$

Each window is processed independently, so different windows from the same utterance are never concatenated before the encoder transformer (Rehman et al., 21 Aug 2025).

The front-end is a hierarchical 1D-CNN stack. With base channel count $n=256$, Conv1D-1 maps $W = 1920$0 to $W = 1920$1 using kernel $W = 1920$2, stride $W = 1920$3, padding $W = 1920$4; Conv1D-2 outputs $W = 1920$5 with kernel $W = 1920$6, stride $W = 1920$7, padding $W = 1920$8; Conv1D-3 outputs $W = 1920$9 with kernel $F_w = 10$0, stride $F_w = 10$1, padding $F_w = 10$2; and Conv1D-4 outputs $F_w = 10$3 with kernel $F_w = 10$4, stride $F_w = 10$5, padding $F_w = 10$6. Each stage uses BatchNorm, GELU, and dropout 0.1. A Squeeze-and-Excitation-style frequency-attention block then reweights channels after global average pooling over time.

The convolutional representation is split into two streams. The temporal stream uses grouped Conv1d layers with kernels $F_w = 10$7 and $F_w = 10$8, stride $F_w = 10$9, paddings $d = 512$0 and $d = 512$1, to model temporal patterns across the 10 frames. The spectral stream uses two $d = 512$2 convolutions, first expanding to $d = 512$3 and then returning to $d = 512$4, to model instantaneous spectral relations at each frame. The streams are concatenated and fused by a $d = 512$5 convolution followed by BatchNorm and GELU, returning a $d = 512$6 representation.

A lightweight transformer then processes each window. The 2048-channel features are linearly projected to hidden size 512, and a 4-layer transformer encoder with 8 attention heads, pre-norm layer normalization, and dropout 0.25 produces $d = 512$7. Its receptive field is restricted to the 10 frames of one 120 ms window; it does not attend across windows (Rehman et al., 21 Aug 2025).

Two heads are defined. In supervised mode, the FT-Classifier is a 2-layer MLP $d = 512$8, where $d = 512$9 phoneme classes in the main mapping. In self-supervised mode, the PT-Projection head is $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$0 with LayerNorm, GELU, dropout 0.1, and a residual connection.

Because windows overlap, CUPE stitches windowwise logits into a continuous timeline by cosine-weighted fusion:

$S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$1

The weighting emphasizes the center of each window and de-emphasizes edges, which is intended to preserve temporal coherence while helping recognize phonemes shorter than 120 ms (Rehman et al., 21 Aug 2025).

4. Training objectives, phoneme inventory, and multilingual supervision

CUPE is trained and evaluated on a multilingual stack designed to separate pretraining, supervised training, and zero-shot testing. FLEURS contributes 102 languages and 455 hours after trimming silences for self-supervised pretraining. MSWC provides 32 high-resource languages for supervised training, totaling 181 hours, and 6 low-resource evaluation languages—lt, mt, ia, sk, ka, and as. The UCLA Phonetic Corpus contributes read speech from 95 languages and is partitioned into UPC-eval languages unseen in XLS-R pretraining and FLEURS, UPC-seen languages overlapping those pretraining sets, and 6 validation languages used during supervised training (Rehman et al., 21 Aug 2025).

The phoneme space is a manually constructed 65-class mapping. Frequent and perceptually distinct contrasts are preserved, including palatalized consonants such as $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$2, $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$3, and $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$4, vowel contrasts such as $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$5 versus $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$6 and $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$7 versus $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$8, and length distinctions for frequent vowels such as $S^C \in \{0,1\}^{|C| \times |P_{\text{uni}}|}$9, $\mathbf{x} \in \mathbb{R}^{B \times T}$0, $\mathbf{x} \in \mathbb{R}^{B \times T}$1, $\mathbf{x} \in \mathbb{R}^{B \times T}$2, and $\mathbf{x} \in \mathbb{R}^{B \times T}$3. Rare phonemes are mapped to frequent acoustically similar ones, for example $\mathbf{x} \in \mathbb{R}^{B \times T}$4, $\mathbf{x} \in \mathbb{R}^{B \times T}$5, and $\mathbf{x} \in \mathbb{R}^{B \times T}$6; common affricates such as $\mathbf{x} \in \mathbb{R}^{B \times T}$7, $\mathbf{x} \in \mathbb{R}^{B \times T}$8, and $\mathbf{x} \in \mathbb{R}^{B \times T}$9 remain distinct.

Supervised training uses CTC on stitched frame-level outputs, augmented with an explicit silence-awareness term:

$W = 1920$0

with $W = 1920$1, and

$W = 1920$2

Here $W = 1920$3 is the blank-token probability and $W = 1920$4 is the silence mask. The term raises blank probability in silence regions and lowers it in non-silence regions. Supervised optimization uses AdamW, OneCycleLR, gradient clipping at $W = 1920$5, BF16 mixed precision, batch size 300 words, and roughly 20 epochs on MSWC-train, taking about 7 hours on 2 $W = 1920$6 A6000 GPUs (Rehman et al., 21 Aug 2025).

Self-supervised CUPE-PT retains the same backbone but replaces the classifier by the 256-dimensional projection head. Pretraining uses FLEURS, the same 120 ms windows with 80 ms stride, masking of 40% of features per batch with per-batch mask proportions constrained between 10% and 80%, and a vector quantizer with 256 entries updated by EMA with decay 0.99 and Laplace smoothing. The loss combines smooth L1 reconstruction, contrastive learning with curriculum scheduling, codebook diversity regularization, and similarity regularization. During downstream evaluation, the feature extractor—CNN, temporal/spectral streams, and fusion—is frozen; the prediction head is removed; and only the transformer encoder plus FT-Classifier are fine-tuned (Rehman et al., 21 Aug 2025).

The evaluation metrics are PER, macro and weighted ground-truth probability (GPm and GPw), and aligned-sequence F1. GP measures how much probability the model assigns to the correct phoneme at aligned time steps, making it sensitive to near misses that PER treats as full errors (Rehman et al., 21 Aug 2025).

5. Empirical performance and cross-lingual generalization

In supervised training without self-supervised pretraining, CUPE:120ms reaches PER $W = 1920$7, GPm $W = 1920$8, GPw $W = 1920$9, and F1 $s = 1280$0 on MSWC-eval. On the same benchmark, XLSR:word records PER $s = 1280$1, GPm $s = 1280$2, GPw $s = 1280$3, and F1 $s = 1280$4, while XLSR:120ms records PER $s = 1280$5. On UPC-eval, CUPE:120ms reaches PER $s = 1280$6, GPm $s = 1280$7, GPw $s = 1280$8, and F1 $s = 1280$9, compared with XLSR:word at PER $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$0, GPm $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$1, GPw $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$2, and F1 $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$3. Longer context does not dominate uniformly: CUPE:360ms improves PER to $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$4 on MSWC-eval and $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$5 on UPC-eval, but 120 ms remains stronger in GP and F1, which the authors interpret as finer phonetic quality and more balanced phoneme behavior (Rehman et al., 21 Aug 2025).

With self-supervised pretraining, CUPE-PT further strengthens the local-context regime. On MSWC-eval, fully fine-tuned CUPE-PT reaches PER $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$6, GPm $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$7, GPw $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$8, and F1 $N = \left\lfloor\frac{T - W}{s}\right\rfloor + 1.$9, compared with pretrained XLS-R at PER $n=256$0, GPm $n=256$1, GPw $n=256$2, and F1 $n=256$3. Under frozen-backbone evaluation, FB-CUPE-PT reaches PER $n=256$4 and F1 $n=256$5, whereas FB-XLSR reaches PER $n=256$6 and F1 $n=256$7. On UPC-eval, CUPE-PT records PER $n=256$8 versus $n=256$9 for XLS-R; on UPC-seen it records $W = 1920$00 versus $W = 1920$01. The gap under frozen-backbone conditions is especially notable because it suggests that the local feature extractor itself captures transferable phonetic information before task-specific fine-tuning (Rehman et al., 21 Aug 2025).

CUPE is also compared with prior universal phone-recognition systems under published UPC splits. On 47 unseen languages, a prior system reported PER $W = 1920$02, whereas CUPE reported $W = 1920$03. On a 10-language split, a prior system reported $W = 1920$04, whereas CUPE reported $W = 1920$05. On 77 languages, a prior system reported $W = 1920$06, whereas CUPE reported $W = 1920$07. On the 84-language Allophant comparison, Allophant reported PER $W = 1920$08 using 35 articulatory attributes, whereas CUPE reported $W = 1920$09 using 65 mapped phoneme classes. The paper explicitly notes that direct comparison is imperfect because phoneme inventories and label spaces differ (Rehman et al., 21 Aug 2025).

A central quantitative theme is efficiency. CUPE has about 30M parameters, with about 13M in transformer layers, whereas the XLS-R baseline has 300M+ parameters. The reported result is therefore not merely competitive accuracy at a smaller scale, but competitive zero-shot phoneme recognition under a deliberately local architectural prior (Rehman et al., 21 Aug 2025).

6. Downstream realization in forced alignment

The most explicit downstream deployment of CUPE is Bournemouth Forced Aligner. Its pipeline has three stages: CUPE as the acoustic front-end, multilingual phonemization via espeak-ng followed by mapping from IPA to CUPE’s universal phoneme inventory, and a CTC-based dynamic-programming decoder that aligns the target phoneme sequence to time. CUPE emits posterior probabilities for a 67-class phoneme set and a 17-class phoneme-group set, both including blank tokens. The CTC path is constructed as

$W = 1920$10

and the forward Viterbi recursion is

$W = 1920$11

Blank states model inter-phoneme gaps and silences, enabling separate onset and offset prediction rather than forced boundary abutment (Rehman et al., 27 Sep 2025).

Three training variants are defined. BFA$W = 1920$12 is trained on LibriSpeech for English optimization. BFA$W = 1920$13 is trained on Multilingual LibriSpeech across 7 European languages excluding English. BFA$W = 1920$14 is trained on MSWC across 35 languages excluding English. This setup provides a concrete test of the claim that CUPE’s phoneme representation can generalize cross-lingually even when the target language is absent from training (Rehman et al., 27 Sep 2025).

On TIMIT at 20 ms tolerance, phone-level recall is $W = 1920$15 for MFA, $W = 1920$16 for BFA$W = 1920$17, $W = 1920$18 for BFA$W = 1920$19, and $W = 1920$20 for BFA$W = 1920$21. On Buckeye at 20 ms tolerance, recall is $W = 1920$22 for MFA, $W = 1920$23 for BFA$W = 1920$24, $W = 1920$25 for BFA$W = 1920$26, and $W = 1920$27 for BFA$W = 1920$28. The discussion highlights that BFA$W = 1920$29 and BFA$W = 1920$30, both trained without English data, remain competitive on English test sets, which is presented as evidence for universal phoneme representation (Rehman et al., 27 Sep 2025).

The temporal behavior of CUPE in this pipeline is distinctive. On TIMIT, the percentage of phonemes with preceding gaps is $W = 1920$31 for MFA but $W = 1920$32, $W = 1920$33, and $W = 1920$34 for BFA$W = 1920$35, BFA$W = 1920$36, and BFA$W = 1920$37, respectively. On Buckeye the corresponding values are $W = 1920$38, $W = 1920$39, $W = 1920$40, and $W = 1920$41. This blank-aware decoding yields explicit inter-phoneme gaps in roughly 30–40% of transitions rather than near-zero rates, and mean boundary distance on TIMIT remains close to MFA: 13.2 ms for MFA versus 14.1 ms, 14.4 ms, and 16.4 ms for the three BFA models (Rehman et al., 27 Sep 2025).

Speed is a major practical consequence of the contextless front-end. On TIMIT, with average clips of 3.1 s, MFA requires about 1 min/clip whereas BFA requires 0.25 s/clip. On Buckeye, with average clips of 535 s, MFA requires 45 min/clip whereas BFA requires 60 s/clip. Real-time factors are reported as 52–194$W = 1920$42 slower than real time for MFA and $W = 1920$43–$W = 1920$44 for BFA, כלומר 10–20$W = 1920$45 faster than real time. An ablation also shows that decoder-side calibration matters: for BFA$W = 1920$46, removing probability boosting drops TIMIT 20 ms recall from $W = 1920$47 to $W = 1920$48, even though boundary-distance metrics remain largely unaffected (Rehman et al., 27 Sep 2025).

7. Limitations, misconceptions, and open directions

A common misconception is that “contextless” implies zero local context. The published CUPE architecture still contains a local transformer over 10 frames within each 120 ms window, and the BFA interpretation explicitly allows dependence on any local receptive field of the encoder. What is excluded is cross-window or long-span sequence conditioning at the acoustic-encoder stage, not all local temporal integration (Rehman et al., 21 Aug 2025, Rehman et al., 27 Sep 2025).

Another misconception is that “universal” means exhaustive phonological coverage. The 120 ms CUPE system uses a 65-class mapped inventory that deliberately merges rare phonemes into acoustically similar frequent ones, which can obscure subtle contrasts. The BFA phoneme mapping is derived from multilingual frequency analysis and is described as optimized for Indo-European languages; its evaluation is only on English corpora. Earlier articulatory-constraint work is likewise limited by representation granularity: with only manner and place categories, multiple phonemes share identical attribute vectors, and all vowels collapse to “vowel” in both categories (Rehman et al., 21 Aug 2025, Rehman et al., 27 Sep 2025, Yen et al., 2023).

The fixed 120 ms window is itself a trade-off. The CUPE paper states that it is too long for very short phonemes such as stops and too short to capture all long phonemes; languages with strong length contrasts are correspondingly harder. It also reports that the 30M design is specialized for local contexts and does not scale well to word-level context, where frozen-backbone performance degrades. The self-supervised objective is described as sub-optimal, with open issues in target quantization and loss weighting. In BFA, the broadest multilingual model, BFA$W = 1920$49, underperforms the English and European variants and is especially sensitive to probability boosting, indicating that universality remains conditioned by inventory design and calibration (Rehman et al., 21 Aug 2025, Rehman et al., 27 Sep 2025).

The open research directions are consistent across the literature. The explicit CUPE paper proposes variable window sizes or multi-scale architectures, improved SSL objectives and codebook design, expansion beyond the current 65-class inventory, and sentence-level style encoders built on allophone embeddings. BFA identifies tonal languages, multilingual speech-processing pipelines, and adaptive boundary-prediction strategies as future targets. Earlier universal attribute work suggests richer articulatory inventories beyond manner and place. A plausible implication is that future CUPE systems will combine a context-restricted acoustic front-end with richer universal phonological factorization rather than relying on a single flat phoneme label space (Rehman et al., 21 Aug 2025, Rehman et al., 27 Sep 2025, Yen et al., 2023).

A final boundary condition comes from contextual phoneme modeling in TTS. Mixed-Phoneme BERT shows that contextual mixed phoneme and sup-phoneme representations improve prosody and perceived naturalness in English TTS. This suggests that a contextless phonetic encoder and a contextual prosodic module solve different problems: the former isolates local acoustic-phonetic content, while the latter can be advantageous when sentence-level semantic and prosodic structure are primary objectives (Zhang et al., 2022).