VocAlign: Domain Adaptation & Alignment Framework

• VocAlign is a domain adaptation framework that aligns visual and textual features for open-vocabulary semantic segmentation without requiring source domain labels.
• It employs vocabulary alignment to aggregate concept probabilities and utilizes LoRA-based fine-tuning with Top-K class selection to enhance segmentation metrics.
• The framework exemplifies a family of explicit alignment mechanisms across modalities, bridging research in segmentation, cross-lingual, phoneme, and transcript alignments.

VocAlign most specifically denotes a source-free domain adaptation framework for open-vocabulary semantic segmentation with vision–LLMs, introduced as a student–teacher method that combines vocabulary alignment, LoRA-based fine-tuning, and Top-K class selection (Mazzucco et al., 18 Sep 2025). In a broader research usage reflected by adjacent work, the term is also associated with systems that explicitly align words, subwords, tokens, phonemes, codec sequences, lyric words, or transcript spans across languages and modalities rather than treating alignment as an incidental by-product of pretraining or attention (Chi et al., 2021, Han et al., 2024, Borgholt et al., 29 Sep 2025). Across these settings, the shared technical theme is the construction of explicit correspondence structures—probability matrices, monotonic pointers, token-mapping lexica, or segment-level paths—that are optimized for transfer, generation, evaluation, or synchronization.

1. Scope and conceptual landscape

In the material summarized here, the name “VocAlign” has both a narrow and a broad meaning. Narrowly, it is the title of a segmentation method for source-free domain adaptation in open-vocabulary semantic segmentation (Mazzucco et al., 18 Sep 2025). More broadly, it functions as a useful label for a family of alignment problems in which discrete units must be matched across languages, modalities, or tokenizations: cross-lingual word alignment, tokenizer adaptation, phoneme–audio alignment, lyrics alignment, transcript alignment, and bitext alignment (Chi et al., 2021, Kang et al., 2023, Borgholt et al., 29 Sep 2025).

Research setting	Aligned units	Representative systems
Open-vocabulary segmentation	Pixels, classes, text concepts	VocAlign (Mazzucco et al., 18 Sep 2025)
Cross-lingual NLP and LLM adaptation	Words, subwords, token IDs	XLM-Align, ALIGN-MLM, TokAlign (Chi et al., 2021, Tang et al., 2022, Li et al., 4 Jun 2025)
Speech and singing	Phonemes, codec tokens, lyric words, learned audio tokens	VALL-E R, HCLAS-X, PairAlign (Han et al., 2024, Kang et al., 2023, Banerjee et al., 7 May 2026)
Transcript and bitext alignment	Word spans, sentences, sentence groups	error-align, AIlign (Borgholt et al., 29 Sep 2025, Kraif, 2024)

This breadth matters because the same alignment primitives recur in otherwise distant areas. Optimal transport and pointer networks are used to align bilingual tokens; monotonic pointers and forced-alignment outputs connect phonemes to acoustic tokens; dynamic programming and beam search refine transcript-to-transcript correspondences; and text-space concept aggregation improves pseudo-labeling in segmentation. This suggests that “VocAlign” is best understood as a family of explicit alignment mechanisms rather than a single architecture.

2. VocAlign as a framework for open-vocabulary semantic segmentation

VocAlign, in the explicit sense of the term, is a source-free domain adaptation framework for open-vocabulary semantic segmentation built on CAT-Seg, a CLIP-based segmenter with an image encoder, a text encoder, a cost volume, two Swin-transformer aggregation modules, and a decoder (Mazzucco et al., 18 Sep 2025). The source domain is unavailable during adaptation; only unlabeled target images are provided. The method therefore combines pseudo-labeling with text-space augmentation.

The core segmentation setting uses dense visual features

$$\mathcal{D}^V = \phi^V(x^T) \in \mathbb{R}^{H \times W \times d_V}$$

and textual embeddings

$$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$

with a cost volume $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$ formed from cosine similarities between pixel features and prompted class embeddings. The teacher $g_\phi$ produces pseudo-labels for target images, the student $f_\theta$ learns from masked target inputs, and the teacher is updated by exponential moving average,

$$\phi_{t+1} \leftarrow \alpha \phi_t + (1-\alpha)\theta_t,$$

with $\alpha = 0.99$ (Mazzucco et al., 18 Sep 2025).

The distinctive mechanism is vocabulary alignment in the text space. Each dataset class $n$ is augmented with additional concepts $c_h(n)$, and concept-level probabilities

$$p_{ij}^c = \frac{e^{x_{ij}^c}}{\sum_{c'=1}^{n_{\text{tot}}} e^{x_{ij}^{c'}}}$$

are aggregated back to dataset classes by

$$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$0

This changes pseudo-label formation from dependence on a single class phrase to dependence on a set of synonymous or closely related concepts. The same aggregated probabilities are used for Top-K class selection. For each class $$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$1,

$$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$2

and only the Top-K classes with highest $$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$3 are kept for the student, reducing memory and focusing training on classes the teacher considers likely to be present (Mazzucco et al., 18 Sep 2025).

LoRA is used to make adaptation parameter-efficient. The adapted weight is

$$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$4

with rank $$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$5, and VocAlign applies LoRA to attention projection matrices in the first 4 layers of both the CLIP image encoder and the CLIP text encoder, while the remainder of CAT-Seg is frozen (Mazzucco et al., 18 Sep 2025). This design is intended to preserve the original open-vocabulary capability while reducing computational overhead.

Empirically, the framework reports a zero-shot Cityscapes mIoU of 47.56 and a VocAlign mIoU of 53.67, a gain of 6.11 mIoU. Per-class behavior is uneven but instructive: “truck” rises from 23.22 to 46.76, “wall” from 9.77 to 34.32, and “terrain” from 0.02 to 45.95, while “car” drops from 76.38 to 67.48 (Mazzucco et al., 18 Sep 2025). On reduced 10-prompt CAT-Seg settings, the full combination of masking, vocabulary alignment, and Top-K yields 49.58 on Cityscapes, 57.01 on PASCAL-Context 59, and 27.39 on ADE20K-150; with the full 80-prompt CAT-Seg, the corresponding values are 48.97, 57.32, and 27.40 (Mazzucco et al., 18 Sep 2025). A plausible implication is that the vocabulary-alignment term is most valuable when dataset label names are poorly matched to the text distribution learned by CLIP.

3. Cross-lingual lexical and token alignment

A large strand of related work treats alignment itself as the pretraining target. XLM-Align introduces denoising word alignment (DWA) for cross-lingual LLMs by alternating an E-step that self-labels word alignments with optimal transport and an M-step that predicts aligned tokens with a pointer network under masking (Chi et al., 2021). For a source sentence $$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$6 and target sentence $$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$7, the model computes a doubly stochastic alignment matrix $$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$8 by optimal transport over contextual embeddings and extracts high-precision hard links after Itermax-style filtering. In the denoising phase, aligned masked tokens are trained with a pointer network using scaled dot-product attention and cross-entropy loss. The method improves XTREME average score from 66.4 to 68.9, improves TyDiQA F1 from 55.4 to 62.1, and yields the best average AER among the reported aligners, with OT alignment on XLM-Align at 21.05 versus 24.08 for SimAlign on XLM-R (Chi et al., 2021). In the broader VocAlign sense, this is a direct example of “vocabulary alignment” as a pretraining objective.

ALIGN-MLM argues that word embedding alignment is a central determinant of multilingual transfer and formalizes this through an auxiliary loss over bilingual dictionary pairs (Tang et al., 2022). The objective is

$$\mathcal{D}^L = \phi^L(t) \in \mathbb{R}^{N_c \times P \times d_L},$$9

with $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$0, where $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$1 increases the cosine similarity of embeddings corresponding to words in different languages with the same meaning. In controlled language transformations, ALIGN-MLM outperforms XLM and MLM by 35 and 30 F1 points on POS-tagging when languages differ in both script and word order, and the reported Spearman correlations between alignment and transfer are 0.727 for XNLI, 0.781 for NER, and 0.734 for POS (Tang et al., 2022). This directly challenges the common assumption that multilingual masked language modeling alone is sufficient.

A complementary discriminative line of work models token-pair alignment probabilities directly on top of a Transformer MT model. The discriminative neural model for cross-lingual word alignment computes contextual encoder and decoder states, projects them into a shared space with a 3-layer feed-forward network, forms a similarity matrix, applies a $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$2 convolution over that matrix, and predicts

$$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$3

for each source–target token pair (Stengel-Eskin et al., 2019). With only 4,871 labeled Chinese sentence pairs and 1,687 labeled Arabic sentence pairs, it reports F1 73.41 on English–Chinese and 82.74 on English–Arabic, yielding improvements of 11–27 F1 over unsupervised baselines. It also raises projected Chinese NER from 37.77 F1 under FastAlign-based projection to 51.57 under the discriminative aligner (Stengel-Eskin et al., 2019). One recurrent lesson is explicit in the paper: attention weights from MT are not reliable substitutes for alignment.

The tokenizer-adaptation literature pushes the alignment unit below the word. TokAlign replaces an LLM vocabulary by learning a one-to-one mapping matrix for token IDs from global token co-occurrences, rearranging embeddings and LM-head parameters, and progressively fine-tuning the model on the new vocabulary (Li et al., 4 Jun 2025). It reports a normalized perplexity of $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$4 after initialization, compared with $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$5 for strong baseline methods, improves multilingual text compression rates by 29.2% across 13 languages, and restores vanilla-model performance in as few as 5k steps (Li et al., 4 Jun 2025). TokAlign++ extends this by treating source and target vocabularies as different languages, learning a bilingual token lexicon with VecMap and CSLS, and using either GloVe or source-LLM hidden states as token representations (Li et al., 13 May 2026). It reports that the average token length after tokenization is reduced by 47.6% across 15 languages, that performance can be restored in as few as 1k steps, and that token-level distillation becomes markedly stronger after vocabulary unification, with a 1B model reaching or surpassing vanilla 7B in-context ability under some settings after only 235M tokens of distillation (Li et al., 13 May 2026). This suggests that vocabulary alignment can be treated not only as a semantic correspondence problem but also as a systems problem in sequence length, compression, and distillation compatibility.

4. Phoneme, codec, lyrics, and audio alignment

Speech and singing alignment research uses the same alignment logic but with different units. VALL-E R is a zero-shot TTS system that introduces a phoneme monotonic alignment strategy so that acoustic tokens are constrained to remain aligned to a current phoneme or its immediate successor (Han et al., 2024). The model uses an external aligner, Montreal Forced Aligner, during training to derive a repeated frame-level phoneme sequence $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$6, and at inference it updates a phoneme pointer with a Bernoulli rule under the constraint

$$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$7

The paper identifies three properties—locality, monotonicity, and completeness—and explicitly links them to robustness against typos, omissions, and repetition. On the continuation task it reports WER 1.58 for VALL-E R, near the Encodec upper bound of 1.62 and substantially below VALL-E’s 2.37; on the cross-sentence task it reports 3.18 versus 5.48 for VALL-E. With codec merging, the method reduces autoregressive steps from 750 to 375 per 10 seconds and reaches 3.67 seconds inference time, a reduction of more than 60% relative to VALL-E’s 10.27 seconds (Han et al., 2024). Because VALL-E R explicitly predicts aligned phoneme sequences, it also supports prosody control; the VALL-E R-Prosody configuration reports MCD-DTW-SL 7.82 versus 8.55 for VALL-E R and 9.03 for VALL-E (Han et al., 2024).

At finer temporal granularity, the VAE-based phoneme alignment model extends one TTS alignment by adding a VAE constraint, gradient annealing, SSL acoustic features, and state-level linguistic units (Koriyama, 2024). The alignment loss is a forward-sum objective over monotonic no-skip paths,

$$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$8

with $$\mathcal{C} \in \mathbb{R}^{H \times W \times P \times N_c}$$9 defined by acoustic and linguistic embeddings and a beta-binomial position prior. Gradient annealing smooths state occupancy probabilities with a Gaussian over the state dimension, beginning with $g_\phi$0 and multiplying by $g_\phi$1 every 1000 steps (Koriyama, 2024). Using HuBERT(ja) L0, VAE, annealing, and 3-state linguistic units, the method reports MAE 12.91 ms, median 8.25 ms, 20 ms tolerance error 16.1%, and 50 ms tolerance error 2.59%, outperforming MFA at 16.46 ms MAE and conventional OTA at 23.10 ms MAE (Koriyama, 2024). The result is notable because no phoneme boundary labels are used during training.

For long-form songs, HCLAS-X aligns lyrics and vocals hierarchically by first predicting sentence-level synced time and then word-level synced time from shorter slices (Kang et al., 2023). Its central representation is a multimodal cross-correlation tensor

$g_\phi$2

computed between CBHG-encoded text and audio features and then processed by a UNet plus GRU to produce an alignment matrix $g_\phi$3. Word or sentence start time is extracted by

$g_\phi$4

On Jamendo, the in-house model HX-IH reports MAE 0.16, MedAE 0.043, Perc 89.0%, Mauch$g_\phi$5 0.93, and Mauch$g_\phi$6 0.91; average end-to-end processing time is 4.82 seconds per song, of which 0.74 seconds is sentence-level inference and 0.11 seconds is word-level inference (Kang et al., 2023). On the Mandarin Pop Song dataset, the sentence-level module reports MAE 0.12 and Perc 98.2 (Kang et al., 2023). The design principle is explicit: hierarchical processing is required because the cross-correlation stage has quadratic memory in sequence length.

The earlier DNN-HMM literature on lyrics-to-audio alignment addresses polyphonic mixes by adapting solo-singing acoustic models and enriching the feature space with auditory, energy, chroma, spectral, and voicing descriptors (Gupta et al., 2019). After adapting on 70 polyphonic DALI songs, the best reported configuration achieves mean AE 0.93 s on Hansen-poly and 1.93 s on Mauch-poly, with median AEs 0.15 s and 0.18 s respectively (Gupta et al., 2019). This is a different alignment regime from the CBHG/UNet line, but it emphasizes the same issue: non-vocal regions and accompaniment must be modeled explicitly or at least rendered separable in the feature space.

PairAlign moves one step further by learning symbolic audio sequences through self-alignment rather than using phoneme labels (Banerjee et al., 7 May 2026). It treats tokenization as conditional sequence generation and trains two content-preserving views so that each view’s token sequence is likely under the other view’s representation while unrelated examples provide competing sequences. On 3-second speech, PairAlign reduces archive token count by 55% on TIMIT retrieval while preserving edit-distance search, and in a continuous-sweep probe 65.8% of adjacent pairs have edit distance at most 20 versus 13.2% for the geometric baseline (Banerjee et al., 7 May 2026). This suggests an alternative interpretation of VocAlign: alignment need not presuppose a fixed phoneme inventory if the symbolic units themselves are learned to preserve cross-view edit geometry.

5. Transcript and bitext alignment as evaluation infrastructure

Alignment is also central to evaluation. The text-to-text alignment algorithm released through the error-align package is designed for ASR error analysis rather than recognition itself (Borgholt et al., 29 Sep 2025). It represents an alignment as a sequence of substring-range pairs $g_\phi$7, uses dynamic programming to construct a backtrace graph, and then performs beam search with a transition cost that distinguishes matches, insertions, deletions, and substitutions according to whether characters are voiced, unvoiced, vowels, or consonants. The per-edge transition cost is

$g_\phi$8

and the beam-search score includes penalties for deviations from the backtrace graph and for substitutions within open segments (Borgholt et al., 29 Sep 2025). With beam size 100, the method reports consistently higher Global-to-Local Edits scores than power alignment, optimized word-level alignment, and standard word-level Levenshtein. On Common Voice English with Whisper, for example, character GLE is 78.8 for the proposed method, versus 77.0 for power alignment, 65.8 for optimized word-level alignment, and 58.9 for word-level Levenshtein; phoneme GLE is 74.2 versus 73.1, 60.6, and 54.2 (Borgholt et al., 29 Sep 2025). The paper argues that improved local alignment is necessary because WER obscures errors in rare terms, named entities, and domain-specific vocabulary.

At the sentence level, AIlign uses multilingual sentence embeddings to identify anchor points that can guide alignment even when texts are only locally parallel or locally monotonic (Kraif, 2024). It computes a cosine-similarity matrix over sentence embeddings, extracts anchors through mutual k-best selection, a margin threshold of 0.05, a cosine threshold of 0.4, and local density filtering, then runs dynamic programming only within alignable intervals (Kraif, 2024). The per-group distance combines embedding distance, a neighbor-similarity correction, a group-size penalty, and a length term,

$g_\phi$9

with $f_\theta$0, and the method supports groupings up to size 4 (Kraif, 2024). On Text+Berg, MD.ar-en, and BAF it achieves results close to Bertalign with much lower runtime; on a laptop the reported times are 119 seconds versus 590 seconds on Text+Berg, 2166 versus 8114 seconds on MD.ar-en, and 1437 versus 10882 seconds on BAF (Kraif, 2024). On the Grimm corpus, which involves severe reordering and partial translation, it reaches tale-level precision 98.7, recall 92.8, and F1 95.7 (Kraif, 2024). In the wider VocAlign landscape, this work shows how anchor-point density and interval detection can replace global monotonicity assumptions.

6. Recurring principles, limitations, and research directions

A recurrent principle across these works is that alignment is most effective when it is modeled explicitly rather than inferred indirectly from generic attention or pretraining objectives. The discriminative neural aligner reports attention F1 as low as 40.65 on English–Chinese and 13.42 on English–Arabic, while its alignment-specific head reaches 73.41 and 82.74 respectively (Stengel-Eskin et al., 2019). ALIGN-MLM similarly argues that explicit embedding alignment is crucial, and its controlled experiments show that MLM and XLM can fail badly once script overlap and word order correspondence are broken (Tang et al., 2022). This suggests a general misconception to reject: shared representations do not automatically imply usable alignments.

Another recurring principle is that alignment quality depends on the quality of side information. In open-vocabulary segmentation, concept quality determines whether vocabulary alignment sharpens or blurs pseudo-labels; the paper explicitly notes smaller gains on ADE20K-150 and identifies concept quality as a limitation (Mazzucco et al., 18 Sep 2025). VALL-E R still depends on Montreal Forced Aligner during training, and its authors identify replacement of MFA by learned alignment as an open question (Han et al., 2024). The ASR error-align work notes tokenization dependence, Python-level speed limits, and a Latin-script focus, with C++ bindings and more scripts as future work (Borgholt et al., 29 Sep 2025). TokAlign++ observes that adaptation becomes harder when the target vocabulary is smaller than the source vocabulary and identifies true zero-shot vocabulary adaptation and better long-tail token alignment as open challenges (Li et al., 13 May 2026). These are not incidental engineering issues; they delimit where alignment methods can be trusted.

The family resemblance among these systems is structural. VocAlign in segmentation aggregates multiple concept probabilities back to a target class. XLM-Align alternates between self-labeled alignment induction and denoising prediction. VALL-E R constrains decoding with a stay-or-step monotonic pointer. error-align anchors beam search to a classical backtrace graph. AIlign detects anchor intervals before local dynamic programming. TokAlign and TokAlign++ rearrange embedding and output matrices according to a learned bilingual token lexicon. PairAlign learns a symbolic sequence precisely so that edit distance is a useful proxy for content similarity. Taken together, these works suggest that alignment is most effective when represented as an object in its own right: a class-concept aggregation, a doubly stochastic matrix, a pointer trajectory, a substring lattice, a sentence-anchor interval structure, or a token-mapping lexicon.

Future directions stated in the literature are correspondingly explicit. VocAlign proposes hierarchical vocabulary alignment, adaptive concept selection, and richer text prompts for open-vocabulary segmentation (Mazzucco et al., 18 Sep 2025). VALL-E R points toward learned monotonic alignment modules and more explicit duration control (Han et al., 2024). error-align proposes faster implementations and broader language coverage (Borgholt et al., 29 Sep 2025). TokAlign++ targets true zero-shot tokenizer transfer and more robust low-resource token alignment (Li et al., 13 May 2026). ALIGN-MLM points toward explicit alignment objectives in larger multilingual pretraining regimes beyond synthetic language pairs (Tang et al., 2022). The shared implication is that alignment is becoming a first-class optimization target across modern machine learning systems, whether the units are pixels, words, subwords, phonemes, or learned symbolic audio tokens.