VQ-CAlign Module: Diverse Quantization Alignment

• VQ-CAlign is a family of modules integrating vector quantization with alignment or calibration objectives across varied tasks such as compression, text-image synthesis, and VQA.
• It employs techniques like cosine similarity assignment, hard codeword selection, and optimal transport with tailored losses and gradient estimators for each context.
• These mechanisms enhance performance by offering discrete representation learning, improved calibration, and semantic coupling without necessarily adding extra VQ-specific losses.

Searching arXiv for the cited papers and the term "VQ-CAlign" to ground the article in the current literature. arXiv query: VQ-CAlign module (Gou et al., 25 Apr 2026, Liang et al., 3 Mar 2025, Pandey et al., 14 Nov 2025, Barbera et al., 20 May 2026, Alaniz et al., 2022) “VQ-CAlign” is not a single standardized module in the arXiv literature summarized here. Instead, the label is used for several paper-specific mechanisms that combine vector quantization with alignment, calibration, or coupling objectives in distinct problem settings: quantization-aware training and model compression, text-aligned codebook learning, confidence calibration in Visual Question Answering, multiclass local calibration, and semantically coupled image synthesis (Gou et al., 25 Apr 2026, Liang et al., 3 Mar 2025, Pandey et al., 14 Nov 2025, Barbera et al., 20 May 2026, Alaniz et al., 2022). This suggests that the term functions as a local architectural name rather than a universally fixed design. Across these uses, the common substrate is discrete representation learning through codebooks, but the surrounding objectives, losses, and inference semantics differ substantially.

1. Terminological scope and paper-specific meanings

In the sources considered here, “VQ-CAlign” denotes at least five distinct modules.

Paper	Domain	Core role of “VQ-CAlign”
"Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks" (Gou et al., 25 Apr 2026)	model weight compression	cosine-similarity–based vector-quantization alignment
"Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text" (Liang et al., 3 Mar 2025)	image quantization and cross-modal learning	hierarchical codebook–text alignment
"Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQA" (Pandey et al., 14 Nov 2025)	VQA calibration	calibration-alignment component
"Divide et Calibra: Multiclass Local Calibration via Vector Quantization" (Barbera et al., 20 May 2026)	multiclass calibration	local calibration via vector quantization
"Semantic Image Synthesis with Semantically Coupled VQ-Model" (Alaniz et al., 2022)	semantic image synthesis	semantically coupled VQ-Model

The heterogeneity is substantive rather than merely terminological. In one case, the module performs hard codeword selection for quantized weights using cosine similarity and a straight-through estimator; in another, it aligns image code sequences with words, phrases, and sentences using sampling-based optimal transport; in another, it is a calibration-aware fine-tuning and aggregation mechanism for VQA agents; in another, it constructs region-specific Dirichlet calibration maps from vector-quantized latent regions; and in another, it couples semantic and image latents through a shared codebook (Gou et al., 25 Apr 2026, Liang et al., 3 Mar 2025, Pandey et al., 14 Nov 2025, Barbera et al., 20 May 2026, Alaniz et al., 2022).

A common misconception would be to treat “VQ-CAlign” as a single canonical block with a fixed loss or quantizer. The literature summarized here does not support that interpretation. Some variants use no additional VQ-specific losses, some retain standard VQ-VAE commitment losses, and some are not primarily codebook-learning modules at all, but calibration mechanisms layered on top of VLM or classifier outputs (Gou et al., 25 Apr 2026, Liang et al., 3 Mar 2025, Pandey et al., 14 Nov 2025).

2. Cosine-similarity assignment in quantization-aware training

In "Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks" (Gou et al., 25 Apr 2026), VQ-CAlign is a cosine-similarity–based vector-quantization alignment module for model weight compression. Let $x\in\mathbb{R}^d$ be an input vector and let $C=\{c_1,\cdots,c_N\}$ be a learnable codebook of $N$ entries $c_j\in\mathbb{R}^d$. The module first forms $L_2$-normalized keys

$k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$

then computes cosine similarities

$$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$

Optionally, these similarities are converted into a soft assignment

$p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$

with the stated caveat that this distribution is used only in the backward pass (Gou et al., 25 Apr 2026).

The forward pass is explicitly hard. The module picks the index

$m = \arg\max_j p_j,$

forms a one-hot vector $z\in\{0,1\}^N$, and outputs

$C=\{c_1,\cdots,c_N\}$0

Because $C=\{c_1,\cdots,c_N\}$1 is discrete, the method applies the standard straight-through estimator. During back-propagation it pretends $C=\{c_1,\cdots,c_N\}$2, so that

$C=\{c_1,\cdots,c_N\}$3

Since $C=\{c_1,\cdots,c_N\}$4, one obtains

$C=\{c_1,\cdots,c_N\}$5

The reconstruction step is therefore simply

$C=\{c_1,\cdots,c_N\}$6

When inserted into a network layer, the forward pass substitutes each block of $C=\{c_1,\cdots,c_N\}$7 floating-point parameters or activations by its single corresponding codeword before further computation (Gou et al., 25 Apr 2026).

A defining feature of this variant is the absence of additional VQ-specific losses. Unlike many VQ-VAE or DKM schemes, it does not introduce extra reconstruction, commitment-, or codebook-update losses. Instead, it relies solely on the task loss, may optionally add standard weight decay, and in mixed-quantization with ProxylessNAS adds a storage penalty. The overall QAT loss in the ProxylessNAS setting is

$C=\{c_1,\cdots,c_N\}$8

with

$C=\{c_1,\cdots,c_N\}$9

The paper attributes the absence of extra VQ losses to two stated mechanisms: codebook vectors receive direct gradient signals via STE, and cosine assignment balances codebook usage, mitigating collapse (Gou et al., 25 Apr 2026).

The same work situates VQ-CAlign inside each quantized layer during QAT. Full-precision weights are grouped into $N$0-dimensional vectors, each passed through the VQ-CAlign block to produce $N$1, and gradients update both the original vectors and the codebook entries. The authors further state that cosine-similarity assignment emphasizes directional diversity of codewords and naturally yields a more uniform assignment of vectors to centroids, directly reducing the chance of “dead” codewords without explicit eviction or regularization. Because forward and backward graphs are identical to inference, no temperature scheduling is required. In a larger mixed-quantization pipeline, VQ-CAlign competes against a conventional linear quantizer, with per-layer switches learned via another STE (Gou et al., 25 Apr 2026).

3. Hierarchical codebook–text alignment with long text

In "Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text" (Liang et al., 3 Mar 2025), VQ-CAlign is a hierarchical codebook–text alignment module in the TA-VQ framework. The text side begins from a long text $N$2 generated by a VLM and splits it into three sets: words $N$3, phrases $N$4, and sentences $N$5. Only visual-relevant tokens are retained at the word level, specifically nouns, adjectives, and quantifiers. All three granularities are encoded with a shared BERT backbone, producing $N$6, $N$7, and $N$8 (Liang et al., 3 Mar 2025).

On the image side, a single CNN encoder $N$9 produces three grid-feature maps,

$c_j\in\mathbb{R}^d$0

with down-sampling factors $c_j\in\mathbb{R}^d$1, $c_j\in\mathbb{R}^d$2, and $c_j\in\mathbb{R}^d$3. Each grid feature $c_j\in\mathbb{R}^d$4 is quantized via a shared codebook $c_j\in\mathbb{R}^d$5 by

$c_j\in\mathbb{R}^d$6

This produces code sequences $c_j\in\mathbb{R}^d$7 for each granularity (Liang et al., 3 Mar 2025).

The alignment mechanism is not cross-attention. At each granularity $c_j\in\mathbb{R}^d$8, the module aligns $c_j\in\mathbb{R}^d$9 with $L_2$0 through a sampling-based optimal-transport module. The code distribution is modeled as Gaussian. For sentences, the paper defines

$L_2$1

The text distribution is discrete uniform,

$L_2$2

The procedure then samples $L_2$3 text embeddings from $L_2$4, samples $L_2$5 code proposals $L_2$6, projects these proposals by another FNN, and computes a Sinkhorn-Knopp approximation of the $L_2$7 OT cost: $L_2$8 The same procedure yields $L_2$9 and $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$0 (Liang et al., 3 Mar 2025).

Unlike the QAT-oriented variant above, this module is embedded in a standard VQ-VAE-style objective. The reconstruction and commitment losses are

$k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$1

and the overall loss is

$k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$2

where typically $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$3 (Liang et al., 3 Mar 2025).

Integration into base VQ models is described concretely. The single-scale encoder is replaced with a hierarchical one producing three intermediate feature maps; the original VQ-VAE or VQ-GAN decoder remains unchanged and uses only the deepest codes $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$4 for reconstruction; and three lightweight 2-layer MLPs are added for Gaussian parameterization and code-to-feature mapping. Reported hyperparameters include codebook size $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$5, latent dimension $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$6, down-sample factors $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$7, and Sinkhorn sample size $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$8 such as $k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N,$9 or $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$0 if smaller (Liang et al., 3 Mar 2025).

The empirical evidence reported for TA-VQ is explicitly multimodal. Reconstruction is evaluated on CelebA-HQ, CUB-200, and MS-COCO using FID; downstream tasks include unconditional and conditional generation, Text-to-Image, semantic synthesis, image completion, visual grounding, image captioning, and VQA. The paper states that TA-VQ consistently lowers FID in reconstruction versus VQ-GAN, LG-VQ and VQCT; that sentence alignment $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$1 alone gives the largest FID drop, while combining all three losses is best; that multi-granularity text encoding improves FID from $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$2 on CUB; that cosine(codebook, text) increases from $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$3; and that sampling-based alignment reduces train-time per epoch by $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$4 versus full OT (Liang et al., 3 Mar 2025).

4. Calibration-aware alignment in multi-agent VQA

In "Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQA" (Pandey et al., 14 Nov 2025), VQ-CAlign is not a codebook-learning block for images or weights. It is the calibration-alignment component of AlignVQA, a two-stage debate framework for Visual Question Answering.

Stage 1 instantiates $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$5 specialized VQA agents, each built on a different vision-language backbone and prompted with a distinct reasoning strategy, including Chain-of-Thought, Self-Ask, Search-Augmented, and GENREAD. Each agent produces $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$6, where $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$7 is its confidence, computed as the geometric mean of its token-level next-token probabilities. Lexically different but semantically equivalent answers are clustered into $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$8 unique stances $$s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}.$$9. For each stance $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$0, the framework records the frequency $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$1 and mean confidence $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$2. Stage 2 instantiates $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$3 generalist agents, all using the same backbone, each randomly assigned one of the $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$4 stances with probability proportional to $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$5. Each generalist generates supporting and opposing arguments, ratings or feedback on logical consistency, factuality, and clarity via Chain-of-Verification, then a refined answer $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$6 and a new confidence $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$7. Aggregation proceeds by stance; the final answer is the majority-vote stance $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$8, and the final confidence is $p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)},$9 (Pandey et al., 14 Nov 2025).

The calibration component is formalized through bin-based metrics and the AlignCal loss. The paper defines Expected Calibration Error as

$m = \arg\max_j p_j,$0

Adaptive Calibration Error as

$m = \arg\max_j p_j,$1

and Maximum Calibration Error as

$m = \arg\max_j p_j,$2

Letting $m = \arg\max_j p_j,$3 be the softmax output, $m = \arg\max_j p_j,$4, and $m = \arg\max_j p_j,$5 the probability of the true class, the AlignCal loss per example is

$m = \arg\max_j p_j,$6

The total training loss combines Focal Loss and AlignCal: $m = \arg\max_j p_j,$7 The paper further argues that this objective minimizes an upper bound on calibration error by connecting it to an Upper-Bound Calibration Error expression involving $m = \arg\max_j p_j,$8 (Pandey et al., 14 Nov 2025).

The fine-tuning procedure uses LoRA adapters on specialized agents. The implementation details provided are AdamW (fused), LoRA rank $m = \arg\max_j p_j,$9, $z\in\{0,1\}^N$0, dropout $z\in\{0,1\}^N$1 in $z\in\{0,1\}^N$2 and $z\in\{0,1\}^N$3, $z\in\{0,1\}^N$4 epochs on VQARad with $z\in\{0,1\}^N$5 training samples, $z\in\{0,1\}^N$6 epochs on ScienceQA with $z\in\{0,1\}^N$7 samples, batch size $z\in\{0,1\}^N$8, learning rate $z\in\{0,1\}^N$9, $C=\{c_1,\cdots,c_N\}$00, and focal $C=\{c_1,\cdots,c_N\}$01 (Pandey et al., 14 Nov 2025).

The reported results focus on calibration quality. On ScienceQA with $C=\{c_1,\cdots,c_N\}$02 questions, Base Gemma 3 4B has $C=\{c_1,\cdots,c_N\}$03 and $C=\{c_1,\cdots,c_N\}$04; adding Focal Loss reduces ECE to $C=\{c_1,\cdots,c_N\}$05; using FL plus AlignCal in Stage 1 reduces ECE to $C=\{c_1,\cdots,c_N\}$06; agentic debate alone yields ECE $C=\{c_1,\cdots,c_N\}$07; and agentic plus AlignCal yields $C=\{c_1,\cdots,c_N\}$08, $C=\{c_1,\cdots,c_N\}$09, $C=\{c_1,\cdots,c_N\}$10, and $C=\{c_1,\cdots,c_N\}$11. On VQARad, Base Gemma 3 4B has $C=\{c_1,\cdots,c_N\}$12 and $C=\{c_1,\cdots,c_N\}$13; FL plus AlignCal reduces ECE to $C=\{c_1,\cdots,c_N\}$14; agentic debate alone gives $C=\{c_1,\cdots,c_N\}$15, $C=\{c_1,\cdots,c_N\}$16; and agentic plus AlignCal yields $C=\{c_1,\cdots,c_N\}$17, $C=\{c_1,\cdots,c_N\}$18, $C=\{c_1,\cdots,c_N\}$19, and $C=\{c_1,\cdots,c_N\}$20 (Pandey et al., 14 Nov 2025).

5. Divide et Calibra: local multiclass calibration via vector quantization

In "Divide et Calibra: Multiclass Local Calibration via Vector Quantization" (Barbera et al., 20 May 2026), VQ-CAlign denotes a structured local calibration method. The architecture begins with an input $C=\{c_1,\cdots,c_N\}$21, a frozen encoder $C=\{c_1,\cdots,c_N\}$22, and a continuous representation $C=\{c_1,\cdots,c_N\}$23 with $C=\{c_1,\cdots,c_N\}$24. The representation is split into $C=\{c_1,\cdots,c_N\}$25 contiguous subvectors $C=\{c_1,\cdots,c_N\}$26, each assigned to its nearest codeword $C=\{c_1,\cdots,c_N\}$27 from a shared codebook $C=\{c_1,\cdots,c_N\}$28. The assignments $C=\{c_1,\cdots,c_N\}$29 define a quantized vector

$C=\{c_1,\cdots,c_N\}$30

A small linear VQ-head $C=\{c_1,\cdots,c_N\}$31 produces uncalibrated logits $C=\{c_1,\cdots,c_N\}$32 and probabilities $C=\{c_1,\cdots,c_N\}$33 (Barbera et al., 20 May 2026).

The “Calibra” component introduces indexed Dirichlet-based calibration. Two codebooks $C=\{c_1,\cdots,c_N\}$34 and $C=\{c_1,\cdots,c_N\}$35, each of size $C=\{c_1,\cdots,c_N\}$36, provide class-wise factors $C=\{c_1,\cdots,c_N\}$37 and $C=\{c_1,\cdots,c_N\}$38. Using the same vector-quantization indices $C=\{c_1,\cdots,c_N\}$39, the method assembles region-specific matrices $C=\{c_1,\cdots,c_N\}$40 and $C=\{c_1,\cdots,c_N\}$41, then forms the Dirichlet concentration vector

$C=\{c_1,\cdots,c_N\}$42

with $C=\{c_1,\cdots,c_N\}$43 or $C=\{c_1,\cdots,c_N\}$44 to ensure positivity. A more explicit form is given by

$C=\{c_1,\cdots,c_N\}$45

or class-wise,

$C=\{c_1,\cdots,c_N\}$46

This replaces an exponential number of region-specific parameters with a compositional factorization indexed by shared codewords (Barbera et al., 20 May 2026).

The calibrated posterior is then log-linear in $C=\{c_1,\cdots,c_N\}$47: $C=\{c_1,\cdots,c_N\}$48 where

$C=\{c_1,\cdots,c_N\}$49

In practice,

$C=\{c_1,\cdots,c_N\}$50

The paper also notes an equivalent view in which $C=\{c_1,\cdots,c_N\}$51 is $C=\{c_1,\cdots,c_N\}$52, while emphasizing that the log-linear form allows calibration on logits (Barbera et al., 20 May 2026).

Training uses a two-stage procedure. Stage 1 learns the VQ codebook and quantized head with

$C=\{c_1,\cdots,c_N\}$53

where the standard VQ-VAE loss per slot is

$C=\{c_1,\cdots,c_N\}$54

Stage 2 freezes $C=\{c_1,\cdots,c_N\}$55, $C=\{c_1,\cdots,c_N\}$56, and $C=\{c_1,\cdots,c_N\}$57, then learns $C=\{c_1,\cdots,c_N\}$58, $C=\{c_1,\cdots,c_N\}$59, and $C=\{c_1,\cdots,c_N\}$60 by minimizing

$C=\{c_1,\cdots,c_N\}$61

An optional joint objective is also given: $C=\{c_1,\cdots,c_N\}$62 with $C=\{c_1,\cdots,c_N\}$63 and $C=\{c_1,\cdots,c_N\}$64 reported to work well in the experiments (Barbera et al., 20 May 2026).

The reported empirical behavior emphasizes local calibration. For $C=\{c_1,\cdots,c_N\}$65, values in $C=\{c_1,\cdots,c_N\}$66 and $C=\{c_1,\cdots,c_N\}$67 in $C=\{c_1,\cdots,c_N\}$68 are stated as best observed, while small $C=\{c_1,\cdots,c_N\}$69 such as $C=\{c_1,\cdots,c_N\}$70 is too coarse and large $C=\{c_1,\cdots,c_N\}$71 leads to excessive fragmentation. Ablations indicate that VQ alone cuts local calibration error by $C=\{c_1,\cdots,c_N\}$72 versus global methods, and the bilinear indexed parametrization further improves global coherence without sacrificing local gains. On CIFAR-10 with ResNet-50, the paper reports Base $C=\{c_1,\cdots,c_N\}$73, $C=\{c_1,\cdots,c_N\}$74, $C=\{c_1,\cdots,c_N\}$75; Best global (SM) $C=\{c_1,\cdots,c_N\}$76, $C=\{c_1,\cdots,c_N\}$77, $C=\{c_1,\cdots,c_N\}$78; and VQ-CAlign $C=\{c_1,\cdots,c_N\}$79, $C=\{c_1,\cdots,c_N\}$80, $C=\{c_1,\cdots,c_N\}$81. On CIFAR-100, Base $C=\{c_1,\cdots,c_N\}$82 and VQ-CAlign $C=\{c_1,\cdots,c_N\}$83. On TissueMNIST, Base $C=\{c_1,\cdots,c_N\}$84 and VQ-CAlign $C=\{c_1,\cdots,c_N\}$85 (Barbera et al., 20 May 2026).

6. Semantically coupled vector quantization for image synthesis

In "Semantic Image Synthesis with Semantically Coupled VQ-Model" (Alaniz et al., 2022), VQ-CAlign is a semantically coupled VQ-Model that jointly auto-encodes an RGB image $C=\{c_1,\cdots,c_N\}$86 and its semantic map $C=\{c_1,\cdots,c_N\}$87. A shared encoder $C=\{c_1,\cdots,c_N\}$88 takes the concatenated input $C=\{c_1,\cdots,c_N\}$89 and produces two continuous latent feature maps of spatial size $C=\{c_1,\cdots,c_N\}$90 and channel dimension $C=\{c_1,\cdots,c_N\}$91: $C=\{c_1,\cdots,c_N\}$92 Both are quantized against a single shared codebook $C=\{c_1,\cdots,c_N\}$93, forcing the two modalities to pick code-vectors from the same discrete dictionary (Alaniz et al., 2022).

The decoder wiring is asymmetric. The semantic decoder $C=\{c_1,\cdots,c_N\}$94 takes only the quantized semantic latents $C=\{c_1,\cdots,c_N\}$95 to reconstruct the semantic map,

$C=\{c_1,\cdots,c_N\}$96

whereas the image decoder $C=\{c_1,\cdots,c_N\}$97 takes the quantized image latents and a stopped-gradient copy of the semantic latents,

$C=\{c_1,\cdots,c_N\}$98

The stop-gradient is used so that each decoder “owns” its latent, although the encoder sees both modalities (Alaniz et al., 2022).

The paper defines image reconstruction and semantic reconstruction losses as

$C=\{c_1,\cdots,c_N\}$99

with optional GAN and perceptual losses, and

$N$00

The commitment terms are standard VQ terms applied to both modalities, and the total loss is

$N$01

The paper explicitly states that there is no extra explicit coupling regularizer beyond the shared codebook and decoder wiring, so $N$02 (Alaniz et al., 2022).

Training details include codebook size $N$03, embedding dimension $N$04, latent spatial size $N$05, $N$06, $N$07, Adam with learning rate $N$08, $N$09, $N$10, and batch size $N$11 on the unified Cityscapes + ADE20K + COCO-Stuff dataset. After training the VQ-CAlign auto-encoder to convergence, the encoder, decoders, and quantization are frozen, and an autoregressive Transformer is trained on the flattened and concatenated index maps $N$12 and $N$13 to model

$N$14

The reported semantic-image synthesis results compare VQGAN-T against sVQGAN-T. On ADE20K, FID improves from $N$15 to $N$16; on Cityscapes, the values are $N$17 and $N$18 $N$19; and on COCO-Stuff, FID improves from $N$20 to $N$21. The paper further states that semantic adherence and reconstruction metrics also improve relative to separate VQVAE or VQGAN auto-encoders, and that varying $N$22 shows a trade-off between image fidelity and semantic accuracy, with $N$23 giving the best Transformer generation quality (Alaniz et al., 2022).

7. Comparative structure, recurring motifs, and major distinctions

Across these five instantiations, vector quantization serves different technical functions. In the QAT compression setting, VQ-CAlign is a hard codeword-substitution mechanism for grouped weights or activations, with cosine-similarity assignment and STE-based gradient flow (Gou et al., 25 Apr 2026). In TA-VQ, vector quantization produces code sequences that are then aligned with word-, phrase-, and sentence-level text through sampling-based optimal transport (Liang et al., 3 Mar 2025). In Divide et Calibra, VQ partitions the latent space into reusable regions that index local Dirichlet calibration factors (Barbera et al., 20 May 2026). In the semantically coupled VQ-Model, the shared codebook is the coupling device through which image and semantic latents become jointly structured (Alaniz et al., 2022). In AlignVQA, by contrast, the central operation is calibration-aware debate and aggregation rather than codebook quantization of visual representations, even though the same label is used for the module (Pandey et al., 14 Nov 2025).

The treatment of losses is one of the clearest dividing lines. The cosine-assignment QAT variant explicitly avoids additional “VQ reconstruction” or “commitment” terms and relies solely on task loss, optional weight decay, and, in ProxylessNAS, a storage penalty (Gou et al., 25 Apr 2026). TA-VQ retains a standard VQ-VAE reconstruction and commitment loss and adds three alignment losses weighted by $N$24, $N$25, and $N$26 (Liang et al., 3 Mar 2025). Divide et Calibra also retains a standard VQ-VAE loss in Stage 1, then optimizes a calibration negative log-likelihood in Stage 2 (Barbera et al., 20 May 2026). The semantically coupled VQ-Model uses image reconstruction, semantic reconstruction, and commitment losses, while stating that no explicit coupling regularizer is introduced (Alaniz et al., 2022). AlignVQA uses Focal Loss plus the differentiable AlignCal term as a calibration-aware objective for specialized agents (Pandey et al., 14 Nov 2025).

The alignment target also varies sharply. One variant aligns vectors to codewords via cosine similarity (Gou et al., 25 Apr 2026); one aligns codebook-induced image representations to long text at multiple granularities (Liang et al., 3 Mar 2025); one aligns agent confidence with actual correctness (Pandey et al., 14 Nov 2025); one aligns local calibration maps to vector-quantized latent regions (Barbera et al., 20 May 2026); and one aligns semantics and image latents through a shared codebook and decoder conditioning (Alaniz et al., 2022). A plausible implication is that “alignment” in the name is semantically overloaded across these papers: it may refer to assignment geometry, cross-modal correspondence, confidence fidelity, latent-region calibration, or semantic coupling, depending on context.

A second recurring motif is the handling of discreteness. Hard assignments with straight-through estimation appear in the cosine-assignment QAT formulation (Gou et al., 25 Apr 2026). Standard nearest-codeword quantization and stop-gradient are used in TA-VQ, Divide et Calibra, and the semantically coupled VQ-Model (Liang et al., 3 Mar 2025, Barbera et al., 20 May 2026, Alaniz et al., 2022). This commonality does not imply architectural equivalence; rather, it indicates that the discrete bottleneck is reused for different downstream objectives.

Taken together, the literature does not define a monolithic VQ-CAlign module. It defines a family of paper-specific constructs in which vector quantization is combined with alignment or calibration mechanisms tailored to the target task. Any interpretation of “VQ-CAlign” therefore depends on the paper context, the loss design, and the role assigned to the codebook in the overall system.