--- title: 'VQ-CAlign Module: Diverse Quantization Alignment' url: https://www.emergentmind.com/topics/vq-calign-module type: topic --- # VQ-CAlign Module: Diverse Quantization Alignment Searching arXiv for the cited papers and the term "VQ-CAlign" to ground the article in the current literature. arXiv query: VQ-CAlign module 2604.23172 2503.01261 2511.11169 2605.21060 2209.02536 “VQ-CAlign” is not a single standardized module in the arXiv literature summarized here. Instead, the label is used for several paper-specific mechanisms that combine vector quantization with alignment, calibration, or coupling objectives in distinct problem settings: quantization-aware training and model compression, text-aligned codebook learning, confidence calibration in Visual Question Answering, multiclass local calibration, and semantically coupled image synthesis [2604.23172] [2503.01261] [2511.11169] [2605.21060] [2209.02536]. This suggests that the term functions as a local architectural name rather than a universally fixed design. Across these uses, the common substrate is discrete representation learning through codebooks, but the surrounding objectives, losses, and inference semantics differ substantially. ## 1. Terminological scope and paper-specific meanings In the sources considered here, “VQ-CAlign” denotes at least five distinct modules. | Paper | Domain | Core role of “VQ-CAlign” | |---|---|---| | "Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks" [2604.23172] | model weight compression | cosine-similarity–based vector-quantization alignment | | "Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text" [2503.01261] | image quantization and cross-modal learning | hierarchical codebook–text alignment | | "Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQA" [2511.11169] | VQA calibration | calibration-alignment component | | "Divide et Calibra: Multiclass Local Calibration via Vector Quantization" [2605.21060] | multiclass calibration | local calibration via vector quantization | | "Semantic Image Synthesis with Semantically Coupled VQ-Model" [2209.02536] | semantic image synthesis | semantically coupled VQ-Model | The heterogeneity is substantive rather than merely terminological. In one case, the module performs hard codeword selection for quantized weights using cosine similarity and a straight-through estimator; in another, it aligns image code sequences with words, phrases, and sentences using sampling-based optimal transport; in another, it is a calibration-aware fine-tuning and aggregation mechanism for VQA agents; in another, it constructs region-specific Dirichlet calibration maps from vector-quantized latent regions; and in another, it couples semantic and image latents through a shared codebook [2604.23172] [2503.01261] [2511.11169] [2605.21060] [2209.02536]. A common misconception would be to treat “VQ-CAlign” as a single canonical block with a fixed loss or quantizer. The literature summarized here does not support that interpretation. Some variants use no additional VQ-specific losses, some retain standard VQ-VAE commitment losses, and some are not primarily codebook-learning modules at all, but calibration mechanisms layered on top of VLM or classifier outputs [2604.23172] [2503.01261] [2511.11169]. ## 2. Cosine-similarity assignment in quantization-aware training In "Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networks" [2604.23172], VQ-CAlign is a cosine-similarity–based vector-quantization alignment module for model weight compression. Let \(x\in\mathbb{R}^d\) be an input vector and let \(C=\{c_1,\cdots,c_N\}\) be a learnable codebook of \(N\) entries \(c_j\in\mathbb{R}^d\). The module first forms \(L_2\)-normalized keys \[ k_j = \frac{c_j}{\|c_j\|_2}, \qquad j=1\ldots N, \] then computes cosine similarities \[ s_j = \mathrm{cos\_sim}(x,c_j) = x^\top k_j = \frac{x^\top c_j}{(\|x\|_2 \|c_j\|_2)}. \] Optionally, these similarities are converted into a soft assignment \[ p_j = \frac{\exp(s_j)}{\sum_{i=1}^N \exp(s_i)}, \] with the stated caveat that this distribution is used only in the backward pass [2604.23172]. The forward pass is explicitly hard. The module picks the index \[ m = \arg\max_j p_j, \] forms a one-hot vector \(z\in\{0,1\}^N\), and outputs \[ \hat x = \sum_{j=1}^N z_j c_j = c_m. \] Because \(z\) is discrete, the method applies the standard straight-through estimator. During back-propagation it pretends \(z=p\), so that \[ \frac{\partial z_j}{\partial p_k} \approx \delta_{jk}, \qquad \frac{\partial \mathcal{L}}{\partial p_j} \approx \frac{\partial \mathcal{L}}{\partial z_j}. \] Since \(\hat x=z^\top C\), one obtains \[ \frac{\partial \mathcal{L}}{\partial z_j} = \left(\frac{\partial \mathcal{L}}{\partial \hat x}\right)^\top c_j, \qquad \frac{\partial \mathcal{L}}{\partial p_j} \approx \left(\frac{\partial \mathcal{L}}{\partial \hat x}\right)^\top c_j. \] The reconstruction step is therefore simply \[ \hat x = Q(x;C) = c_{\arg\max_j (x^\top k_j)}. \] When inserted into a network layer, the forward pass substitutes each block of \(d\) floating-point parameters or activations by its single corresponding codeword before further computation [2604.23172]. A defining feature of this variant is the absence of additional VQ-specific losses. Unlike many VQ-VAE or DKM schemes, it does not introduce extra reconstruction, commitment-, or codebook-update losses. Instead, it relies solely on the task loss, may optionally add standard weight decay, and in mixed-quantization with ProxylessNAS adds a storage penalty. The overall QAT loss in the ProxylessNAS setting is \[ L = L_{CE}(W;\mathrm{data}) + \lambda \|W\|_2^2 + \beta \cdot E[\mathrm{storage}], \] with \[ E[\mathrm{storage}] = p\cdot (N/L)\cdot Q_{vq} + (1-p)\cdot (N)\cdot Q_{lq}. \] The paper attributes the absence of extra VQ losses to two stated mechanisms: codebook vectors receive direct gradient signals via STE, and cosine assignment balances codebook usage, mitigating collapse [2604.23172]. The same work situates VQ-CAlign inside each quantized layer during QAT. Full-precision weights are grouped into \(d\)-dimensional vectors, each passed through the VQ-CAlign block to produce \(\hat x\), and gradients update both the original vectors and the codebook entries. The authors further state that cosine-similarity assignment emphasizes directional diversity of codewords and naturally yields a more uniform assignment of vectors to centroids, directly reducing the chance of “dead” codewords without explicit eviction or regularization. Because forward and backward graphs are identical to inference, no temperature scheduling is required. In a larger mixed-quantization pipeline, VQ-CAlign competes against a conventional linear quantizer, with per-layer switches learned via another STE [2604.23172]. ## 3. Hierarchical codebook–text alignment with long text In "Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text" [2503.01261], VQ-CAlign is a hierarchical codebook–text alignment module in the TA-VQ framework. The text side begins from a long text \(t\) generated by a VLM and splits it into three sets: words \(t_w\), phrases \(t_p\), and sentences \(t_s\). Only visual-relevant tokens are retained at the word level, specifically nouns, adjectives, and quantifiers. All three granularities are encoded with a shared BERT backbone, producing \(E_t(w_i)\), \(E_t(p_i)\), and \(E_t(s_i)\) [2503.01261]. On the image side, a single CNN encoder \(E_{\theta_e}\) produces three grid-feature maps, \[ \hat Z_{f_1}, \hat Z_{f_2}, \hat Z_{f_3} \in \mathbb{R}^{H/f_j \times W/f_j \times d_z}, \] with down-sampling factors \(f_1=4\), \(f_2=8\), and \(f_3=16\). Each grid feature \(\hat z_i\) is quantized via a shared codebook \(C=\{e_1,\ldots,e_K\}\) by \[ z_i = \arg\min_{e_k}\|\hat z_i - e_k\|_2. \] This produces code sequences \(Z_{f_j}\) for each granularity [2503.01261]. The alignment mechanism is not cross-attention. At each granularity \(j\in\{w,p,s\}\), the module aligns \(Z_{f_j}\) with \(t_j\) through a sampling-based optimal-transport module. The code distribution is modeled as Gaussian. For sentences, the paper defines \[ m_{f_3} = \mathrm{Mean}(Z_{f_3}),\quad \mu_{f_3} = \mathrm{FNN}_u(m_{f_3}),\quad \Sigma_{f_3} = \mathrm{diag}\bigl(\exp(\mathrm{FNN}_\sigma(m_{f_3}))\bigr). \] The text distribution is discrete uniform, \[ \mathcal P_{t_s} = \frac1{|t_s|}\sum_{s\in t_s}\delta_s. \] The procedure then samples \(q\) text embeddings from \(\mathcal P_{t_s}\), samples \(q\) code proposals \(\xi_i\sim\mathcal N(\mu_{f_3},\Sigma_{f_3})\), projects these proposals by another FNN, and computes a Sinkhorn-Knopp approximation of the \(q\times q\) OT cost: \[ \mathcal L_{t_s} = \mathcal W\bigl(\{y_i^{\mathrm{pre}}\}, \{y_j^{\mathrm{tar}}\}\bigr). \] The same procedure yields \(\mathcal L_{t_p}\) and \(\mathcal L_{t_w}\) [2503.01261]. Unlike the QAT-oriented variant above, this module is embedded in a standard VQ-VAE-style objective. The reconstruction and commitment losses are \[ \mathcal L_{vq} = \|x-\widetilde x\|_2^2 + \|\mathrm{sg}[E_{\theta_e}(x)]-Z\|_2^2 + \beta \|E_{\theta_e}(x)-\mathrm{sg}[Z]\|_2^2, \] and the overall loss is \[ \mathcal L = \mathcal L_{vq} + \alpha\,\mathcal L_{t_w} + \beta\,\mathcal L_{t_p} + \gamma\,\mathcal L_{t_s}, \] where typically \(\alpha=\beta=\gamma=10^{-3}\) [2503.01261]. Integration into base VQ models is described concretely. The single-scale encoder is replaced with a hierarchical one producing three intermediate feature maps; the original VQ-VAE or VQ-GAN decoder remains unchanged and uses only the deepest codes \(Z_{f_3}\) for reconstruction; and three lightweight 2-layer MLPs are added for Gaussian parameterization and code-to-feature mapping. Reported hyperparameters include codebook size \(K=1024\), latent dimension \(d_z=256\), down-sample factors \(4,8,16\), and Sinkhorn sample size \(q\) such as \(32\) or \(|t_j|\) if smaller [2503.01261]. The empirical evidence reported for TA-VQ is explicitly multimodal. Reconstruction is evaluated on CelebA-HQ, CUB-200, and MS-COCO using FID; downstream tasks include unconditional and conditional generation, Text-to-Image, semantic synthesis, image completion, visual grounding, image captioning, and VQA. The paper states that TA-VQ consistently lowers FID in reconstruction versus VQ-GAN, LG-VQ and VQCT; that sentence alignment \(\mathcal L_s\) alone gives the largest FID drop, while combining all three losses is best; that multi-granularity text encoding improves FID from \(4.91\to 4.60\) on CUB; that cosine(codebook, text) increases from \(0.1021\to 0.1444\); and that sampling-based alignment reduces train-time per epoch by \(\sim7\%\) versus full OT [2503.01261]. ## 4. Calibration-aware alignment in multi-agent VQA In "Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQA" [2511.11169], VQ-CAlign is not a codebook-learning block for images or weights. It is the calibration-alignment component of AlignVQA, a two-stage debate framework for Visual Question Answering. Stage 1 instantiates \(N\) specialized VQA agents, each built on a different vision-language backbone and prompted with a distinct reasoning strategy, including Chain-of-Thought, Self-Ask, Search-Augmented, and GENREAD. Each agent produces \((\hat y_i,p_i)\), where \(p_i\) is its confidence, computed as the geometric mean of its token-level next-token probabilities. Lexically different but semantically equivalent answers are clustered into \(K\) unique stances \(s_1,\ldots,s_K\). For each stance \(s_k\), the framework records the frequency \(f_k\) and mean confidence \(\bar c_k\). Stage 2 instantiates \(M\) generalist agents, all using the same backbone, each randomly assigned one of the \(K\) stances with probability proportional to \(f_k\). Each generalist generates supporting and opposing arguments, ratings or feedback on logical consistency, factuality, and clarity via Chain-of-Verification, then a refined answer \(y'_j\) and a new confidence \(\mathrm{Conf}(y'_j)\). Aggregation proceeds by stance; the final answer is the majority-vote stance \(s^*\), and the final confidence is \(\hat c_{s^*}\) [2511.11169]. The calibration component is formalized through bin-based metrics and the AlignCal loss. The paper defines Expected Calibration Error as \[ \mathrm{ECE} = \sum_{m=1}^M \frac{|B_m|}{n}\bigl|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\bigr|, \] Adaptive Calibration Error as \[ \mathrm{ACE} = \frac{1}{B}\sum_{b=1}^B \bigl|\mathrm{acc}(B_b)-\mathrm{conf}(B_b)\bigr|, \] and Maximum Calibration Error as \[ \mathrm{MCE} = \max_{b=1,\dots,B}\bigl|\mathrm{acc}(B_b)-\mathrm{conf}(B_b)\bigr|. \] Letting \(\mathbf p=(p_1,\dots,p_K)\) be the softmax output, \(p_{\max}=\max_i p_i\), and \(p_y\) the probability of the true class, the AlignCal loss per example is \[ \mathcal L_{\mathrm{AlignCal}}(p_y,p_{\max}) = p_y(1-p_{\max}) + (1-p_y)p_{\max}. \] The total training loss combines Focal Loss and AlignCal: \[ \mathcal L_{\mathrm{total}} = \mathcal L_{\mathrm{FL}} + \lambda\,\mathcal L_{\mathrm{AlignCal}}. \] The paper further argues that this objective minimizes an upper bound on calibration error by connecting it to an Upper-Bound Calibration Error expression involving \(t=\mathbf 1\{\hat y=y\}\) [2511.11169]. The fine-tuning procedure uses LoRA adapters on specialized agents. The implementation details provided are AdamW (fused), LoRA rank \(8\), \(\alpha=8\), dropout \(0.05\) in \(q\_proj\) and \(v\_proj\), \(6\) epochs on VQARad with \(1\,793\) training samples, \(10\) epochs on ScienceQA with \(12\,000\) samples, batch size \(2\), learning rate \(2\times 10^{-4}\), \(\lambda=2\), and focal \(\gamma=2\) [2511.11169]. The reported results focus on calibration quality. On ScienceQA with \(21\,208\) questions, Base Gemma 3 4B has \(\mathrm{Acc}=71.0\%\) and \(\mathrm{ECE}=0.398\); adding Focal Loss reduces ECE to \(0.302\); using FL plus AlignCal in Stage 1 reduces ECE to \(0.058\); agentic debate alone yields ECE \(0.270\); and agentic plus AlignCal yields \(\mathrm{ECE}=0.055\), \(\mathrm{ACE}=0.110\), \(\mathrm{MCE}=0.331\), and \(\mathrm{Acc}=76.1\%\). On VQARad, Base Gemma 3 4B has \(\mathrm{Acc}=59.4\%\) and \(\mathrm{ECE}=0.375\); FL plus AlignCal reduces ECE to \(0.137\); agentic debate alone gives \(\mathrm{ECE}=0.146\), \(\mathrm{ACE}=0.144\); and agentic plus AlignCal yields \(\mathrm{ECE}=0.098\), \(\mathrm{ACE}=0.095\), \(\mathrm{MCE}=0.267\), and \(\mathrm{Acc}=68.2\%\) [2511.11169]. ## 5. Divide et Calibra: local multiclass calibration via vector quantization In "Divide et Calibra: Multiclass Local Calibration via Vector Quantization" [2605.21060], VQ-CAlign denotes a structured local calibration method. The architecture begins with an input \(x\in\mathbb{R}^m\), a frozen encoder \(E\), and a continuous representation \(z\in\mathbb{R}^{d\cdot w}\) with \(D=d\cdot w\). The representation is split into \(w\) contiguous subvectors \(z^{(i)}\in\mathbb{R}^d\), each assigned to its nearest codeword \(c_{s(i)}\) from a shared codebook \(C=\{c_1,\ldots,c_K\}\subset\mathbb{R}^d\). The assignments \(s=(s(1),\ldots,s(w))\) define a quantized vector \[ q = [c_{s(1)},\ldots,c_{s(w)}]\in\mathbb{R}^D. \] A small linear VQ-head \(f_{VQ}:\mathbb{R}^D\to\mathbb{R}^K\) produces uncalibrated logits \(\ell\) and probabilities \(\hat p=\mathrm{softmax}(\ell)\) [2605.21060]. The “Calibra” component introduces indexed Dirichlet-based calibration. Two codebooks \(A\) and \(B\), each of size \(K\), provide class-wise factors \(A_k\in\mathbb{R}^{|Y|}\) and \(B_k\in\mathbb{R}^{|Y|}\). Using the same vector-quantization indices \(s(i)\), the method assembles region-specific matrices \(A^{(V)}\in\mathbb{R}^{w\times |Y|}\) and \(B^{(V)}\in\mathbb{R}^{w\times |Y|}\), then forms the Dirichlet concentration vector \[ \alpha^{(V)} = \phi\!\left(A^{(V)\top}\mathrm{diag}(\sigma^2)B^{(V)}\right)\in\mathbb{R}^{|Y|}, \] with \(\phi=\mathrm{softplus}\) or \(\exp\) to ensure positivity. A more explicit form is given by \[ H^{(V)} = A^{(V)\top}\mathrm{diag}(\sigma^2)B^{(V)} \in \mathbb{R}^{|Y|\times |Y|},\qquad \alpha^{(V)} = \phi(\mathrm{diag}(H^{(V)})), \] or class-wise, \[ \alpha_j^{(V)} = \phi\!\left(\sum_{i=1}^w \sigma_i^2 A_{s(i),j}B_{s(i),j}\right). \] This replaces an exponential number of region-specific parameters with a compositional factorization indexed by shared codewords [2605.21060]. The calibrated posterior is then log-linear in \(\log \hat p\): \[ \log \tilde p(y=j\mid \hat p,V) = b_{j,V} + w_{j,V}^\top \log \hat p + \mathrm{const}, \] where \[ w_{j,V} = \alpha_j^{(V)} - 1,\qquad b_{j,V} = \log \pi_{j\mid V} - \log B(\alpha^{(V)}). \] In practice, \[ \tilde p = \mathrm{softmax}([b_V] + [W_V]\log \hat p). \] The paper also notes an equivalent view in which \(\tilde p\) is \(E_{q\sim \mathrm{Dir}(\alpha^{(V)})}[q] = \alpha^{(V)}/\sum_j \alpha_j^{(V)}\), while emphasizing that the log-linear form allows calibration on logits [2605.21060]. Training uses a two-stage procedure. Stage 1 learns the VQ codebook and quantized head with \[ L_{\mathrm{stage1}} = L_{\mathrm{cls}}(f_{VQ}(q),y) + \gamma L_{vq}(z,q), \] where the standard VQ-VAE loss per slot is \[ L_{vq} = \|\mathrm{sg}[z^{(i)}]-c_{k^*}\|^2 + \beta \|z^{(i)}-\mathrm{sg}[c_{k^*}]\|^2. \] Stage 2 freezes \(E\), \(C\), and \(f_{VQ}\), then learns \(A\), \(B\), and \(\sigma^2\) by minimizing \[ L_{\mathrm{cal}} = -\sum_{(x,y)\in D_{\mathrm{cal}}}\log \tilde p(y\mid \hat p(x),V(x)). \] An optional joint objective is also given: \[ L_{\mathrm{total}} = L_{\mathrm{cls}} + \gamma L_{vq} + \lambda L_{\mathrm{cal}}, \] with \(\gamma=10^{-3}\) and \(\lambda=10^{-3}\) reported to work well in the experiments [2605.21060]. The reported empirical behavior emphasizes local calibration. For \(w\), values in \(\{32,64\}\) and \(K\) in \(\{32,64\}\) are stated as best observed, while small \(w\) such as \(16\) is too coarse and large \(w\ge 128\) leads to excessive fragmentation. Ablations indicate that VQ alone cuts local calibration error by \(\sim30\%\) versus global methods, and the bilinear indexed parametrization further improves global coherence without sacrificing local gains. On CIFAR-10 with ResNet-50, the paper reports Base \( \mathrm{LCE}\approx 0.0154\), \( \mathrm{MLCE}\approx 0.873\), \( \mathrm{ECCE}\approx 0.0066\); Best global (SM) \( \mathrm{LCE}\approx 0.0102\), \( \mathrm{MLCE}\approx 0.803\), \( \mathrm{ECCE}\approx 0.0008\); and VQ-CAlign \( \mathrm{LCE}\approx 0.0059\), \( \mathrm{MLCE}\approx 0.560\), \( \mathrm{ECCE}\approx 0.0013\). On CIFAR-100, Base \( \mathrm{LCE}\approx 0.0032\) and VQ-CAlign \( \mathrm{LCE}\approx 0.0017\). On TissueMNIST, Base \( \mathrm{LCE}\approx 0.0739\) and VQ-CAlign \( \mathrm{LCE}\approx 0.0083\) [2605.21060]. ## 6. Semantically coupled vector quantization for image synthesis In "Semantic Image Synthesis with Semantically Coupled VQ-Model" [2209.02536], VQ-CAlign is a semantically coupled VQ-Model that jointly auto-encodes an RGB image \(x\in\mathbb{R}^{H\times W\times 3}\) and its semantic map \(s\in\mathbb{R}^{H\times W\times C}\). A shared encoder \(f_{\mathrm{Enc}}\) takes the concatenated input \([x;s]\in\mathbb{R}^{H\times W\times (3+C)}\) and produces two continuous latent feature maps of spatial size \(h\times w\) and channel dimension \(D\): \[ e_{\mathrm{img}} = f_{\mathrm{Enc\_img}}([x;s]) \in \mathbb{R}^{h\times w\times D},\qquad e_{\mathrm{sem}} = f_{\mathrm{Enc\_sem}}([x;s]) \in \mathbb{R}^{h\times w\times D}. \] Both are quantized against a single shared codebook \(C=\{e_k\}_{k=1}^K\subset\mathbb{R}^D\), forcing the two modalities to pick code-vectors from the same discrete dictionary [2209.02536]. The decoder wiring is asymmetric. The semantic decoder \(f_{\mathrm{Dec\_sem}}\) takes only the quantized semantic latents \(z_q^{\mathrm{sem}}\) to reconstruct the semantic map, \[ \hat S = f_{\mathrm{Dec\_sem}}(z_q^{\mathrm{sem}}), \] whereas the image decoder \(f_{\mathrm{Dec\_img}}\) takes the quantized image latents and a stopped-gradient copy of the semantic latents, \[ \hat X = f_{\mathrm{Dec\_img}}(z_q^{\mathrm{img}},\mathrm{sg}(z_q^{\mathrm{sem}})). \] The stop-gradient is used so that each decoder “owns” its latent, although the encoder sees both modalities [2209.02536]. The paper defines image reconstruction and semantic reconstruction losses as \[ L_{\mathrm{recon\_img}} = \|x_{\mathrm{img}} - f_{\mathrm{Dec\_img}}(z_q^{\mathrm{img}},\mathrm{sg}(z_q^{\mathrm{sem}}))\|_1 \] with optional GAN and perceptual losses, and \[ L_{\mathrm{recon\_sem}} = \mathrm{CrossEntropy}(x_{\mathrm{sem}}, f_{\mathrm{Dec\_sem}}(z_q^{\mathrm{sem}})). \] The commitment terms are standard VQ terms applied to both modalities, and the total loss is \[ L_{\mathrm{VQ\text{-}CAlign}} = L_{\mathrm{recon\_img}} + \lambda L_{\mathrm{recon\_sem}} + \beta \sum_i \Bigl( \|\mathrm{sg}[z_e^{\mathrm{img}}(i)] - z_q^{\mathrm{img}}(i)\|_2^2 + \|z_e^{\mathrm{img}}(i) - \mathrm{sg}[z_q^{\mathrm{img}}(i)]\|_2^2 + \|\mathrm{sg}[z_e^{\mathrm{sem}}(i)] - z_q^{\mathrm{sem}}(i)\|_2^2 + \|z_e^{\mathrm{sem}}(i) - \mathrm{sg}[z_q^{\mathrm{sem}}(i)]\|_2^2 \Bigr). \] The paper explicitly states that there is no extra explicit coupling regularizer beyond the shared codebook and decoder wiring, so \(L_{\mathrm{couple}}=0\) [2209.02536]. Training details include codebook size \(K=16\,384\), embedding dimension \(D=256\), latent spatial size \(h=w=16\), \(\lambda=0.1\), \(\beta=0.25\), Adam with learning rate \(2\times 10^{-4}\), \(\beta_1=0.9\), \(\beta_2=0.999\), and batch size \(32\) on the unified Cityscapes + ADE20K + COCO-Stuff dataset. After training the VQ-CAlign auto-encoder to convergence, the encoder, decoders, and quantization are frozen, and an autoregressive Transformer is trained on the flattened and concatenated index maps \(z_q^{\mathrm{sem}}\) and \(z_q^{\mathrm{img}}\) to model \[ p(z_q^{\mathrm{img}} \mid z_q^{\mathrm{sem}}) = \prod_{t=1}^{h\cdot w} p(z_q^{\mathrm{img}}(t) \mid z_q^{\mathrm{sem}}, z_q^{\mathrm{img}}(