---
title: Jointly Optimized Codebook Construction (JOCC)
url: https://www.emergentmind.com/topics/jointly-optimized-codebook-construction-jocc
type: topic
---

# Jointly Optimized Codebook Construction (JOCC)

Jointly Optimized Codebook Construction (JOCC) denotes a family of formulations in which a codebook is optimized together with the mappings, inference modules, or transmission mechanisms that use it. In the cited literature, the term spans codebook-enabled quantization for digital semantic communications, site-specific limited-feedback beamforming, memory-footprint compression through jointly learnable codebooks and mappings, second-order representation learning with codebook-conditioned factorization, code-domain NOMA autoencoders, Bayes-decoded error-correction code design, and joint BS/RIS beam codebooks for near-field beam training [2510.07108] [2604.14524] [2309.17361] [1906.01972] [2104.00818] [1805.07429] [2508.18582]. The common feature is that the codebook is not fixed a priori: it is part of a system-level optimization objective such as mutual information, reconstruction fidelity, CSI-capture efficiency, BER, Bayes risk, or beam-pattern matching.

## 1. Terminological scope and definitional core

In current arXiv usage, JOCC is not tied to a single canonical architecture. The 2025 semantic-communications paper studies a “theoretically-grounded codebook” through joint optimization of quantization efficiency, transmission efficiency, and robust performance; the 2026 beamforming paper uses JOCC for the coupled design of a probing codebook and a subspace-inference network; the 2023 compression paper presents the same general idea under the name “jointly learnable codebooks and mappings” (JLCM); and the 2019 retrieval paper presents a related formulation as “Joint Optimization of Codebook and Factorization” (JCF) [2510.07108] [2604.14524] [2309.17361] [1906.01972]. This suggests that the acronym is best understood as a co-design principle rather than a standardized single algorithm.

The term “codebook” also changes meaning with application. In vector-quantized semantic communication it is a discrete latent vocabulary or constellation lookup; in model compression it is a set of quantization centroids assigned to weights; in beamforming it is a probing beam set or a set of BS/RIS beam codewords; and in error-correction coding it is a mapping from source symbols to binary codewords [2603.02536] [2508.18582] [1805.07429].

| Setting | Jointly optimized objects | Representative objective |
|---|---|---|
| Digital semantic communications | codebook, encoder/decoder, channel-aware loss | $L_{\mathrm{total}} = L_{\mathrm{task}} + L_Q + \lambda L_{\mathrm{dist}}$ |
| Site-specific limited-feedback beamforming | probing codebook $B$, inference network $\Psi_\Theta$ | $\{B^*,\Theta^*\} = \arg\max_{B,\Theta} E_{h\sim p_s}[\eta_p(h;B,\Psi_\Theta)]$ |
| Model compression | multiple codebooks and index-score tensor $I$ | $\mathcal L(C,I)=\|\widetilde f(\tilde X)-f(X)\|_2^2+\mathcal L_1+\lambda\mathcal L_2$ |
| XL-RIS beam training | BS codeword $w_t$, RIS phases $\bm\phi$, phase-adjust $\psi$ | $\min \|\bm\phi^H\widetilde{\mathbf H}(w_t)-(p\odot\psi)^T\|_2^2$ |

## 2. Shared mathematical structure

A recurrent JOCC pattern is discrete representation induced by nearest-neighbor partitions or softened assignments. In the semantic-communications derivation, a learnable codebook $C=\{c_1,\dots,c_K\}\subset\mathbb R^n$ defines Voronoi cells
$$
V_k = \{ s \in \mathbb R^n : \|s - c_k\|_2 \le \|s - c_j\|_2, \forall j\neq k \},
$$
and the quantizer $Q:\mathbb R^n\to\{1,\dots,K\}$ is
$$
Q(s)=k \quad \text{if and only if} \quad s \in V_k.
$$
The same paper treats the continuous semantic feature $S\sim p_S(s)$ and discrete index $Z=Q(S)$, derives
$$
I(S;Z)=H(Z),
$$
and uses the empirical entropy
$$
\hat H(Z)= -\sum_{k=1}^K \hat \pi_k \log \hat \pi_k
$$
to regularize codeword utilization through
$$
\hat L_Q = \frac{1}{N} \sum_{i=1}^N \|s_i - c_{Q(s_i)}\|^2 - \beta \hat H(Z)
$$
or an equivalent cross-entropy form [2510.07108].

Other JOCC formulations retain the same discrete core but modify the assignment mechanism. In compression, one learns an index-score tensor $I\in\mathbb R^{n_o\times n_i\times r}$, defines soft assignments
$$
p_{i,j}^{(\ell)}=\mathrm{softmax}(I_{i,j,0},\dots,I_{i,j,r-1})_\ell,
$$
and constructs
$$
\widetilde W_{i,j}=\sum_{\ell=0}^{r-1} p_{i,j}^{(\ell)}\, C_{\lfloor \sigma_i\frac{k}{n_o}\rfloor}[\ell].
$$
The per-layer objective combines a distillation loss, a weight-reconstruction term, and a one-hot regularizer,
$$
\mathcal L(C,I)=\|\widetilde f(\tilde X)-f(X)\|_2^2+\|Cp(I)-W\|_2^2+\lambda\sum_{i,j}\Bigl[1-|2p_{i,j}-1|^\beta\Bigr].
$$
A custom gradient update replaces the standard $\partial p/\partial I$ with
$$
D_{i,j,\ell}
=
\frac{\mathrm{sign}(C_{k^*}-C_{\ell})}
{1 + |C_{k^*}-C_{\ell}|}
-
[I_{i,j,\ell}==1],
$$
to enforce a proximal search of codebooks and mappings [2309.17361].

In second-order representation learning, soft assignments to a codebook $C=\{c_1,\dots,c_N\}$ are
$$
a_{i,k}=\frac{\exp(\tau\langle c_k,x_i\rangle)}{\sum_{j=1}^N \exp(\tau\langle c_j,x_i\rangle)},
$$
and the codebook is integrated directly into bilinear pooling and low-rank factorization. In the compact JCF form,
$$
z_i=\sum_i \langle U_i a_i,x_i\rangle \,\langle V_i a_i,x_i\rangle,
$$
with end-to-end optimization under the N-pair loss plus weight decay [1906.01972].

A further generalization appears in ESC-MVQ, where JOCC means jointly training multiple codebooks $\{\mathcal C_k\}_{k=1}^K$ and trainable bit-flip probabilities $\{p_k\}$ under a parallel-BSC model,
$$
P(y\mid s)
=
\prod_{i=1}^N\prod_{b=1}^B
p_{k_i}^{\mathbf 1[y_{i,b}\neq s_{i,b}]}
(1-p_{k_i})^{1-\mathbf 1[y_{i,b}\neq s_{i,b}]},
$$
together with an end-to-end reconstruction loss and codebook regularization [2504.11709].

## 3. Semantic communications and channel-aware JOCC

Digital semantic communication is the area in which JOCC is most explicitly tied to quantization, mutual information, and channel robustness. The 2025 theoretically grounded formulation establishes a formal equivalence between semantic synonymy and Voronoi-based many-to-one quantization, derives the mutual information objective $I(S;Z)=H(Z)$, introduces entropy-regularized end-to-end codebook training, and models channel-induced semantic distortion under bit-flip errors through
$$
d(k,\ell)=\|c_k-c_\ell\|_2^2,
\qquad
P_e=1-(1-p_b)^L,
$$
leading to
$$
L_{\mathrm{dist}} = P_e \sum_k \pi_k \bar\Delta_k^2.
$$
Its full joint objective is
$$
L_{\mathrm{total}} = L_{\mathrm{task}} + L_Q + \lambda L_{\mathrm{dist}},
$$
implemented with a VQ-VAE backbone, straight-through estimation, Adam, learning rate $\approx 1\times 10^{-4}$, batch size $\approx 64$, $\gamma=0.1$ for the entropy term, $\omega=0.1$ for the channel loss, 64-QAM modulation, Rayleigh fading, and codebook size fixed at $K=256$ for the reported results. On image reconstruction tasks at SNR $=10$ dB, the reported improvement is $24.1\%$ in PSNR and $46.5\%$ in LPIPS compared to existing codebook designs; ablations report that “+Index Entropy” recovers balanced $\pi_k$ and improves PSNR at high SNR, while “+Channel-Aware” sharply reduces LPIPS at low SNR by minimizing semantic drift under bit flips [2510.07108].

A distinct two-stage JOCC-style semantic pipeline appears in the Transformer-based generative system. Stage 1 jointly trains semantic encoder $\Theta$, codebook $\mathcal C$, and decoder $\Xi$ with a VQ-VAE-style loss that combines $\mathcal L_{\rm result}$, $\mathcal L_{\rm in}$, and $\mathcal L_{\rm GAN}$:
$$
\mathcal L_{\mathrm{Stage1}}
=
\|I_h-I_{\rm out}\|_1
+\|\phi(I_h)-\phi(I_{\rm out})\|_2^2
+\|\sg(\mathbf z_h)-\mathbf z_c\|_2^2
+\beta\|\mathbf z_h-\sg(\mathbf z_c)\|_2^2
+\mathcal L_{\rm GAN}.
$$
The codebook has size $L=1024$ and dimension $q=256$. Stage 2 freezes encoder, decoder, and codebook, then trains a nine-block Transformer encoder with $d_{\mathrm{model}}=256$, eight heads, feed-forward dimension $1024$, and learned $2$D positional embeddings to recover the correct codebook indices from noisy latent maps. On FFHQ-test, the reported averages are: at $1$ dB, JOCC yields PSNR $14.5$, SSIM $0.51$, and LPIPS $0.57$; at $9$ dB, JOCC yields PSNR $19.8$, SSIM $0.75$, and LPIPS $0.33$ [2402.16868].

The satellite-terrestrial SFSC framework uses JOCC at the interface of semantic coding and digital modulation. Here the semantic encoder $f_{se}(\cdot;\theta_{se})$ and semantic codebook $E\in\mathbb R^{K\times N}$ are jointly optimized, with the codebook acting as both quantizer and constellation lookup. The composite objective is
$$
\mathcal L(\theta)=\mathcal L_{\mathrm{rec}}+\lambda_{\mathrm{trans}}\mathcal L_{\mathrm{trans}}+\lambda_{\mathrm{quant}}\mathcal L_{\mathrm{quant}},
$$
where $\mathcal L_{\mathrm{rec}}$ is end-to-end MSE, $\mathcal L_{\mathrm{trans}}$ is cross-entropy over indices, and $\mathcal L_{\mathrm{quant}}$ is a VQ-VAE style regularizer. The framework further injects instantaneous SNR through FiLM layers,
$$
[\gamma,\beta]=f_g(\mu;\theta_g),\qquad
\mathrm{FiLM}(F_{c,h,w})=\gamma_{c,h,w}F_{c,h,w}+\beta_{c,h,w}.
$$
The reported hyperparameters are $K=8$, embedding dimension $N=256$, batch size $16$, initial learning rate $1\times10^{-4}$ with cosine annealing to $1\times10^{-5}$, $\lambda_{\rm trans}=1.0$, $\lambda_{\rm quant}=0.25$, and $\beta_q=0.25$. Under SL-SNR $=-10$ dB, the reported PSNR is $23.5$ dB versus $15.3$ dB for digital joint coding and modulation, with spectral efficiency $r=1/24$ versus $r=1/16$ for 64-QAM, corresponding to a $33\%$ bandwidth saving; in the MDMA scenario, CS-MDMA with JOCC achieves a $4$–$7$ dB PSNR improvement at $-10$ dB SNR over classical MDMA and NOMA-JSCC [2603.02536].

ESC-MVQ extends the semantic-communication interpretation of JOCC from one codebook to many. It jointly trains multiple VQ codebooks and their associated bit-flip probabilities with a single encoder-decoder pair, then solves an alternating communication-strategy problem over codebook assignment, modulation order, and power allocation. The reported empirical outcome is up to $3$–$5$ dB PSNR gain over single-codebook schemes under the same rate, adaptation over a $20$ dB SNR range, and a $K$-fold reduction in model storage compared to separately trained single-codebook networks [2504.11709].

## 4. Feedback, coding, and beam-oriented JOCC

In limited-feedback beamforming, JOCC becomes a coupled design problem between measurement codebooks and inference networks. The site-specific Type-II framework defines a probing codebook $B\in\mathbb C^{N_t\times K}$ with unit-norm columns, an inference network $\Psi_\Theta$, and an inferred subspace basis $C_p=\Psi_\Theta(r_B)\in\mathbb C^{N_t\times Q}$ that is orthonormalized so $C_p^H C_p=I_Q$. The central objective maximizes the normalized CSI-capture efficiency
$$
\eta_p(h;B,\Psi_\Theta)\equiv \frac{\|P_p(h)h\|^2}{\|h\|^2},
\qquad
P_p(h)=C_pC_p^H,
$$
through
$$
\{B^*,\Theta^*\}
=
\arg\max_{B,\Theta} E_{h\sim p_s}[\,\eta_p(h;B,\Psi_\Theta)\,].
$$
The RSRP measurement vector is
$$
r_B = 10\log_{10}(|B^H h|^2 P_{\mathrm{SSB}})+\text{noise},
$$
and the offline solver updates both $B$ and $\Theta$ via mini-batch backpropagation. Under standard smoothness and bounded-variance assumptions, mini-batch SGD converges in expectation to a first-order stationary point at rate $O(1/\sqrt I)$. The reported results include an ablation in “asu_campus_3p5” with $\eta_p=0.89$ for JOCC versus $0.62/0.73$ for random and DFT probing; online UE complexity is $O(Q^2)$, whereas Type-II requires $O(N_t^2 Q O_D + Q^3)$ [2604.14524].

In code-domain NOMA, JOCC is realized as an autoencoder for multi-user multidimensional modulation. Han et al. formulate a joint optimization over the multi-user constellation $\mathcal C$, bit-to-symbol mappings $\mathcal F_b$, and resource-mapping matrix $F$, subject to a power constraint. The distinctive architectural element is dense resource mapping combined with a global power-normalization layer,
$$
\alpha =
\sqrt{\frac{J\,P}{\sum_{j=1}^J \|\tilde s^{(j)}\|_2^2}},
\qquad
s^{(j)}=\alpha\,\tilde s^{(j)},
$$
so that the sum of powers across all users and resources is fixed while power allocation remains flexible. Training proceeds in two stages: first with a loss weighted by the Hamming distance between true and decoded bits,
$$
\mathcal L_1
=
\frac{1}{J}\sum_{j=1}^J
(\mu+\delta\, d_H(b^{(j)},\hat b^{(j)}))
\|r^{(j)}-\hat r^{(j)}\|_2,
$$
and then with pure Euclidean shaping. In the reported $J=6$, $K=4$, $M=4$ setting, JOCC reaches BER $=10^{-3}$ at $\sim 9.5$ dB $E_b/N_0$, while the equivalent single-user MDM autoencoder achieves $\sim 9.2$ dB, conventional SCMA lies at $\sim 11.6$ dB, and a power-imbalanced SCMA heuristic lies at $\sim 10.8$ dB [2104.00818].

An earlier coding-theoretic JOCC formulation treats the codebook itself as the object of source-symbol-aware error-control design. Given source symbols $S=\{s_1,\dots,s_M\}$, a codebook $C$ assigns each $s$ a binary codeword $\Phi(s)\in\{0,1\}^n$, and a decoder $g:\mathbb R^n\to S$ is chosen to minimize
$$
L(C,g)=E_{s,n}\bigl[\ell(s,g(y))\bigr].
$$
For any fixed codebook, the Bayes-optimal decoder is
$$
g(y)=
\arg\min_{s'\in S}
\sum_{s\in S} \ell(s,s')\,p(y\mid \Phi(s))\,p(s).
$$
The JOCC search alternates between Bayes-decoder updates and codebook moves such as flipping one bit in a single codeword, swapping two entire codewords, or permuting columns. At SNR $=0$ dB for rate-$4/7$ codes, the reported $e_{\delta_2}$ values are $0.42/0.38/0.36$ for Hamming hard/soft/Bayes decoding and $0.36/0.30/0.25$ for JOCC-optimized $C$ with $\ell_2$; the corresponding $e_{\delta_1}$ values are $0.28/0.24/0.23$ for Hamming and $0.24/0.20/0.18$ for JOCC-optimized $C$ with $\ell_1$ [1805.07429].

Near-field XL-RIS beam training provides yet another meaning of JOCC, now as joint construction of BS precoders and RIS phase-shift codewords. At each beam-training level, JOCC minimizes
$$
\min_{w_t,\bm\phi,\psi}
\left\|
\bm\phi^H\widetilde{\mathbf H}(w_t) - (p\odot\psi)^T
\right\|_2^2
$$
subject to the BS power limit $\|w_t\|_2^2\le P_{\max}$, unit-modulus phase-adjust variables, and $v$-bit discrete RIS phases. The alternating-optimization procedure updates the BS codeword in closed form,
$$
w_t^{\ell+1}
=
(\widetilde{\mathbf H}_2^H\widetilde{\mathbf H}_2+\lambda_w I_M)^{-1}
\widetilde{\mathbf H}_2^H(p\odot \psi^\ell),
$$
then updates RIS phases via an IPDD-based projection onto the $2^v$-PSK set, and finally updates
$$
\psi_i^{\ell+1}
=
\exp\!\bigl[j\,\angle(\phi^{\ell+1,H}\widetilde{\mathbf H}(w_t^{\ell+1})_i)\bigr].
$$
The reported per-iteration complexity is
$$
{\cal O}\bigl(M^3 + (M+N+MN)\,S_l^xS_l^z\bigr),
$$
empirical convergence occurs in $20$–$30$ outer iterations, and runtime is reported as $\sim 5\times$ slower than SOCC for $M=4,v=2$. In achievable-rate comparisons after beam training, JOCC is reported as approximately equal to ideal-RIS, with SA-BS codebook about $2$–$3$ dB worse [2508.18582].

## 5. Compression and representation learning

In network compression, JOCC is centered on multi-codebook weight quantization with no mapping overhead. The method clusters output neurons, permanently reorders rows of the weight matrix so neurons in the same cluster become contiguous, and ties each row to one of $k$ distinct codebooks by the rule $\lfloor \sigma_i \frac{k}{n_o}\rfloor$. The resulting quantization scheme allows different groups to use different codebooks while avoiding the memory-expensive mapping used by prior multi-codebook methods. Optimization is performed one layer at a time on a calibration batch, with gradients propagated through quantized weights and a proximal index update that favors small moves toward nearby codewords rather than jumps toward extreme values [2309.17361].

The reported empirical profile is broad. On ImageNet-trained ResNet-18, with a per-tensor fp16$\to$3-bit compression target $\alpha\approx 5.33$, the fully optimized method reaches $62.94\%$ top-1 versus $68.63\%$ for the fp16 baseline and $53.00\%$ for NUPES. On ViT-b16 at the same $\alpha=5.33$, the reported top-1 is $80.56\%$ versus $80.97\%$ fp16. On Stable Diffusion v2.0 at $4$-bit weights ($\alpha=3.9$), the initialization alone recovers CLIP score $28.47$ versus $27.50$ for PowerQuant. On Llama-7B, fp16$\to$3-bit compression yields $55.94\%$ on the common-sense benchmark versus $49.94\%$ for OPTQ and $36.43\%$ for RED++, while at $\alpha\approx 7\times$ the footprint is $2$ GB and the retained score is $53.08\%$; the abstract summarizes this as compression to “2Go” and loading on “5-year-old smartphones” [2309.17361].

The second-order representation-learning variant integrates a trainable codebook into compact bilinear pooling. Starting from local descriptors $x_1,\dots,x_{h\cdot w}\in\mathbb R^d$, the standard bilinear feature
$$
B=\sum_i x_i x_i^T
$$
is augmented by codeword-conditioned soft assignments $a_i$, and then factorized jointly with low-rank projections. The JCF-N form
$$
z_i=\sum_i \langle U_i a_i,x_i\rangle \,\langle V_i a_i,x_i\rangle
$$
uses $N d D$ parameters for $U$ and the same for $V$, while JCF-N-R replaces codeword-specific projections with shared basis projections $\{\tilde U_r\}_{r=1}^R$, $\{\tilde V_r\}_{r=1}^R$ and recombination matrices $P,Q\in\mathbb R^{N\times R}$, reducing the count to $2RdD+2NR$ [1906.01972].

This JOCC/JCF representation is trained end-to-end under the N-pair loss and optional codeword normalization. On Stanford Online Products, the reported recall@1 is $77.4\%$ for JCF-32 and $76.6\%$ for JCF-32-8, compared with $74.8\%$ for HTL, $73.7\%$ for Proxy-NCA, and $72.7\%$ for Margin. On CUB-200-2011, JCF-32 reaches $60.1\%$ versus $57.1\%$ for Ge. On Cars-196, JCF-32 reaches $82.6\%$ versus $81.4\%$ for Ge. Parameter counts range from $0.8$ M for JCF-4-4 to $8.9$ M for JCF-32-32, with JCF-32-8 reported at $2.6$ M parameters [1906.01972].

## 6. Recurring design patterns, trade-offs, and common misconceptions

A persistent misconception is that JOCC simply means “using a learnable codebook.” The cited work shows a stricter pattern: the codebook is almost always coupled to another optimized object and to an explicit system loss. In semantic communication, that coupling may be entropy regularization, mutual information, or channel-aware semantic distortion; in beamforming it is CSI-capture efficiency through a learned inference subspace; in coding it is Bayes risk under a significance-aware loss; and in second-order retrieval it is factorization under metric learning [2510.07108] [2604.14524] [1805.07429] [1906.01972]. This suggests that the defining characteristic of JOCC is joint system optimization, not merely codeword learning.

A second misconception is that JOCC belongs only to latent quantization. The surveyed papers use the term for VQ codebooks, multiple codebooks with trainable bit-flip probabilities, beam-probing codebooks, NOMA multidimensional constellations, source-symbol ECC codebooks, and joint BS/RIS beam codewords [2504.11709] [2104.00818] [2508.18582]. The shared abstraction is a discrete design space whose geometry is made task-aware by end-to-end optimization.

The main trade-offs also recur across domains. Codebook cardinality affects both representation fidelity and robustness: in the theoretically grounded semantic formulation, $K$ is chosen by minimizing $D_S(K,p_b)+\lambda R(K)$ and is swept in practice over values such as $64,128,256$; in the Transformer-based generative system, “the codebook size $L$ trades off reconstruction detail vs. robustness to index errors” [2510.07108] [2402.16868]. Joint optimization often improves end performance while relocating complexity: the site-specific beamforming design “pushes the heavy inference into the BS,” reducing UE complexity to $O(Q^2)$; the compression method incurs an offline calibration run over each layer and a one-time neuron reordering; the XL-RIS design yields the highest beam-focusing accuracy but with higher design time and memory than SOCC [2604.14524] [2309.17361] [2508.18582].

Resource footprints remain application-specific rather than uniformly small. In satellite-terrestrial semantic forwarding, the total model is reported at approximately $2.9$ M parameters and approximately $61$ GFLOPs per $512\times512\times3$ image, the codebook occupies approximately $8$ KB, and the on-board satellite workload is approximately $0.2$ GFLOPs per image; in contrast, the Transformer-based generative system explicitly notes that training a large VQ-AE and a nine-block Transformer end-to-end is computationally demanding [2603.02536] [2402.16868]. The overall literature therefore does not support a universal claim that JOCC is either lightweight or heavyweight. It supports a narrower conclusion: JOCC systematically exchanges additional offline or centralized optimization for codebooks whose discrete structure is aligned with the downstream distortion measure, channel model, or task objective.

Source: https://www.emergentmind.com/topics/jointly-optimized-codebook-construction-jocc