Compositional Code Embeddings (CCE)

• Compositional Code Embeddings (CCE) are a technique that reconstructs token embeddings as the sum of basis vectors from multiple codebooks, significantly reducing memory usage.
• It achieves embedding compression rates of up to 98% and encoder reductions of 20–34%, while retaining over 97.5% semantic parsing performance across various transformer models.
• CCE is integrated post-pretraining to replace dense embedding tables, making transformer-based models more viable for on-device inference in voice assistants and edge deployments.

Compositional Code Embeddings (CCE) are a compression method for neural embedding tables in which each token embedding is reconstructed as a composition of a small number of basis vectors drawn from multiple codebooks, rather than stored as an independent dense vector. In transformer-based task-oriented semantic parsing, this formulation was used to reduce the sizes of BERT-base, RoBERTa-base, DistilBERT, ALBERT-base, and ALBERT-large while preserving greater than 97.5% semantic parsing performances, with reported embedding compression rates of 95.15% ~ 98.46% and encoder compression rates of 20.47% ~ 34.22% (Prakash et al., 2020).

1. Definition and problem setting

The immediate motivation for CCE in semantic parsing is that the current state-of-the-art task-oriented semantic parsing models use BERT or RoBERTa as pretrained encoders, and these models have huge memory footprints. This poses a challenge to their deployment for voice assistants such as Amazon Alexa and Google Assistant on edge devices with limited memory budgets. In the reported setting, word embeddings and encoder parameters constitute much of this size, so compressing the embedding layer yields a direct route to reducing overall encoder size (Prakash et al., 2020).

CCE addresses this problem by replacing a standard embedding table with a structured representation composed from shared basis vectors. The method is applied after pretrained transformer variants have their embedding tables extracted, and the learned CCE layer is then substituted for the original embedding layer. This makes CCE a model-compression technique targeted at the embedding parameterization rather than a redesign of the transformer architecture itself.

A related antecedent appears in "Compressing Word Embeddings via Deep Compositional Code Learning" (Shu et al., 2017), which proposed constructing embeddings with few basis vectors and learning discrete codes end-to-end. In that earlier formulation, the motivation was similarly the large storage or memory footprint of word embeddings, especially for deployment on devices with limited storage or memory.

2. Codebook formulation and reconstruction objective

In the CCE architecture for transformer compression, there are $M$ codebooks, and each codebook contains $K$ basis vectors of the same dimension $D$ as the original embeddings. For each vocabulary token $w$, the model associates a discrete code vector $(C_1, \ldots, C_M)$, where each $C_i \in \{1,\ldots,K\}$. The embedding is then composed as the sum of the selected basis vectors from the $M$ codebooks (Prakash et al., 2020):

$$\text{Let } \mathcal{E}_C \text{ be the set of } M \text{ codebooks, each of size } K \times D$$

$$\text{For token } w: \quad e_C(w) = \sum_{i=1}^M E_C^i(C_i)$$

where $E_C^i(C_i)$ is the $K$0-th vector from the $K$1-th codebook.

The learning objective is a reconstruction problem against a pretrained reference embedding table:

$K$2

where $K$3 is the pretrained embedding of $K$4. The discrete codes are not directly differentiable, so the Gumbel-Softmax reparameterization trick is used to enable efficient end-to-end training.

The storage comparison is central to the method. The original embedding table has $K$5 parameters stored as 32-bit floats. By contrast, CCE stores $K$6 parameters as 32-bit floats, plus a code for each token of size $K$7, with each code entry stored as $K$8 bits. Since usually $K$9 and $D$0, the storage reduction is substantial. For BERT-base, the paper gives the example $D$1, $D$2, with $D$3 and $D$4.

3. Integration with transformer encoders and training procedure

The reported workflow has two stages (Prakash et al., 2020). In Stage 1, code embedding learning is performed offline. The reference is the pretrained embedding matrix, and the CCE parameters—codebooks and discrete codes—are trained using mean squared error to reconstruct the reference embedding table. Typical hyperparameters found via validation are $D$5, $D$6, and epochs $D$7. The paper also notes that performance plateaus for code embedding training beyond $D$8 epochs.

In Stage 2, the model is finetuned on downstream semantic parsing tasks after the embedding layer has been replaced with CCE. During this phase, the discrete codes for each word are fixed after code embedding learning, while the codebook basis vectors and other model parameters are finetuned. The output layer is randomly initialized. For downstream losses, intent prediction uses a softmax over the [CLS] token, slot prediction uses a softmax over all tokens, and training uses cross-entropy loss.

The method is described as orthogonal and complementary to existing transformer compression strategies. DistilBERT shrinks overall model size through distillation; ALBERT applies parameter sharing and embedding factorization; CCE can be applied on top of these for greater compression. This is operationally important because the reported experiments do not restrict CCE to uncompressed transformers: DistilBERT, ALBERT-base, and ALBERT-large are already compressed BERT variants, and CCE further compresses their embedding layers.

The trade-off between code size and task performance follows the reconstruction regime. Larger $D$9 yields better performance and reconstruction. Increasing $w$0 helps, but less so when $w$1 is already large. Very high compression, corresponding to smaller $w$2 and $w$3, eventually causes performance to drop. A common misunderstanding is to treat CCE as a fully dynamic discrete optimization during downstream finetuning; in the reported recipe, the discrete codes are frozen after the code embedding learning stage.

4. Empirical compression and semantic parsing performance

The empirical evaluation uses the semantic parsing benchmarks SNIPS, ATIS, and Facebook TOP (Prakash et al., 2020). The reported compression rates are summarized below.

Model	Embedding Compression	Encoder Compression
RoBERTa-base	98.46%	30.33%
BERT-base-uncased	97.80%	20.82%
DistilBERT-base-uncased	97.80%	34.22%
ALBERT-large-v2	95.15%	20.47%
ALBERT-base-v2	95.15%	30.94%

Across these models, embedding tables are compressed by 95–98%, and total encoder size is reduced by 20–34%. For ALBERT-large, the encoder is reduced to 54MB; for ALBERT-base, to 31MB. The paper also gives a concrete ALBERT-large-v2 example in which the embedding is reduced from 14.65MB to 0.71MB and the encoder from 68MB to 54MB, with $w$4 task performance retained. For RoBERTa-base, the embedding size drops from 147MB to just over 2MB.

The reported downstream degradation is small. CCE models retain $w$5 of the original model’s exact match performance on semantic parsing tasks. On SNIPS, ALBERT-large-cc achieves 92.43% EM, matching the original 92.43%, while BERT-base-cc achieves 90.71% EM versus 92.29% for the original. On ATIS and Facebook TOP, the paper reports similar negligible drops, within $w$6 in EM or intent accuracy.

These results delimit the practical scope of the method. The largest gains occur in the embedding layer, and the encoder-level reduction is consequently smaller than the embedding-level reduction. This suggests that CCE is most effective when embedding tables constitute a large share of the memory footprint.

5. Deployment implications and relation to edge inference

The deployment rationale for CCE is explicit: multiple compressed variants fit comfortably within memory limits of common edge devices, while retaining near state-of-the-art task performance (Prakash et al., 2020). In the semantic parsing setting, this directly targets on-device inference for voice assistants and related systems operating under strict memory budgets.

The reported implications are not limited to storage. The paper states that the retained performance suggests minimal need for cloud-offload and enhanced privacy, latency, and robustness. These implications follow from reducing the memory footprint sufficiently to make local deployment more feasible without replacing the pretrained encoder family or substantially rewriting the downstream model.

CCE is therefore best understood as a memory-oriented compression layer that can be stacked with other compression methods. It is not presented as an alternative to distillation or parameter sharing, but as an additive mechanism that further compresses embedding layers even when the base transformer has already been compressed.

6. Historical lineage, related variants, and terminological scope

The transformer-based semantic parsing formulation builds on prior work in compositional code learning for word embeddings. "Compressing Word Embeddings via Deep Compositional Code Learning" (Shu et al., 2017) proposed to construct embeddings with few basis vectors, assign each word a multi-component discrete code such as $w$7, and directly learn the discrete codes in an end-to-end neural network by applying the Gumbel-softmax trick. That paper reported a 98% compression rate in a sentiment analysis task and 94% ~ 99% in machine translation tasks without performance loss, and it emphasized that the method is language-independent and does not require modifications to the network architecture.

The acronym CCE is not unique in the broader literature. In recommendation systems, "Clustered Compositional Embeddings" denotes a different method that combines clustering-based compression with dynamic hashing-based methods for embedding tables during training (Tsang et al., 2022). In concept learning, "Compositional Concept Extraction" denotes an unsupervised method for finding concept representations that obey compositionality properties in image and text embeddings (Stein et al., 2024). A plausible implication is that references to “CCE” require local disambiguation from context, since the same acronym spans compression, recommendation, and interpretability literatures.

Within the specific literature on Compositional Code Embeddings, however, the term “code” has a precise technical meaning: a token-specific discrete assignment over multiple codebooks, used to compose the final embedding by summing selected basis vectors. That usage is consistent across the word-embedding precursor and the transformer semantic parsing application.