Matryoshka Representations in Adaptive Models

• Matryoshka representations are nested, resource-adaptive embeddings that provide multiple inference operating points from a single model.
• They employ joint training objectives across various scales to ensure that each subrepresentation maintains high fidelity for its task.
• Their versatile formulation has been successfully applied to domains such as speaker verification, multimodal VQA, and recommendation systems.

Matryoshka representations are nested representations of increasing granularity contained within a single model output or parameterization. In the now-standard machine-learning formulation, a model produces one full representation, and each lower-dimensional prefix, slice, or submodel is trained to remain useful on its own, so that dimensionality, depth, token budget, or model width can be chosen at inference time without retraining (Kusupati et al., 2022). This nesting principle has since been instantiated in speaker embeddings, sentence embeddings, retrieval systems, multimodal LLMs, recommendation models, sparse autoencoders, concept bottleneck models, state-space models, and clustering heads (Wang et al., 2024, Cai et al., 2024, Bussmann et al., 21 Mar 2025). The term also has distinct mathematical uses, where it denotes nested categorical embeddings or indecomposable self-extensions rather than adaptive computational representations (Neto et al., 13 Feb 2026, Thierry-Mieg et al., 2022).

1. Core definition and nesting patterns

The foundational formulation of Matryoshka Representation Learning (MRL) defines an encoder $F:\mathcal X\to\mathbb R^d$ that maps an input $x$ to a full vector $z=F(x;\theta_F)\in\mathbb R^d$, together with a nested set of fidelity sizes $\mathcal M=\{m_1<m_2<\dots<m_K\}$. Each prefix $z_{1:m_k}\in\mathbb R^{m_k}$ is required to be a high-quality embedding, so that a single forward pass yields a family of usable subrepresentations (Kusupati et al., 2022). In supervised settings, the original formulation attaches a classifier to each prefix, while an efficient variant, MRL-E, ties the weights so that the classifier for a smaller prefix is the corresponding prefix of one shared classifier matrix (Kusupati et al., 2022).

Although the earliest formulation was one-dimensional in width, later work generalized the same nesting principle along other axes. In Two-dimensional Matryoshka Sentence Embeddings, the full representation is organized as an $N\times D$ tensor of layerwise hidden states, and any embedding can be selected as a pair $(n,d)$ consisting of a layer prefix and a dimension prefix (Li et al., 2024). MatMamba extends the idea to state-space models by arranging Mamba2 block parameters so that smaller sub-SSMs are prefixes of a larger block, yielding nested models $M_1\subset M_2\subset\cdots\subset M_g$ (Shukla et al., 2024). In visual and multimodal systems the nested object need not be a flat vector: M3 defines nested sets of visual tokens, and Franca defines nested feature slices whose coarse-to-fine clustering heads refine semantic granularity as dimension increases (Cai et al., 2024, Venkataramanan et al., 18 Jul 2025).

A recurring structural property is that the earlier coordinates or earlier slices are forced to carry the most salient information. In M-Vec, this is stated explicitly: because every classifier sees only the first $m$ dimensions, the encoder is forced to order its representation such that the most salient speaker information is in the early coordinates (Wang et al., 2024). Related work in recommendation, concept bottlenecks, and duration-aware speaker verification similarly assigns coarse, broad, or short-context information to earlier prefixes and finer distinctions to later ones (Lai et al., 2024, Chen et al., 20 May 2026, Jung et al., 20 Jan 2026).

2. Joint training objectives and information ordering

The standard optimization pattern is multi-scale supervision. In the original MRL formulation, training minimizes a sum of losses over all requested dimensions,

$$\min_{\theta_F,\{\mathbf W^{(m)}\}} \frac1N\sum_{i=1}^N\sum_{m\in\mathcal M} c_m\, \mathcal L\bigl(\mathbf W^{(m)} z_{1:m}^{(i)},y_i\bigr),$$

with default $x$0 (Kusupati et al., 2022). The same pattern reappears across domains with task-specific losses. M-Vec combines Matryoshka Representation Learning with Additive Angular Margin Softmax and sums AAM-Softmax losses across $x$1, again with $x$2 in the reported experiments (Wang et al., 2024). In cross-lingual speech-text retrieval, the Matryoshka loss is a sum of InfoNCE losses over all slice sizes $x$3, applied to truncated query and document embeddings (Sy et al., 23 Feb 2026).

Several systems extend per-prefix supervision with explicit alignment terms. Two-dimensional Matryoshka Sentence Embeddings adds losses for the last layer, a sampled shallow layer, a sampled sub-dimension at the last layer, a sampled sub-dimension at a shallow layer, and a shallow-to-last-layer alignment term based on KL divergence (Li et al., 2024). MIPIC argues that training useful prefixes requires coordination of both internal structure and model depth; it introduces Self-Distilled Intra-Relational Alignment, which aligns teacher and student attention distributions and token-level geometry using KL divergence and CKA, and Progressive Information Chaining, which uses InfoNCE between successive $x$4 checkpoints (Huy et al., 27 Apr 2026).

A notable technical point is that Matryoshka methods do not uniformly rely on orthogonality constraints. M-Vec states that no explicit orthogonality or decorrelation constraints are used and that the nesting is learned implicitly (Wang et al., 2024). The French-Wolof speech-text model likewise notes that in its setting the nesting is purely done by ordered slicing, and no extra orthogonality term is added (Sy et al., 23 Feb 2026). By contrast, later work sometimes revisits this design space indirectly through self-distillation, progressive chaining, or learned selection mechanisms rather than direct orthogonality penalties (Huy et al., 27 Apr 2026, Zhang et al., 14 Oct 2025).

3. Adaptive inference, efficiency, and representative trade-offs

At inference time, a canonical Matryoshka model exposes multiple operating points from one checkpoint. In the original MRL formulation, the model computes $x$5 once, and any prefix $x$6 is immediately available “for free” (Kusupati et al., 2022). M-Vec uses the same rule: given a target dimension $x$7, the system slices $x$8 and applies cosine scoring or any back-end, with no retraining or fine-tuning required to change $x$9 (Wang et al., 2024). M3 selects the coarsest visual scale whose token count does not exceed a budget $z=F(x;\theta_F)\in\mathbb R^d$0 (Cai et al., 2024). Two-dimensional Matryoshka Sentence Embeddings choose both a layer index $z=F(x;\theta_F)\in\mathbb R^d$1 and a width $z=F(x;\theta_F)\in\mathbb R^d$2, so inference cost is reduced by stopping early in depth and truncating width (Li et al., 2024). In audio-visual speech recognition, Llama-MTSK pre-computes token counts and approximate FLOPs for all compression pairs $z=F(x;\theta_F)\in\mathbb R^d$3 and selects the coarsest pair satisfying a compute budget (Cappellazzo et al., 9 Mar 2025).

Representative efficiency results reported for Matryoshka systems are summarized below (Kusupati et al., 2022, Wang et al., 2024, Cai et al., 2024, Sy et al., 23 Feb 2026).

Setting	Full scale	Reduced scale / result
ImageNet-1K classification	512-dim FF achieves 76.3% top-1	MRL-AC attains 76.3% top-1 using only $z=F(x;\theta_F)\in\mathbb R^d$4 dims on average
Speaker verification (M-Vec)	256 D: EER = 1.153%; storage 9765.62 MB; time 759.31 ms	16 D: EER = 2.605%; storage 610.35 MB; time 76.86 ms
Image VQA on MMBench	576 tokens: 65.9%	9 tokens: 63.1%; 1 token: 59.5%
French-Wolof retrieval	1024 d: nDCG@5 = 69.85, @10 = 74.49	256 d: 61.58 / 67.05; 128 d: 56.13 / 62.30

These results support the central systems claim of the literature: Matryoshka representations are intended for deployment regimes in which memory, latency, token count, or annotation budget is not known at training time. The recurrent emphasis is not merely on compression, but on preserving a smooth accuracy degradation curve as the operating budget changes (Kusupati et al., 2022, Wang et al., 2024, Cai et al., 2024).

4. Modalities and architectural embodiments

In speaker and speech systems, Matryoshka representations are usually realized as nested embedding prefixes. M-Vec uses a frame-level encoder such as ResNet34, statistics pooling, projection to $z=F(x;\theta_F)\in\mathbb R^d$5, and separate linear heads for $z=F(x;\theta_F)\in\mathbb R^d$6 (Wang et al., 2024). DAME keeps the same embedding-level nesting idea but aligns it to utterance duration: short utterances supervise smaller prefixes, longer utterances supervise larger prefixes, and inference still uses the full $z=F(x;\theta_F)\in\mathbb R^d$7-dimensional embedding with no additional cost (Jung et al., 20 Jan 2026).

In multimodal large-model settings, the nested object is often a token sequence or a cross-modal fusion pathway. M3 represents an image as nested sets of visual tokens with canonical scales $z=F(x;\theta_F)\in\mathbb R^d$8, using a pre-trained CLIP-ViT-L-336 and a Vicuna/LLaVA interface, and averages answer likelihood over all scales during training (Cai et al., 2024). The French-Wolof retrieval model uses a frozen text Matryoshka backbone, Qwen3-0.6-Embedding, and finds that Late-Fusion—HuBERT features, a small CNN, a trainable matrix $z=F(x;\theta_F)\in\mathbb R^d$9, and a frozen Qwen3 LLM—performs best (Sy et al., 23 Feb 2026). Llama-MTSK applies the same principle to audio-visual speech recognition by jointly training all audio and video compression ratios and activating only the chosen projector and LoRA adapter at inference (Cappellazzo et al., 9 Mar 2025). MoME further adds routed and shared experts with a shared router across scales, so coarse and fine streams can activate similar expert subsets (Cappellazzo et al., 5 Oct 2025).

Recommendation models use Matryoshka structure to encode hierarchical user preferences and item features. MRL4Rec defines user and item prefixes $\mathcal M=\{m_1<m_2<\dots<m_K\}$0 and introduces Matryoshka Negative Sampling, which provides a distinct hard negative for each level; the paper argues that this level-specific construction is necessary for genuine hierarchical specialization (Lai et al., 2024). fMRLRec extends the idea to multimodal sequential recommendation by projecting language and image features into an aligned space, processing them with Linear Recurrent Units, and using binary masks so that one large weight matrix contains the valid weights for all smaller scales (Wang et al., 2024). In federated learning, FedMRL fuses a global homogeneous representation and a local heterogeneous representation through a personalized projector, then trains nested coarse and fine heads on the fused feature (Yi et al., 2024).

Vision self-supervision provides another embodiment. Franca defines nested slices $\mathcal M=\{m_1<m_2<\dots<m_K\}$1 from a ViT encoder output and feeds each slice to its own projection-plus-clustering head with prototype count $\mathcal M=\{m_1<m_2<\dots<m_K\}$2 proportional to $\mathcal M=\{m_1<m_2<\dots<m_K\}$3, producing a coarse-to-fine hierarchy of clusters (Venkataramanan et al., 18 Jul 2025). Here the Matryoshka mechanism is not only a deployment tool, but also a way to encode ambiguity in clustering semantics across multiple granularities (Venkataramanan et al., 18 Jul 2025).

5. Extensions beyond width-only embeddings

A major line of development generalizes Matryoshka structure from output dimensionality to model depth. Two-dimensional Matryoshka Sentence Embeddings define the family of embeddings $\mathcal M=\{m_1<m_2<\dots<m_K\}$4 by taking the first $\mathcal M=\{m_1<m_2<\dots<m_K\}$5 coordinates of the CLS vector at layer $\mathcal M=\{m_1<m_2<\dots<m_K\}$6, so both layers and dimensions become adaptive inference knobs (Li et al., 2024). m3BERT adopts the same principle at pretraining scale: it jointly optimizes selected layers $\mathcal M=\{m_1<m_2<\dots<m_K\}$7 and target dimensions $\mathcal M=\{m_1<m_2<\dots<m_K\}$8 during MLM pretraining, then continues with an infinite-batch contrastive objective over nested slices in domain-specific retrieval pretraining (Wang et al., 19 May 2026). MatMamba translates nesting into the internal dimensions and head counts of Mamba2 blocks, training a single large model that yields smaller nested models “for free” (Shukla et al., 2024).

Another extension applies Matryoshka structure to interpretable latent spaces. Matryoshka Sparse Autoencoders define a shared encoder and decoder with nested dictionaries $\mathcal M=\{m_1<m_2<\dots<m_K\}$9, and each level reconstructs the input using only the first $z_{1:m_k}\in\mathbb R^{m_k}$0 latents (Bussmann et al., 21 Mar 2025). The paper reports a trade-off: Matryoshka sacrifices ~1–2 pp of variance explained but improves feature absorption, splitting, sparse probing, spurious correlation removal, targeted perturbation behavior, and scaling stability (Bussmann et al., 21 Mar 2025). Matryoshka Concept Bottleneck Models order concepts by a Minimum Redundancy Maximum Relevance criterion, train heads for multiple concept budgets, and prove monotonic improvement bounds under added interventions; under the balanced condition $z_{1:m_k}\in\mathbb R^{m_k}$1, the expected intervention cost is reduced from $z_{1:m_k}\in\mathbb R^{m_k}$2 to $z_{1:m_k}\in\mathbb R^{m_k}$3 (Chen et al., 20 May 2026).

These developments shift the meaning of “nested representation” from a literal prefix of one output vector to a more general organizational principle: depth-wise, concept-wise, dictionary-wise, or submodel-wise nesting inside a single training run. A plausible implication is that Matryoshka methods are best understood as a family of resource-adaptive parameter-sharing schemes rather than a single embedding trick.

6. Misconceptions, limitations, and active research directions

A common misconception is that Matryoshka behavior can be obtained by simply truncating a conventionally trained representation after the fact. Multiple papers explicitly argue against this interpretation. The original MRL formulates nesting as a joint training objective rather than post-hoc compression (Kusupati et al., 2022). M-Vec attributes its behavior to the fact that every classifier sees only the first $z_{1:m_k}\in\mathbb R^{m_k}$4 dimensions, forcing the encoder to order salient speaker information into early coordinates (Wang et al., 2024). MRL4Rec goes further: if the same negative item is used at every level, the resulting matryoshka loss yields exactly the same gradient direction, up to a positive scalar, on each shell as the single-level BPR loss on the entire embedding, so no true hierarchical specialization can emerge (Lai et al., 2024).

Another misconception is that Matryoshka methods must always use fixed positional truncation. While prefix slicing is the dominant mechanism, several works question or relax it. SMEC argues that original MRL’s parallel multi-scale training can inflate gradient variance, then proposes Sequential Matryoshka Representation Learning, Adaptive Dimension Selection, and Selectable Cross-batch Memory; in its analysis, MRL exhibits 5× higher gradient variance than SMRL on the reported MiniLM-on-BEIR study (Zhang et al., 14 Oct 2025). The French-Wolof speech-text model reports that for $z_{1:m_k}\in\mathbb R^{m_k}$5, 90% of total variance is captured by the first 200 components and full 100% energy is reached by ≈600 dims, while lower dimensions can remain critical for retrieval, leading the authors to suggest future work on data-driven subspace decompositions rather than fixed prefix slices (Sy et al., 23 Feb 2026).

The literature also identifies open directions. The original MRL names automatic loss weighting, multi-objective fidelity, learned indexing structures, and joint optimization of search pipelines with MRL (Kusupati et al., 2022). M3 proposes a scale-selector network $z_{1:m_k}\in\mathbb R^{m_k}$6 that predicts the minimal visual scale needed for a question (Cai et al., 2024). Two-dimensional Matryoshka Sentence Embeddings point to early-exit or partial-attention mechanisms to reduce training cost and to learned subspace projections or gated subnetworks for small $z_{1:m_k}\in\mathbb R^{m_k}$7 (Li et al., 2024). The French-Wolof retrieval model suggests structured sparsity, bit-level Matryoshkas, and extension to additional under-represented languages and modalities (Sy et al., 23 Feb 2026). These proposals indicate that the field is moving from fixed nested prefixes toward adaptive structure selection.

7. Non-ML mathematical uses of the term

Outside contemporary machine learning, “Matryoshka representations” designates different constructions. In the category of partial representations of a finite abelian group, the Matryoshka’s Theorem states that for any subgroup $z_{1:m_k}\in\mathbb R^{m_k}$8 there is a faithful, additive, and strongly monoidal embedding

$z_{1:m_k}\in\mathbb R^{m_k}$9

so that categories of partial representations of subgroups sit inside that of the ambient group as nested multifusion subcategories (Neto et al., 13 Feb 2026). The theorem is formulated via an algebra map between partial group algebras and a restriction-of-scalars functor, not via adaptive dimensional truncation (Neto et al., 13 Feb 2026).

In the representation theory of type I Lie superalgebras and quasi-reductive Lie superalgebras, a Matryoshka module $N\times D$0 is an indecomposable $N\times D$1-fold self-extension of a Kac module $N\times D$2, with a Loewy series containing $N\times D$3 successive copies of the seed module (Thierry-Mieg et al., 2022). The construction uses the derivative with respect to the continuous odd Dynkin weight and yields block-upper-triangular actions with generalized Cabibbo angles in the super-diagonal blocks (Thierry-Mieg et al., 2022). Here the “Russian doll” metaphor refers to nested indecomposable modules rather than flexible computational representations.

The coexistence of these usages is conceptually coherent but technically distinct. In machine learning, Matryoshka typically refers to nested computational representations that support adaptive inference budgets. In category theory and Lie-superalgebra theory, it refers to nested embeddings of categories or nested indecomposable extensions. The shared intuition is containment across scales; the formal objects, however, are different.