Disentangled Continuous Semantic Representation Model

• The paper introduces DCSRM, a framework that decomposes dense embeddings into semantically specialized subspaces to capture finer subdimensions like vision, action, and emotion.
• It employs a linear transformation optimized by multi-objective losses—orthogonality, supervised semantic prediction, contrastive learning, and sparsity—to ensure representation disentanglement.
• Empirical evaluations, including voxel-wise neural encoding, demonstrate that the model preserves core semantic information while revealing meaningful neural correlates.

Searching arXiv for the core DCSRM paper and closely related disentanglement papers. Disentangled Continuous Semantic Representation Model (DCSRM) denotes a framework for transforming dense language-model word embeddings into a structured continuous space in which semantic information is decomposed into multiple semantically specialized sub-embeddings, each intended to isolate a particular semantic subdimension while preserving the geometry of the original embedding space (Zhang et al., 29 Aug 2025). In the formulation introduced in “Discovering Semantic Subdimensions through Disentangled Conceptual Representations” (Zhang et al., 29 Aug 2025), DCSRM is motivated by the claim that the six coarse semantic dimensions—vision, action, social, emotion, time, and space—remain too broad to capture the fine-grained structure of conceptual meaning. The model is therefore designed to discover semantic subdimensions automatically from LLM embeddings, interpret them linguistically, and test their neural plausibility with voxel-wise encoding models (Zhang et al., 29 Aug 2025). Related work situates DCSRM within a broader family of latent-variable approaches that seek disentangled continuous semantic structure in definitions, sentence embeddings, hierarchical sentence VAEs, and domain-adaptive latent spaces (Carvalho et al., 2022, Nastase et al., 2023, Felhi et al., 2020, Cai et al., 2020).

1. Conceptual basis and problem formulation

1. Semantic granularity as the central problem

DCSRM is introduced against two stated limitations of prior semantic dimension systems. The first is overlap and redundancy in predefined dimensions; the second is the need for finer semantic structure inside coarse dimensions such as emotion, action, or time (Zhang et al., 29 Aug 2025). The framework therefore asks what semantic subdimensions exist inside each coarse semantic dimension, and whether those subdimensions are represented in the brain (Zhang et al., 29 Aug 2025).

In the model’s notation, the original word embeddings are $V \in \mathbb{R}^{M \times h}$, the set of continuous semantic attributes is $B = \{b_1,\dots,b_N\}$, and the transformed embeddings are $X \in \mathbb{R}^{M \times h}$ (Zhang et al., 29 Aug 2025). The transformation is linear: $X = V W,$ where $W \in \mathbb{R}^{h \times h}$ is learnable (Zhang et al., 29 Aug 2025).

The intended decomposition is into $N+1$ groups,

$$X_{b_1}, X_{b_2}, \dots, X_{b_N}, X_{\text{unseen}},$$

where each $X_{b_i}$ captures information specific to semantic attribute $b_i$ (Zhang et al., 29 Aug 2025). This formulation defines DCSRM as a disentangling model over continuous semantic attributes rather than a discrete labeling system.

2. What counts as “disentangled” in DCSRM

The model is described as disentangled because it combines an orthogonal transformation to preserve global geometry, supervised regression to preserve target semantic strength, contrastive learning to separate positives and negatives, reconstruction to retain original information, dropout-based sparsity to select dimensions, and distribution alignment to make semantic allocation exclusive (Zhang et al., 29 Aug 2025). It does not produce discrete labels directly; instead, it learns continuous semantic subspaces that are interpreted afterward via PCA and inspection of high-loading words (Zhang et al., 29 Aug 2025).

A plausible implication is that DCSRM treats semantic structure as a set of continuous, selectively allocated subspaces rather than as a single monolithic embedding or a purely symbolic ontology.

2. Mathematical structure and optimization objective

1. Transformation, geometry preservation, and multi-objective training

DCSRM is trained with a multi-objective loss combining orthogonality, supervised semantic prediction, contrastive separation, reconstruction, sparsity, and distribution alignment (Zhang et al., 29 Aug 2025). The orthogonality constraint is

$\mathcal{L}_{\text{ORT}} = \|W^T W - I\|_2.$

Its stated motivation is that if $B = \{b_1,\dots,b_N\}$0, then

$B = \{b_1,\dots,b_N\}$1

so the transformed embedding approximately preserves pairwise similarities (Zhang et al., 29 Aug 2025).

To ensure each sub-embedding retains its target semantic information, the model uses a regression model $B = \{b_1,\dots,b_N\}$2 with the supervised objective

$B = \{b_1,\dots,b_N\}$3

where $B = \{b_1,\dots,b_N\}$4 is the representation of word $B = \{b_1,\dots,b_N\}$5 in sub-embedding $B = \{b_1,\dots,b_N\}$6, and $B = \{b_1,\dots,b_N\}$7 is the corresponding human rating for semantic attribute $B = \{b_1,\dots,b_N\}$8 (Zhang et al., 29 Aug 2025).

For separation of positive and negative items under each attribute, DCSRM uses the semantic contrastive loss

$B = \{b_1,\dots,b_N\}$9

This encourages words strongly associated with a given semantic dimension to cluster together while pushing away unrelated words (Zhang et al., 29 Aug 2025).

Reconstruction is enforced through

$X \in \mathbb{R}^{M \times h}$0

where $X \in \mathbb{R}^{M \times h}$1 is a learned fully connected layer mapping sub-embeddings back to the original space (Zhang et al., 29 Aug 2025). This is used to prevent the model from throwing away too much information during disentanglement.

2. Sparsity and exclusivity mechanisms

Dimension selectivity is induced with variational dropout. Noise is injected as

$X \in \mathbb{R}^{M \times h}$2

where $X \in \mathbb{R}^{M \times h}$3 is the dropout rate (Zhang et al., 29 Aug 2025). Dimensions with dropout rates lower than 40% are treated as belonging to the attribute-specific sub-embedding $X \in \mathbb{R}^{M \times h}$4 (Zhang et al., 29 Aug 2025). This mechanism allows the model to choose which latent dimensions are useful for each semantic attribute.

The distribution alignment loss is

$X \in \mathbb{R}^{M \times h}$5

where $X \in \mathbb{R}^{M \times h}$6 denotes attribute relevance across all $X \in \mathbb{R}^{M \times h}$7, and the authors set $X \in \mathbb{R}^{M \times h}$8 (Zhang et al., 29 Aug 2025). The stated intuition is to maximize sparsity or exclusivity across attributes while enforcing normalized competition among attribute weights (Zhang et al., 29 Aug 2025).

Taken together, these objectives define DCSRM as a structured reparameterization of the original semantic space rather than a generic dimensionality-reduction procedure (Zhang et al., 29 Aug 2025).

3. Discovery and interpretation of semantic subdimensions

1. Post hoc interpretation pipeline

After training, each semantic-specific sub-embedding $X \in \mathbb{R}^{M \times h}$9 is orthogonalized with PCA, yielding transformed sub-embeddings $X = V W,$0 (Zhang et al., 29 Aug 2025). The interpretation procedure then proceeds by computing Pearson correlation and pairwise order consistency with the rating data, keeping only significantly correlated dimensions, inspecting the highest-loading words, prompting multiple LLMs as “linguists” to annotate the semantic meaning of each dimension, and aggregating those annotations into final subdimension labels (Zhang et al., 29 Aug 2025).

This interpretive stage is central to DCSRM. The subdimensions are not assumed beforehand but are discovered from the embedding structure (Zhang et al., 29 Aug 2025). That distinguishes the framework from approaches that begin with a fixed manually specified fine-grained taxonomy.

2. Reported subdimensions

The paper reports that each coarse semantic dimension decomposes into multiple finer subdimensions (Zhang et al., 29 Aug 2025). The following set is explicitly described:

Coarse dimension	Reported subdimensions
Vision	Static vision; Dynamic vision
Action	Outburst acts; Micro-movements; Forceful acts; Downward acts; Functional body acts; Bending/ritual acts
Social	Conflict; Collaboration/exchange
Emotion	Emotional load; Negative valence; Positive valence
Time	Temporal span; Historical change; Commemorative events; Dynastic eras
Space	Regional locations; Sites and orientation; Extreme spaces

The paper characterizes some of these subdimensions as intuitive and aligned with prior semantic or neuroscience literature, and others as novel (Zhang et al., 29 Aug 2025). It further states that polarity is a key organizing principle in semantic decomposition, with explicit examples including static versus dynamic vision, conflict versus collaboration in social semantics, and positive versus negative valence in emotion (Zhang et al., 29 Aug 2025).

A plausible implication is that DCSRM models semantic dimensions not as flat clusters but as internally structured subspaces organized by opposition and hierarchy.

4. Empirical evaluation and ablation findings

1. Intrinsic semantic disentanglement

The main intrinsic evaluation is a semantic prediction task in which ridge regression is trained to predict human semantic ratings from the learned sub-embeddings, with Pearson correlation reported between predicted and gold ratings (Zhang et al., 29 Aug 2025). Target-dimension prediction is compared against non-target dimensions (Zhang et al., 29 Aug 2025). A disentangled sub-embedding is expected to predict its own target dimension well and other dimensions poorly (Zhang et al., 29 Aug 2025).

The reported results state that DCSRM achieves consistently high target correlations across all six dimensions and suppresses non-target correlations relative to target correlations, showing selective encoding (Zhang et al., 29 Aug 2025). Context-aware models such as LLaMA and Alpaca generally outperform GloVe and Word2Vec, and larger models, especially 7B variants, tend to produce stronger semantic disentanglement (Zhang et al., 29 Aug 2025). The strongest models are summarized as achieving approximately

$X = V W,$1

which is presented as strong separation between relevant and irrelevant semantics (Zhang et al., 29 Aug 2025).

The paper also reports that original embeddings and disentangled embeddings perform similarly on semantic prediction, meaning the disentangling transformation preserves the main semantic information rather than destroying it (Zhang et al., 29 Aug 2025).

2. Ablation evidence

The ablation study removes individual losses to determine their contributions (Zhang et al., 29 Aug 2025). Removing $X = V W,$2 or $X = V W,$3 is reported to cause sub-embeddings to disappear under some dropout conditions or to make representations become entangled (Zhang et al., 29 Aug 2025). $X = V W,$4 is described as especially important for preserving the magnitude structure of semantic ratings, whereas $X = V W,$5 is important for decorrelating sub-embeddings and preventing overlap (Zhang et al., 29 Aug 2025). The losses denoted $X = V W,$6, $X = V W,$7, and $X = V W,$8 are reported to further improve semantic separation (Zhang et al., 29 Aug 2025).

This suggests that DCSRM’s disentanglement is not attributable to a single regularizer. Rather, it depends on the interaction between geometry preservation, supervision, selectivity, and exclusivity.

5. Neural plausibility and voxel-wise encoding

1. Encoding pipeline

To test neural plausibility, DCSRM is coupled with voxel-wise encoding models on fMRI data from natural story comprehension (Zhang et al., 29 Aug 2025). The encoding pipeline is described as follows: word embeddings are convolved with a canonical hemodynamic response function and downsampled to the fMRI sampling rate; nuisance regressors include word rate, word length, part of speech, sound envelope, word frequency, and head motion parameters; ridge regression is trained with 5-fold nested cross-validation; prediction performance is evaluated using Pearson correlation between predicted and observed voxel responses; significance is assessed against a null baseline from random Gaussian vectors (Zhang et al., 29 Aug 2025). Cross-validation weights are then averaged, and each voxel is assigned the semantic subdimension with the strongest weight (Zhang et al., 29 Aug 2025).

2. Reported cortical mappings

The sub-embeddings are reported to predict activity in canonical language and semantic regions, including inferior frontal gyrus, superior temporal gyrus, posterior superior temporal sulcus, middle temporal gyrus, inferior temporal gyrus, precuneus, cingulate gyrus, fusiform gyrus, and angular gyrus (Zhang et al., 29 Aug 2025). The paper states that these findings are consistent with established language-semantic networks (Zhang et al., 29 Aug 2025).

The authors further report a contrast between sensory and abstract dimensions: for embodied dimensions such as vision and action, human rating data often predicts broader brain regions than the transformed sub-embeddings, whereas for abstract dimensions such as emotion and time, the transformed sub-embeddings sometimes predict broader or richer neural patterns than the SSDD ratings (Zhang et al., 29 Aug 2025). This is interpreted in the source as evidence that LLMs capture abstract conceptual structure especially well while remaining weaker for grounded sensorimotor semantics (Zhang et al., 29 Aug 2025).

The reported neural correlates are distributed rather than sharply localized (Zhang et al., 29 Aug 2025). Recurrent regions include left AG, left STG, MTG, IFG, CG, and Pcun (Zhang et al., 29 Aug 2025). Some subdimensions are described as overlapping in regions linked to autobiographical memory, value, and social reasoning; examples include collaboration/exchange, positive valence, and dynastic eras (Zhang et al., 29 Aug 2025). The paper also reports novel neural correlates for dynastic eras, historical change, micro-movements, and functional body acts (Zhang et al., 29 Aug 2025).

A plausible implication is that DCSRM is intended not only as an NLP disentangling model but also as a hypothesis-generating tool for semantic neuroscience.

6. Relation to earlier disentangled semantic representation research

1. Definition-role supervision and continuous semantic manifolds

“Learning Disentangled Representations for Natural Language Definitions” (Carvalho et al., 2022) is described as a strong precursor to what DCSRM would aim to do. It learns a continuous latent space for definitional sentences and explicitly encourages that space to separate semantic role factors such as Supertype, Differentia quality, Differentia event, Event location, Purpose, and Accessory quality (Carvalho et al., 2022). The paper uses VAE, DSR-supervised VAE, and Conditional VAE formulations, with objectives

$X = V W,$9

and reports that the proposed model outperforms the unsupervised baseline on 6 out of 8 disentanglement metrics (Carvalho et al., 2022). It also reports improvements in definition modeling, including WordNet perplexity values of 88.59 for the baseline DM, 80.36 for VAE, and 80.27 for DSR (Carvalho et al., 2022). The paper’s use of latent traversals, arithmetic, interpolation, and role-conditioned generation suggests a DCSRM-like view in which continuous latent geometry can be organized around explicit semantic factors (Carvalho et al., 2022).

2. Continuous and discrete channels in sentence embeddings

“Disentangling continuous and discrete linguistic signals in transformer-based sentence embeddings” (Nastase et al., 2023) tests whether transformer sentence embeddings can be compressed into a latent layer that separates continuous lexical-semantic information from discrete structural information (Nastase et al., 2023). Its crucial formulation is a joint latent layer with both continuous and discrete components,

$W \in \mathbb{R}^{h \times h}$0

with separate priors and decomposed KL terms (Nastase et al., 2023). The paper reports that the joint discrete+continuous latent layer gives the best F1 scores across both transformer encoders and both linguistic phenomena, and that adding a discrete component improves performance beyond a continuous vector of the same size (Nastase et al., 2023). This does not redefine DCSRM, but it provides a closely aligned representational argument: continuous semantic representation may need an explicit structural channel when the target phenomena are categorical or symbolic (Nastase et al., 2023).

3. Hierarchical sentence VAEs and factorized semantic slots

“Disentangling semantics in language through VAEs and a certain architectural choice” (Felhi et al., 2020) presents an unsupervised hierarchical VAE with three latent layers $W \in \mathbb{R}^{h \times h}$1 and modified Transformer encoder-decoder components (Felhi et al., 2020). The model reports latent variables associated with verbs, subjects, direct objects, and prepositional objects or secondary arguments, with LV 10, LV 30, LV 35, and LV 43 identified as particularly influential (Felhi et al., 2020). Swapping or resampling individual latent variables yields partial semantic swaps, which the paper treats as evidence of disentangled representation learning (Felhi et al., 2020). This work is conceptually adjacent to DCSRM in that it factorizes sentence meaning into continuous latent slots rather than decomposing word semantics into semantic subdimensions.

4. Disentangled semantic representation in domain adaptation

“Learning Disentangled Semantic Representation for Domain Adaptation” (Cai et al., 2020) decomposes observations into semantic latent variables $W \in \mathbb{R}^{h \times h}$2 and domain latent variables $W \in \mathbb{R}^{h \times h}$3, assumed independent and given Gaussian priors (Cai et al., 2020). A VAE reconstruction block is combined with a dual adversarial network so that $W \in \mathbb{R}^{h \times h}$4 becomes label-predictive and domain-invariant while $W \in \mathbb{R}^{h \times h}$5 becomes domain-specific and label-uninformative (Cai et al., 2020). The paper reports average accuracies of 88.6% on Office-31 and 64.9% on Office-Home (Cai et al., 2020). Although this is not a semantic subdimension model in the sense of (Zhang et al., 29 Aug 2025), it is relevant to DCSRM as a continuous disentangled latent-variable design that separates semantic from nuisance factors.

Across these works, a consistent theme is the replacement of undifferentiated embeddings with structured latent spaces whose components are semantically interpretable and manipulable.

7. Scope, significance, and limitations

1. What DCSRM contributes

Within this research line, DCSRM contributes a data-driven method for splitting a dense word embedding into multiple semantically specialized sub-embeddings, preserving the original semantic structure while isolating finer semantic subdimensions inside coarse dimensions such as vision, action, social, emotion, time, and space (Zhang et al., 29 Aug 2025). Its significance lies in three linked claims: semantic dimensions are not monolithic; polarity is a major organizing principle; and the discovered subdimensions have plausible neural correlates in distributed semantic networks (Zhang et al., 29 Aug 2025).

The framework also sits at an intersection of NLP and cognitive neuroscience. Earlier disentanglement work largely focused on sentence generation, grammatical probing, or transfer robustness (Carvalho et al., 2022, Nastase et al., 2023, Felhi et al., 2020, Cai et al., 2020). DCSRM extends the agenda by explicitly targeting fine-grained conceptual subdimensions and evaluating them with voxel-wise encoding models (Zhang et al., 29 Aug 2025).

2. Common misconceptions and boundaries

A common misconception would be to treat DCSRM as a purely unsupervised discovery model. In the reported formulation, it is not purely unsupervised: it uses supervised semantic prediction from human semantic ratings, contrastive separation based on thresholded ratings, and post hoc linguistic interpretation (Zhang et al., 29 Aug 2025). Another misconception would be to equate it with a discrete symbolic ontology. The model is explicitly continuous; it learns continuous semantic subspaces and interprets them afterward rather than assigning hard symbolic labels during training (Zhang et al., 29 Aug 2025).

The main limitation stated in the evidence base is scope. The DCSRM formulation in (Zhang et al., 29 Aug 2025) operates over coarse semantic dimensions derived from prior semantic ratings and over word embeddings from LLMs, while precursor models in (Carvalho et al., 2022, Nastase et al., 2023), and (Felhi et al., 2020) are specialized to definitional sentences, diagnostic grammatical phenomena, or sentence-level latent manipulation. This suggests that “DCSRM” is best understood as a specific framework for disentangled continuous semantic subspaces and, more broadly, as a family resemblance across methods that seek interpretable factorization of semantic information in continuous latent representations.