Factorized Stream Embeddings

• Factorized stream embeddings are representation schemes that decompose inputs into coordinated streams, each capturing distinct factors like speaker identity, visual style, or modality.
• They enable separate modeling of diverse features, allowing for targeted processing and improved interpretability in tasks such as speaker verification and visual domain generalization.
• Empirical results show significant performance gains and efficiency improvements, though their effectiveness depends on matching the task’s inherent variations.

Searching arXiv for papers on factorized embeddings and related stream/state representations. Factorized stream embeddings are representation schemes in which an input, a latent state, or a structured description is decomposed into multiple coordinated streams aligned with distinct factors and later recombined, aligned, or projected for a downstream task. Recent work uses this pattern for speaker identity and phonetic content in text-dependent speaker verification, original and style-randomized state embeddings in visual domain generalization, factor-specific predictive states in transformer residual streams, byte identity and byte position inside token strings, audio and video emotion streams, and structured video scripts organized into Reference, Shot, Event, and Global streams (Yang et al., 6 Aug 2025, Bi et al., 10 Apr 2025, Shai et al., 2 Feb 2026, Shravan, 28 May 2026, Zhou et al., 2021, Team, 13 Apr 2026). This suggests that the phrase functions as an umbrella concept for a family of structured representations rather than a single standardized model.

1. Conceptual scope and recurring forms

Across the literature, factorized stream embeddings recur whenever a task requires simultaneous preservation of multiple kinds of information that would be harmful to collapse into one monolithic vector. In text-dependent speaker verification, the relevant factors are speaker identity and phrase content; in visual domain generalization, they are original and style-augmented state trajectories; in transformer representation theory, they are latent world factors encoded in orthogonal residual-stream subspaces; in token embedding design, they are byte-position events inside a token string; and in multimodal systems, they are modality-specific or schema-specific streams that remain linked by explicit grounding relations (Yang et al., 6 Aug 2025, Bi et al., 10 Apr 2025, Shai et al., 2 Feb 2026, Shravan, 28 May 2026, Zhou et al., 2021, Team, 13 Apr 2026).

A recurring architectural consequence is that the representation is no longer treated as a single undifferentiated embedding. Instead, it is either split into branches that are jointly optimized, mapped into aligned subspaces or latent paths, or serialized into typed streams that can be addressed independently. This suggests that “factorization” in this area is as much an organizational principle as a particular mathematical constraint.

Setting	Factorized streams or factors	Representative mechanism
Speech speaker verification	speaker and text/phonetic streams	branch extraction and recombination (Yang et al., 6 Aug 2025)
Visual domain generalization	original and style-augmented state streams	latent flow alignment (Bi et al., 10 Apr 2025)
Transformer residual streams	factor-specific predictive subspaces	orthogonal direct-sum geometry (Shai et al., 2 Feb 2026)
Token embeddings	byte identity and byte position	Kronecker codec and projection (Shravan, 28 May 2026)
Audio-visual fusion	audio and video streams	factorized bilinear pooling (Zhou et al., 2021)
Structured video semantics	Reference, Shot, Event, Global	relational grounding (Team, 13 Apr 2026)

2. Stream decomposition and recombination

A concrete neural instance appears in text-dependent speaker verification. The speaker-text factorization network in "Text adaptation for speaker verification with speaker-text factorized embeddings" decomposes an utterance into a text-independent speaker factor and a speaker-independent text factor, using four modules: a shared feature extractor $M_f$, a speaker branch $M_s$, a text branch $M_t$, and a combination network $M_c$. The input is 40-dimensional Fbank features; the shared front end uses the first three TDNN layers; the speaker branch has two TDNN layers, a statistics pooling layer, and two dense layers; the text branch has the same structure but predicts segment-level phoneme distributions; and the fused representation is formed by concatenation and passed to $M_c$, which has two common dense layers and two output heads (Yang et al., 6 Aug 2025). In the paper’s notation,

$$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$

The central point is that the streams are usable independently at inference time: the speaker stream can come from enrollment speech, while the text stream can be replaced by phrase-matched adaptation speech from other speakers.

An analogous recombination pattern appears in audio-visual emotion recognition, but there the streams are modality-specific rather than factor-specific. The audio branch yields attention-weighted local and global embeddings $\boldsymbol e_a^i$ and $\boldsymbol e_a^g$, and the video branch analogously yields $\boldsymbol e_v^i$ and $\boldsymbol e_v^g$. Fusion is not by concatenation alone but by factorized bilinear pooling, where the $M_s$0-th output is written as

$M_s$1

thereby preserving multiplicative cross-stream interactions while avoiding a full $M_s$2 bilinear tensor per output (Zhou et al., 2021). The same work extends this to adaptive weighting based on $M_s$3 and $M_s$4, and to multi-level fusion of synchronized local trunks.

A more symbolic but still structurally factorized design appears in "Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding". Its Multi-Stream Scene Script replaces one monolithic caption with Reference, Shot, Event, and Global streams. The streams are reconnected through persistent entity IDs, shared timestamps, and explicit fields such as references_in_shot, active_events, and speaker (Team, 13 Apr 2026). The paper does not define these as latent vectors, but the same factorized-stream logic holds: persistent information is centralized, local events are localized, and cross-stream consistency is enforced through explicit links.

3. Subspaces, flows, and invariant latent spaces

Not all factorized stream embeddings are built from parallel branches. In visual domain generalization, "DGFamba: Learning Flow Factorized State Space for Visual Domain Generalization" factorizes the state embeddings of each VMamba layer into paired latent-flow descriptions. For a layer output $M_s$5, the method creates a style-randomized counterpart $M_s$6 using AdaIN-style replacement of channel statistics, encodes both with a VAE into $M_s$7 and $M_s$8, and then aligns them through a $M_s$9-step prior/posterior latent probability path constrained by a Hamilton–Jacobi loss. The disclosed training objective is

$M_t$0

Here factorization means that original and stylized states are forced to share a common latent transport structure rather than merely being close in feature space (Bi et al., 10 Apr 2025).

A related but more theoretical formulation is given in "Transformers learn factored representations". The paper studies contextual predictive states in the residual stream under a generalized hidden Markov model and contrasts two hypotheses: a joint representation in the full tensor-product space and a factored representation in orthogonal residual-stream subspaces. Under the Factored World Hypothesis,

$M_t$1

with dimensionality

$M_t$2

The factored representation is lossless on the product-state manifold when token-conditioned latent updates satisfy $M_t$3 (Shai et al., 2 Feb 2026). In this setting, stream factorization is a geometric statement about how predictive information is laid out in residual space.

Streaming graph embedding introduces yet another meaning of factorization. "Space-Invariant Projection in Streaming Network Embedding" assumes a static matrix-factorized embedding $M_t$4, then reuses the original latent basis to generate embeddings for newly arriving nodes. If $M_t$5 is a new target row against the original nodes, the embedding is

$M_t$6

The validity of this projection is tied to a perturbation-theoretic condition involving $M_t$7 and the spectral gap, which yields a restart threshold $M_t$8 for how many new nodes can be added before the latent space is no longer approximately invariant (Zhang et al., 2023). Here factorization is not multi-branch encoding but reuse of a fixed factorized latent space under controlled drift.

4. Token, table, and lookup factorization

A distinct line of work factorizes the embedding process itself. "Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient LLMs" replaces the usual $M_t$9 learned embedding table with a deterministic byte-position codec followed by a single learned projection. For a token byte sequence $M_c$0, the codec is

$M_c$1

where $M_c$2 is a one-hot byte vector and $M_c$3 is a one-hot position vector. The final token embedding is $M_c$4 (Shravan, 28 May 2026). The factorization is over the internal byte stream of each token, not over token IDs through a learned low-rank latent code.

"Training with Multi-Layer Embeddings for Model Reduction" factorizes categorical lookup tables during training rather than at deployment. In its two-layer scheme,

$M_c$5

and the lookup is $M_c$6. When $M_c$7, the paper proves that this factorized parameterization has the same search space as a direct $M_c$8 embedding matrix, yet induces a different optimization bias; after training, the factors are collapsed into $M_c$9 (Ghaemmaghami et al., 2020). In this usage, factorization is a training scaffold that shapes singular-value dynamics.

"Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition" uses a low-rank shared input/output factorization upgraded with a nonlinear bottleneck and embedding-level distillation. Starting from $M_c$0, the compressed form is

$M_c$1

with $M_c$2, reconstruction initialization, and a combined fine-tuning loss

$M_c$3

The method is explicitly designed for shared input/output embeddings (Lioutas et al., 2019).

"DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling" generalizes shallow projective factorization into a deep Map–Expand–Reduce pipeline. A token is first mapped into $M_c$4, expanded through $M_c$5 hierarchical grouped transforms, and then reduced to $M_c$6. The group schedule is

$M_c$7

and the architecture uses skip-connections from the original low-dimensional input to every expansion layer (Mehta et al., 2019). This formulation treats token embedding as a deep factorized transformation rather than a flat lookup.

5. How factorization is learned or enforced

The mechanisms that produce factorized stream embeddings differ sharply across papers. In the speaker-text factorization network, factorization is encouraged structurally and by supervision rather than by explicit independence constraints. The total loss is

$M_c$8

with cross-entropy for speaker classification and Kullback–Leibler divergence for segment-level phoneme-distribution prediction. There is no adversarial loss to remove speaker information from the text branch or text information from the speaker branch; instead, a particularly important operational mechanism is random pairing of $M_c$9, so that the model often combines a speaker embedding from one utterance with a text embedding from another unrelated utterance (Yang et al., 6 Aug 2025). The resulting separation is therefore functional rather than provably disentangled.

DGFamba enforces factorization through controlled perturbation and path alignment. State Style Randomization creates paired original and hallucinated state streams, State Flow Encoding maps both into latent flow space, and State Flow Constraint applies the Hamilton–Jacobi regularizer per layer. The paper explicitly notes that its final objective does not include a VAE reconstruction loss, KL divergence, contrastive loss, MMD, adversarial alignment, or direct consistency loss; the disclosed objective is classification plus the summed $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$0 terms (Bi et al., 10 Apr 2025). This makes factorization a path-regularized invariance mechanism rather than an explicit content/style code separation.

In the transformer setting, factorization is neither a branch architecture nor an auxiliary regularizer. It is a representational hypothesis grounded in conditional independence of latent updates and tested by linear probes, cumulative explained variance, and subspace-overlap analysis. Orthogonality is predicted between factor subspaces, whereas the geometry within each factor subspace remains generally non-orthogonal (Shai et al., 2 Feb 2026). The paper is explicit that the observed result is approximate orthogonality of dominant directions, not a perfect algebraic partition of every residual dimension.

Compression-oriented approaches use different enforcement principles again. Kronecker Embeddings hard-code the factorization by a deterministic codec over byte-position pairs (Shravan, 28 May 2026); Distilled Embedding imposes a low-rank nonlinear bottleneck with reconstruction and embedding-level distillation (Lioutas et al., 2019); DeFINE uses grouped hierarchical transforms with skip-connected deep synthesis (Mehta et al., 2019); and adaptive multi-level factorized bilinear pooling factorizes only the cross-modal interaction matrix, not the unimodal encoders themselves (Zhou et al., 2021). This heterogeneity is central: in the literature, “factorization” may refer to supervised branch separation, low-rank interaction structure, latent-path alignment, deterministic compositional coding, or orthogonal subspace geometry.

6. Empirical behavior, tradeoffs, and limitations

The best-documented gains arise when the factorized streams match the task’s nuisance structure. In text-dependent speaker verification, the factorization network’s speaker branch already improves text-independent verification over a standard TDNN x-vector baseline: on VoxCeleb1-O/E/H, EER drops from $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$1 to $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$2, and on an RSR2015-derived text-independent set from $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$3 to $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$4. Under text-dependent mismatch, the decisive gain comes from recombining speaker and text streams: on RSR2015, baseline TDNN speaker embeddings give $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$5 EER, the factorization network’s speaker embedding gives $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$6, and the integrated “spk+text” representation drops to $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$7, with minDCF improving from $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$8 to $$\mathbf{s}=M_s(M_f(\mathbf{x}_s)), \qquad \mathbf{t}=M_t(M_f(\mathbf{x}_t)), \qquad \mathbf{z}=M_c([\mathbf{s},\mathbf{t}]).$$9. In the harder enrollment-test mismatch condition, average EER remains around $\boldsymbol e_a^i$0 without proper adaptation, but “spk+adapt_text” reduces it to $\boldsymbol e_a^i$1 / $\boldsymbol e_a^i$2 under the two enrollment conditions reported (Yang et al., 6 Aug 2025). The limitation is equally explicit: current adaptation requires target-phrase audio from other speakers, specifically 10 utterances, and the authors state that future work should use “simple plain text instead of the text embedding computed from specific audios.”

In visual domain generalization, DGFamba’s ablations isolate the contribution of latent flow factorization. On PACS, vanilla VMamba gives $\boldsymbol e_a^i$3 average accuracy; adding only SSR yields $\boldsymbol e_a^i$4; SSR+SFE yields $\boldsymbol e_a^i$5; and full SSR+SFE+SFC reaches $\boldsymbol e_a^i$6. Block-wise application improves monotonically from $\boldsymbol e_a^i$7 with $\boldsymbol e_a^i$8 only to $\boldsymbol e_a^i$9 with $\boldsymbol e_a^g$0, and full-benchmark comparisons report $\boldsymbol e_a^g$1 vs. $\boldsymbol e_a^g$2 on PACS, $\boldsymbol e_a^g$3 vs. $\boldsymbol e_a^g$4 on VLCS, $\boldsymbol e_a^g$5 vs. $\boldsymbol e_a^g$6 on OfficeHome, and $\boldsymbol e_a^g$7 vs. $\boldsymbol e_a^g$8 on TerraIncognita relative to DGMamba (Bi et al., 10 Apr 2025). The paper also states that implementation details remain sparse: VAE architecture, latent dimensionality, exact tensor shapes, and the exact interface to VMamba’s selective scan are omitted.

The transformer study provides a different empirical signature: in the main independent-factors setting, the joint predictive geometry would vary in $\boldsymbol e_a^g$9 dimensions, whereas the factored geometry varies in $\boldsymbol e_v^i$0 dimensions, and the model’s effective dimensionality collapses during training to about $\boldsymbol e_v^i$1–$\boldsymbol e_v^i$2 dimensions (Shai et al., 2 Feb 2026). Kronecker Embeddings provide an engineering-oriented counterpart: in a controlled nanoGPT GPT-2 124M comparison over 2.5B FineWeb-Edu tokens, Kronecker reaches $\boldsymbol e_v^i$3 lower validation loss, with gap $\boldsymbol e_v^i$4 nats and roughly $\boldsymbol e_v^i$5 lower validation perplexity, and needs about $\boldsymbol e_v^i$6 fewer optimizer steps to reach the BPE baseline’s converged loss. On 110 clean/typo pairs, top-1 prediction is preserved on $\boldsymbol e_v^i$7 of pairs versus $\boldsymbol e_v^i$8 for BPE, with mean KL $\boldsymbol e_v^i$9 vs. $\boldsymbol e_v^g$0 (Shravan, 28 May 2026). The stated tradeoff is byte-level locality: byte-similar but semantically distant pairs such as compute/commute and nation/notion cluster together, shifting disambiguation to early attention layers.

Multimodal and structured-stream systems show comparable patterns. In audio-visual emotion recognition, G-FBP improves AFEW validation accuracy from $\boldsymbol e_v^g$1 for audio and $\boldsymbol e_v^g$2 for video to $\boldsymbol e_v^g$3, while the ablation ladder $\boldsymbol e_v^g$4 for G-FBP, AG-FBP, M-FBP, and AM-FBP shows incremental value from adaptive weighting and multi-level local fusion. Final test accuracy reaches $\boldsymbol e_v^g$5 on AFEW and $\boldsymbol e_v^g$6 on IEMOCAP, exceeding decision fusion ($\boldsymbol e_v^g$7) and encoder concatenation ($\boldsymbol e_v^g$8) on IEMOCAP (Zhou et al., 2021). In structured video semantics, MTSS is reported to reduce total error rate by an average of $\boldsymbol e_v^g$9 on Video-SALMONN-2 and improve performance by an average of $M_s$00 on the Daily-Omni reasoning benchmark, while prompt replacement alone in multi-shot generation yields a $M_s$01 boost in cross-shot identity consistency, a $M_s$02 boost in audio-visual alignment, and a $M_s$03 boost in temporal controllability (Team, 13 Apr 2026). The paper is equally clear that MTSS is primarily a structured textual representation, not an explicit learned latent embedding model.

For streaming graphs, SIP demonstrates that reuse of a factorized latent basis can be highly efficient when the space remains approximately invariant. SIP-AROPE achieves around $M_s$04, $M_s$05, and $M_s$06 of retrained micro-F1 on Blogcatalog, PPI, and Flickr, using only $M_s$07, $M_s$08, and $M_s$09 of retraining time; SIP-NetMF reaches around $M_s$10, $M_s$11, and $M_s$12 of retrained micro-F1, and exceeds retraining on macro-F1 for Blogcatalog and Flickr with $M_s$13 and $M_s$14, respectively (Zhang et al., 2023). The limitation is architectural scope: the method is geared toward new node insertion under bounded drift, and the threshold $M_s$15 is a sufficient, conservative restart guide rather than an exact failure boundary.

Taken together, these results show that factorized stream embeddings are most effective when the decomposition mirrors the task’s actual sources of variation and when the recombination rule is operationally usable at training or inference time. The same record also shows that “factorization” should not be conflated with strict disentanglement, exact orthogonality, or universal portability across domains.