StrTransformer: Source-Wise Structured Transformers for Unsupervised Blind Source Recovery

Abstract: This paper proposes StrTransformer, a source-wise structured Transformer framework for blind source recovery and branch-wise latent modeling. Instead of using an encoder to infer latent variables, StrTransformer directly optimizes the latent source matrix together with an observation-space mixer and source-wise structural Transformer branches. The mixer enforces reconstruction consistency, while each Transformer branch imposes a differentiable structural constraint on one latent source trajectory. Specifically, each source is converted into multi-scale patch tokens, randomly masked, processed by a locality-biased Transformer, and evaluated through a masked patch reconstruction energy. This energy acts as an implicit source-wise structural prior. To encourage different latent branches to specialize into different temporal regimes, StrTransformer further introduces an ordered multi-scale controller that learns branch-specific patch-scale weights, ordered scale centers, and locality attention slopes. The resulting objective combines observation reconstruction, source-wise structural regularization, and modular auxiliary penalties for separation and scale specialization. We analyze the decoupling and coupling structure of the objective, the regularized exact-reconstruction fiber, and the reduction of permutation symmetry induced by ordered branch descriptors. A controlled case study shows that the learned branches converge to distinct temporal-scale structures and recover source-aligned latent trajectories under post-hoc evaluation.

• The paper proposes a novel Transformer-based approach for unsupervised blind source separation by imposing source-wise structural regularization.
• It utilizes masked patch tokenization and a locality-biased encoder to enforce distinct temporal regimes and reduce latent permutation symmetry.
• Experimental results demonstrate rapid convergence and high recovery fidelity, offering a flexible framework adaptable to nonlinear source recovery.

StrTransformer: Source-Wise Structured Transformers for Unsupervised Blind Source Recovery

Motivation and Context

Blind Source Separation (BSS) is a central problem in unsupervised learning where the objective is to recover latent source signals from observed mixtures absent explicit knowledge of their underlying structure or mixing dynamics. Classical ICA relies on independence and non-Gaussianity for identifiability in linear settings; however, such assumptions become insufficient for nonlinear mixtures and more complex source structures. From the broader perspective of representation learning, the goal extends toward achieving decoupled and potentially identifiable latent representations, with BSS serving as a verifiable substrate for latent modeling advancements.

The limitations of classical formulations, especially in nonlinear scenarios where independence alone is non-identifying [hyvarinen1999nonlinear], underscore the need for additional structural inductive biases. Recent studies have demonstrated that temporal dependence, nonstationarity, and source-specific structural regularization provide partial routes to identifiability in unsupervised settings [hyvarinen2016tcl, khemakhem2020ivae, halva2021snica]. Deep latent-variable models (e.g., VAEs, $\beta$-VAE) improve flexibility, but standard isotropic priors do not inherently promote source-wise structural specialization.

Transformer architectures, with contextual patch aggregation, have proven effective for sequence modeling and self-supervised learning [vaswani2017attention, devlin2019bert, he2022mae]. Extending these principles to latent modeling necessitates architectural innovations that encourage each latent dimension to capture structurally unique source trajectories, breaking exchangeability and enhancing separability.

StrTransformer Architecture

StrTransformer synthesizes a source-wise structured Transformer framework for BSS and branch-wise latent modeling, directly optimizing the source matrix, mixer, and structural Transformer branches. The mixer is agnostic to its form (affine or nonlinear) and enforces reconstruction consistency in the observation space, while the source-wise Transformer branches impose differentiable structural regularization on individual source trajectories via masked patch reconstruction energy. Each latent source is processed as follows:

• Patch Tokenization and Masking: Each trajectory is converted to multi-scale patch tokens, which are randomly masked.
• Locality-Biased Transformer Processing: Masked patches are processed by a source-specific Transformer encoder with controllable locality bias.
• Implicit Structural Energy: The masked patch reconstruction energy acts as an implicit, differentiable prior for individual sources.

A novel ordered multi-scale controller is central to StrTransformer. It learns branch-specific patch-scale weights, ordered scale centers, and locality attention slopes—encouraging ordered branch specialization into distinct temporal regimes, thereby reducing latent permutation symmetry and promoting stable structural assignment.

Objective Formulation

The StrTransformer objective consists of three principal terms:

• Observation Reconstruction: Ensures that the sources and mixer jointly explain the observed data via quadratic loss, weighted by a fidelity coefficient $\nu_y$.
• Source-Wise Structural Regularization: Implements masked patch reconstruction energies for each branch at multiple temporal scales, aggregated via learned soft scale-selection weights.
• Modular Auxiliary Penalties: Includes source decorrelation, smoothness, scale entropy, and ordered scale-gap penalties as adjustable terms to further bias the model towards separation and specialization.

The flexibility in regularization allows for tailored adaptation to specific data settings and separation goals. The optimization decouples source-wise structural curvature from coupling induced by observation reconstruction, providing a distinct mechanism for selecting structurally coherent decompositions from the exact-reconstruction fiber.

Theoretical Analysis

StrTransformer provides both theoretic and practical mechanisms for permutation symmetry reduction. The ordered structural descriptors assigned to each branch restrict latent coordinate exchangeability. Only permutations preserving all branch descriptors are valid, and with pairwise distinct descriptors, permutation symmetry collapses—yielding stable source-specific assignments.

For linear ICA, joint diagonalization of source autocorrelation matrices under temporal contrasts achieves source recovery up to signed permutation. The structural assignment margin induced by ordered branches resolves this ambiguity, ensuring source-aligned recovery.

For nonlinear ICA, exact reconstruction and injectivity of the mixer guarantee invertible latent-source mapping. If source-wise conditional modeling via masked context factorizes appropriately and provides full-rank modulation of sufficient statistics, StrTransformer's objective enforces component-wise invertible mappings and resolves permutation ambiguity via its ordered structural controller.

Experimental Results

A case study with $K=3$ synthetic sources ($T=1000$) demonstrates the empirical properties of StrTransformer:

• Numerical Stability: The objective shows rapid convergence, stabilizing at a low reconstruction and structural loss.
• Branch Specialization: Ordered multi-scale controller yields distinct expected patch scales, log-scale centers, and locality slopes, confirming that branch structure is learned (rather than prescribed).
• Source Recovery: Mean absolute matched correlation approaches unity, with post-hoc aligned recovered sources closely matching reference trajectories across time. Notably, recovery fidelity is slightly below GP-kernel-related source-wise methods [wei2026stradiff, wei2026strebm] when data is highly smooth, reflecting StrTransformer's general-purpose structural regularization and the flexibility of contextual masked reconstruction.

Implications and Future Directions

StrTransformer operationalizes source-wise Transformer regularization for BSS and structured latent modeling. The implications are two-fold:

• Practical: The architecture provides a flexible template for unsupervised source recovery, adaptable to varied mixing and source characteristics, and offers stable branch assignment through ordered structural descriptors. The modular penalty scheme allows for domain-specific tailoring.
• Theoretical: The reduction of latent exchangeability via ordered multi-scale controllers is a substantial advancement for identifiable representation learning. It anchors the development of more robust disentangled modeling frameworks in unsupervised regimes where classical independence-induced identifiability fails.

For future developments, research should examine StrTransformer's efficacy on nonsmooth sources, nonstationary processes, complex real-world measurements, and adaptive determination of latent factor number and structures. The extension of ordered branch-wise regularization to multifactor and hierarchical latent modeling tasks is a promising avenue for advancing identifiable unsupervised learning.

Conclusion

StrTransformer introduces a source-wise structured Transformer framework for unsupervised blind source recovery, combining an encoder-free direct optimization of latent sources and mixer with differentiated structural regularization through masked patch reconstruction energies and ordered multi-scale controllers. This approach enables branch specialization and reduces latent symmetry, facilitating source-aligned recovery as demonstrated empirically. The framework constitutes a flexible and modular foundation for both practical blind source separation and theoretical advancements in identifiable latent representation learning—particularly in scenarios with nonlinear mixing and unknown latent factor structures (2605.25648).