Cross-Scale Adaptive Feature Disentangling

• Cross-scale adaptive feature disentangling is a representation-learning strategy that separates invariant structures from modality-specific variations across multiple resolutions.
• It employs specialized modules—such as CDAP in registration and DAT in adaptation—to suppress leakage of private cues and enhance multi-scale alignment.
• Empirical results demonstrate significant performance gains in registration accuracy and domain transfer, validating the effectiveness of integrated disentanglement and scale-aware losses.

Cross-scale adaptive feature disentangling denotes a class of representation-learning strategies in which invariant structure is separated from domain-, modality-, or appearance-specific variation while maintaining consistency across multiple resolutions. In recent work, the concept appears in two closely related but task-distinct forms. In multimodal image registration, it is formulated as jointly learning a stable shared feature space and a unified hybrid transformation, exemplified by the Hybrid Registration Network (HRNet) with its Cross-scale Disentanglement and Adaptive Projection (CDAP) module (Zhang et al., 20 Mar 2026). In unsupervised domain adaptation, it appears as the combination of content/style disentangling and scale-aware alignment, exemplified by the Domain-Adaptive Cross-Scale Matching (DACSM) framework, which combines a Domain-Adaptive Transformer (DAT) with a Cross-Scale Matching (CSM) module (Zang et al., 18 Mar 2026). Across both formulations, the central objective is to suppress nuisance variation without erasing task-relevant geometry or semantics.

1. Problem setting and motivating limitations

In multimodal image registration, the stated objective is to align images drawn from different sensing modalities so that downstream cross-modal analysis becomes feasible. The registration literature summarized by HRNet identifies two unresolved limitations: some methods use disentanglement to learn shared features but mainly regularize the shared part, allowing modality-private cues to leak into the shared space; and most multi-scale frameworks support only a single transformation type, limiting applicability when global misalignment and local deformation coexist (Zhang et al., 20 Mar 2026).

In unsupervised domain adaptation, DACSM starts from a related but not identical diagnosis. Existing cross-attention-based transformers can align features across domains, yet they struggle to preserve content semantics under large appearance and scale variations. The reported failure mode is not only a domain gap but also a scale gap, which degrades transfer performance when object size differs substantially between source and target (Zang et al., 18 Mar 2026).

Taken together, these formulations place disentanglement and scale handling at the center of alignment rather than treating them as secondary regularizers. A plausible implication is that “cross-scale” is not merely a multiresolution design choice; it is the mechanism by which invariant structure is stabilized against drift across levels of abstraction.

2. Shared architectural pattern

Both frameworks instantiate cross-scale adaptive feature disentangling through a separation between invariant and private information, followed by a scale-aware alignment mechanism. The invariant component is geometry-centric in registration and content-centric in domain adaptation; the private component corresponds to modality-specific appearance, noise, texture, or style.

Framework	Invariant/private split	Scale-aware alignment mechanism
HRNet	Shared backbone with Modality-Specific Batch Normalization; shared and modality-private encoders	Cross-scale attention gating, dynamic shared subspace projection, coarse-to-fine hybrid parameter prediction
DACSM	Query as domain-invariant content; Key/Value as domain-specific style	Cross-scale matching through predefined scale factors and scale-aware sub-centers

In HRNet, multi-scale feature maps extracted from fixed and moving images are processed at each scale by a “Decompose–Gate–Project” pipeline. In DACSM, transformer cross-attention is asymmetrically organized so that the Query stream captures domain-invariant content while the Key/Value streams carry domain-specific style, after which a scale-conditioned classifier resolves scale mismatch (Zhang et al., 20 Mar 2026, Zang et al., 18 Mar 2026).

This structural parallel is notable because the two systems address different tasks. HRNet targets rigid and non-rigid multimodal registration, whereas DACSM targets category-level domain transfer. The commonality lies in the assumption that alignment is more stable when the latent space explicitly distinguishes what must be preserved from what should be ignored.

3. Mechanisms of disentanglement across scales

In HRNet, CDAP operates on multi-scale feature sets

$$F=\{F_0,F_1,\dots,F_L\},\qquad M=\{M_0,M_1,\dots,M_L\},$$

with $L=4$ in the implementation. At each scale $i$, the feature maps are decomposed into shared and private parts through separate encoders:

$$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$

The shared encoder $E_{sh}^i$ shares convolutional weights across the two modalities so as to extract geometry-centric cues, while $E_{pf}^i$ and $E_{pm}^i$ learn appearance/noise that is private to each modality. CDAP then applies Inter-Layer Disentanglement Attention (ILDA), a cross-scale self-attention mechanism on the shared branch, and an analogous process on the private branch, to produce gating masks $\alpha_i^s$ and $\alpha_i^p$. These masks suppress private-to-shared leakage:

$$\widetilde F_i^s = \alpha_i^s \odot F_i^s - \gamma^i(\alpha_i^p \odot F_i^p),\qquad \widetilde M_i^s = \alpha_i^s \odot M_i^s - \gamma^i(\alpha_i^p \odot M_i^p),$$

where $L=4$0 is a small learnable scalar. The gated features are then projected into a dynamically generated low-dimensional stable basis:

$L=4$1

Here $L=4$2 is a tiny MLP that produces an approximately orthonormal basis for the shared subspace (Zhang et al., 20 Mar 2026).

In DACSM, the disentangling mechanism is embedded in transformer cross-attention. The empirical observation reported by the method is that the Query stream tends to capture domain-invariant content, whereas the Key/Value streams carry domain-specific style. DAT explicitly enforces this split by always generating $L=4$3 from one domain and $L=4$4 from the other, while a residual connection preserves $L=4$5. Beneficial noise is injected into the style streams:

$L=4$6

$L=4$7

and cross-attention becomes

$L=4$8

Because $L=4$9 remains unperturbed, content semantics stay stable, while noise on $i$0 acts as a regularizer that encourages the model to ignore fine-grained style artifacts. Stacking layers yields progressively translated representations such as

$i$1

with the analogous construction for $i$2 (Zang et al., 18 Mar 2026).

The two mechanisms differ in implementation but converge on the same operational principle: disentanglement is not achieved solely by partitioning channels or streams; it is enforced by an additional operation that actively filters or perturbs private information before matching.

4. Loss design and stabilization of the shared space

HRNet couples CDAP with four disentanglement losses in addition to standard registration losses. Cross-Covariance Decorrelation, $i$3, forces shared and private channels to be statistically uncorrelated. Basis Orthogonality, $i$4, constrains each projection basis through

$i$5

Cross-Scale Directional Consistency, $i$6, encourages adjacent scales to point in similar semantic directions. Shared vs Private Triplet Loss, $i$7, pulls shared features from the two modalities together while pushing them away from private features. These terms combine as

$i$8

The stated purpose is to ensure that the shared subspace discards modality-private cues, projection bases remain diverse and non-degenerate, and shared features vary smoothly across scales (Zhang et al., 20 Mar 2026).

DACSM organizes its objective around a shared classifier $i$9 and a scale-aware classifier $$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$0. DAT uses source classification on both original and translated source features, target distillation with a KL term between $$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$1 and $$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$2, optional pseudo-label supervision on the target domain, and a style perceptual alignment term based on channel mean and standard deviation. The total DAT loss is

$$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$3

CSM adds a cross-scale classification loss $$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$4 over predefined source scales. The overall training objective is

$$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$5

For unlabeled target features, scale is treated as latent and resolved through a maximum over sub-centers:

$$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$6

Theorem 1 in the appendix is stated to prove that, under mild smoothness assumptions, this “$$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$7 over $$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$8” recovers the correct scale in expectation (Zang et al., 18 Mar 2026).

A shared methodological theme is the use of explicit regularization to prevent degenerate invariance. In HRNet, the danger is contamination of the shared registration space by modality-private cues. In DACSM, the danger is semantic collapse or style leakage under cross-attention and unresolved scale ambiguity.

5. Task-specific realizations

In HRNet, the output of CDAP feeds a non-iterative, coarse-to-fine Hybrid Parameter Prediction Module (HPPM). Using the paired shared features $$F_i^s = E_{sh}^i(F_i),\quad F_i^p = E_{pf}^i(F_i),\quad M_i^s = E_{sh}^i(M_i),\quad M_i^p = E_{pm}^i(M_i).$$9, HPPM jointly estimates global rigid parameters $E_{sh}^i$0—rotation, translation, and scale—and a local nonrigid deformation field $E_{sh}^i$1. At the coarsest level, fused features are used by a rigid head to predict $E_{sh}^i$2, which is decoded into a coarse flow $E_{sh}^i$3, and by a nonrigid head to predict $E_{sh}^i$4; the initial deformation is $E_{sh}^i$5. At each finer scale, the current deformation is upsampled, the moving shared feature is warped, a new fused feature is formed, a nonrigid increment is predicted, and the flow is accumulated until $E_{sh}^i$6 becomes the final hybrid deformation. The design explicitly avoids an external RANSAC or iterative solver: rigid and nonrigid estimation are learned end-to-end in a single forward pass, and the global and local components are coupled by simple vector addition in a common flow space (Zhang et al., 20 Mar 2026).

In DACSM, scale adaptation is handled by the CSM module. A predefined set of scale factors $E_{sh}^i$7 is applied to each source image, producing rescaled inputs $E_{sh}^i$8 whose features $E_{sh}^i$9 and $E_{pf}^i$0 are extracted by the shared DAT backbone. The classifier is extended to a scale-aware form $E_{pf}^i$1. Source supervision uses the known scale index $E_{pf}^i$2, whereas target adaptation applies the maximum over candidate scales. At inference time, the model processes target images only at original resolution and predicts

$E_{pf}^i$3

which is stated to add zero extra inference cost beyond a standard ViT (Zang et al., 18 Mar 2026).

These realizations illustrate that cross-scale adaptive feature disentangling is not tied to a single prediction head. In registration it produces a coherent deformation field; in domain adaptation it produces a scale-resolved class decision. The common substrate is the stabilization of an invariant latent representation across scales before downstream estimation.

6. Empirical results, misconceptions, and methodological implications

HRNet reports extensive experiments on four multimodal datasets: RGB–NIR (Brown et al. ’11), RGB–TIR (TBBR ’10), and two remote-sensing pairs RGB–IR and RGB–SAR (MRSR dataset), with over 3,000 training and 300 test image pairs each. Across all four tasks and both rigid and nonrigid benchmarks, HRNet+CDAP+HPPM yields the lowest RE and highest NCC. On RGB–TIR, rigid RE is reported as $E_{pf}^i$4 versus $E_{pf}^i$5 for the prior best MCNet, and nonrigid RE as $E_{pf}^i$6 versus $E_{pf}^i$7 for MMRNet. The module ablation on rigid RE gives Full HRNet: $E_{pf}^i$8 on RGB–TIR/RGB–SAR, w/o MSBN: $E_{pf}^i$9, and w/o CDAP: $E_{pm}^i$0, leading to the stated conclusion that CDAP alone contributes roughly a $E_{pm}^i$1 gain in RE on TIR and $E_{pm}^i$2 on SAR. The loss ablation on nonrigid RE reports: no disentangle losses $E_{pm}^i$3, then $E_{pm}^i$4 $E_{pm}^i$5, $E_{pm}^i$6 $E_{pm}^i$7, $E_{pm}^i$8 $E_{pm}^i$9, and $\alpha_i^s$0 $\alpha_i^s$1, corresponding to a $\alpha_i^s$2 drop in nonrigid RE on TIR and $\alpha_i^s$3 on SAR (Zhang et al., 20 Mar 2026).

DACSM reports experiments on VisDA-2017, Office-Home, and DomainNet, with the strongest quantitative details given for VisDA-2017. In the reported ablation, the CDTrans baseline achieves $\alpha_i^s$4; adding only DAT yields $\alpha_i^s$5; adding only CSM yields $\alpha_i^s$6; and the full DACSM reaches $\alpha_i^s$7, which is described as up to $\alpha_i^s$8 over CDTrans on VisDA-2017. The “truck” category shows a $\alpha_i^s$9 absolute gain over CDTrans. Qualitative evidence is also reported: reconstructions from $\alpha_i^p$0 preserve source content and adopt target style more cleanly than CDTrans, and t-SNE plots show tighter intra-class clusters and larger inter-class margins for target embeddings (Zang et al., 18 Mar 2026).

Two misconceptions are directly challenged by these results. First, shared feature extraction alone is not sufficient for reliable cross-modal or cross-domain alignment; both papers argue that private cues can leak into the purportedly shared representation unless they are explicitly suppressed or regularized. Second, ordinary multi-scale processing or random crops are not presented as adequate responses to scale variability. HRNet argues that multi-scale frameworks limited to a single transformation type are insufficient when global misalignment and local deformation coexist, while DACSM states that standard random crops or data augmentation only partially address large source-target scale gaps.

A plausible implication is that cross-scale adaptive feature disentangling is best understood as a compound design principle rather than a single module type. Its characteristic elements are: an explicit partition between invariant and private information, a mechanism that prevents the private component from dominating alignment, and a scale-aware objective that ties representations across resolutions to the downstream task.