Subspace-Decoupled Multi-Task ViT

• The paper introduces a parameter-efficient ViT that uses low-rank adapters to decouple task-specific subspaces.
• Orthogonality regularization is applied to reduce negative gradient interference among correlated histological tasks.
• Empirical results on NAFLD scoring demonstrate improved accuracy over traditional single-task and shared-backbone models.

Searching arXiv for the primary paper and the related methods explicitly named in the provided data. Primary paper lookup: (Huang et al., 28 May 2026) Subspace-Decoupled Multi-Task Vision Transformer (ViT) denotes a parameter-efficient multi-task formulation for histological scoring in Non-Alcoholic Fatty Liver Disease (NAFLD) that combines a shared frozen Vision Transformer backbone with task-specific low-rank Adapters and an orthogonality-based regularizer on the Adapter subspaces. In the reported formulation, the method targets the three NAFLD Activity Score (NAS) components—steatosis, hepatocellular ballooning, and lobular inflammation—and is designed to mitigate negative transfer arising from strong inter-task correlation, task imbalance, and differing task difficulty. The approach was introduced in "Parameter-Efficient Subspace Decoupling ViT for Mitigating Multi-Task Negative Transfer in Histological Scoring" (Huang et al., 28 May 2026).

1. Problem setting and motivation

The method is motivated by multi-task learning (MTL) for patch-level NAFLD histology, where the NAS composite score is formed by three correlated components: steatosis, ballooning, and inflammation. The reported empirical observation is that these three indicators show strong pairwise correlations on both a curated dataset and a public cohort. In a naïve shared-backbone MTL setting, objectives with different difficulty or imbalance can co-optimize shared parameters, so that gradients from an easier or less noisy task, such as steatosis, dominate training and degrade harder or noisier tasks, such as inflammation. This correlation-driven interference is identified as the central form of negative transfer in this setting (Huang et al., 28 May 2026).

The formal task definition consists of three discrete multi-class classification problems following the standard NAS scoring system. Steatosis is defined as $S \in \{0,1,2,3\}$, corresponding to percent fat: $<5\%$ (0), $5$–$33\%$ (1), $34$–$66\%$ (2), and $>66\%$ (3). Ballooning is defined as $B \in \{0,1,2\}$: none (0), few ballooned cells (1), and numerous ballooned cells (2). Inflammation is defined as $I \in \{0,1,2,3\}$: none (0), $<2$ foci (1), $<5\%$0–$<5\%$1 foci (2), and $<5\%$2 foci (3). Each task is treated as a multi-class classification problem with cross-entropy loss unless otherwise noted.

A common misunderstanding is to treat the problem purely as one of insufficient model capacity. The reported framing is different: the central issue is destructive interference among correlated tasks within shared parameters. This suggests that the method is less about enlarging the network than about restructuring how task-specific adaptation occurs.

2. Architectural formulation

The architecture builds on a shared Swin Transformer Tiny (Swin-T) backbone pretrained on ImageNet-22k. Input H&E patches are resized to $<5\%$3 and normalized with ImageNet statistics. During multi-task training, the backbone is frozen to enable parameter-efficient adaptation. Shared low-level representations are therefore produced by the frozen encoder, while task-specific adaptation is deferred to lightweight components inserted near the output (Huang et al., 28 May 2026).

Task-specific adaptation is implemented through bottleneck Adapters inserted into the MLP branch of each block in the last Swin-T stage, together with per-task output heads. For each task $<5\%$4, the Adapter applies a residual low-rank mapping with layer normalization and GELU:

$<5\%$5

where $<5\%$6 is the incoming token or feature vector, $<5\%$7 is the down-projection, $<5\%$8 is GELU, and $<5\%$9 is the up-projection. The up-projection $5$0 is zero-initialized so that the Adapter starts near identity and gradually learns task-specific adjustments without disturbing the pretrained representation.

This residual design confines trainable parameters to $5$1 per Adapter per block per task. On top of the adapted shared features, each task uses its own head with the sequence LayerNorm $5$2 GELU-activated fully connected layer $5$3 dropout with $5$4 $5$5 final linear classifier producing task-specific logits. There is no cross-task fusion at the head. Interference is therefore addressed within the shared trunk through Adapter insertion and regularization rather than through explicit cross-head interaction.

The architectural division between a frozen shared trunk and task-specific "last-mile" adaptation preserves shared structure where it is beneficial while decoupling representational directions where tasks interfere. The paper presents this as the core engineering principle of the model.

3. Orthogonality-based subspace decoupling

The decoupling mechanism is defined through the Adapter up-projection matrices. For task $5$6, the paper takes

$5$7

whose columns span a low-dimensional task subspace in the ambient feature space. The key regularizer penalizes overlap between the subspaces associated with different tasks:

$5$8

Here $5$9 is the up-projection of task $33\%$0 at layer $33\%$1. Geometrically, $33\%$2 measures overlap between task subspaces, and driving it toward zero encourages near-orthogonal subspaces (Huang et al., 28 May 2026).

The intended effect is to reduce representational overlap and gradient interference. With correlated tasks, unconstrained sharing allows one task’s gradients to move features in directions that are harmful for another task. By encouraging orthogonal Adapter subspaces, the model constrains each task to update the shared representation along largely disjoint directions. At the same time, the frozen backbone preserves common low-level structure. A common misconception is that the method removes shared representation learning; the reported formulation does not do so. It retains a shared encoder and only decouples the small task-specific steering subspaces.

The paper explicitly distinguishes this approach from gradient-surgery methods such as PCGrad and RotoGrad. In that comparison, the difference is that gradient-surgery methods operate on gradients, whereas the present method reshapes representational geometry through subspace design and regularization. It is also related to orthogonal regularization and disentanglement, but the orthogonality is applied specifically to Adapter weights spanning task subspaces rather than to entire feature spaces or classifier weights. This suggests a targeted form of decoupling with lower parameter overhead.

4. Objective function and optimization

Each task $33\%$3 has a classification loss $33\%$4, with cross-entropy used by default and focal loss stated to be compatible. To address class imbalance and task difficulty, the model adopts homoscedastic uncertainty weighting with optional mild manual weights $33\%$5:

$33\%$6

where $33\%$7 and $33\%$8 are learned per-task noise variances. The implementation optimizes $33\%$9 directly with clamping for stability. In the main experiments, $34$0 for steatosis, ballooning, and inflammation, respectively, in order to mildly up-weight the rarer or more difficult tasks (Huang et al., 28 May 2026).

The full objective is

$34$1

The regularization strength $34$2 is tuned on validation data. The reported best value is $34$3, whereas $34$4 harms performance by over-decoupling. This establishes a central trade-off: insufficient decoupling leaves negative transfer unresolved, but excessive decoupling erodes beneficial sharing.

Optimization is performed with AdamW using learning rate $34$5, weight decay $34$6, batch size $34$7, and $34$8 epochs. Learning rate scheduling uses ReduceLROnPlateau with factor $34$9 on validation stall. The Swin-T backbone is frozen throughout, and only the task-specific Adapters, task heads, and $66\%$0 are trained. A global seed is fixed, each configuration is repeated three times, and averages are reported. The paper also provides concise pseudocode reflecting this training loop.

5. Dataset construction and preprocessing

The reported data source consists of H&E-stained whole-slide images from a controlled mouse NAFLD model based on a high-fat diet, scanned with a Hamamatsu scanner. All procedures were approved by an institutional ethics committee. Regions with clear NAFLD histology and full, unambiguous NAS annotations were extracted into $66\%$1 RGB patches. Patches containing artifacts or ambiguous grading were discarded (Huang et al., 28 May 2026).

After screening, the curated dataset contains 3,192 qualified patches. Annotations were reviewed by domain experts with consensus resolution. The training inputs are resized to $66\%$2 and normalized with ImageNet statistics. Augmentations include horizontal and vertical flips; earlier dataset expansion also used standard rotation and flip to increase robustness. Train/validation splits are fixed and shared across methods for fair comparison.

The reliance on expert-reviewed, consensus labels is methodologically important because the paper identifies label noise—especially for inflammation—as a factor that can amplify negative transfer. This suggests that the subspace-decoupling strategy addresses interference among tasks, but it does not remove the need for high-quality annotation.

6. Empirical results, ablations, and interpretation

The primary metric is per-task accuracy. On the curated NAFLD dataset, using the same multi-task heads and losses, the reported encoder comparison gives the following results: Swin-Tiny-22k achieves steatosis $66\%$3, ballooning $66\%$4, and inflammation $66\%$5; Swin-Small-22k achieves steatosis $66\%$6, ballooning $66\%$7, and inflammation $66\%$8; DINOv2 ViT-S achieves steatosis $66\%$9, ballooning $>66\%$0, and inflammation $>66\%$1. On the public Farzi NAFLD dataset, Swin-Tiny-22k achieves steatosis $>66\%$2, ballooning $>66\%$3, and inflammation $>66\%$4; Swin-Small-22k achieves steatosis $>66\%$5, ballooning $>66\%$6, and inflammation $>66\%$7; DINOv2 ViT-S achieves steatosis $>66\%$8, ballooning $>66\%$9, and inflammation $B \in \{0,1,2\}$0. The reported conclusion is that the framework is robust across backbones, with Swin-Tiny offering a favorable accuracy–size trade-off (Huang et al., 28 May 2026).

Against separate single-task InceptionV3 CNNs trained per NAS component, attributed in the paper to Farzi et al., the reported baseline scores are steatosis $B \in \{0,1,2\}$1, ballooning $B \in \{0,1,2\}$2, and inflammation $B \in \{0,1,2\}$3. The proposed multi-task subspace-decoupled ViT attains steatosis $B \in \{0,1,2\}$4, ballooning $B \in \{0,1,2\}$5, and inflammation $B \in \{0,1,2\}$6. The comparison is presented as evidence that a unified MTL model can outperform separate single-task models in this setting.

The orthogonality ablation isolates the effect of $B \in \{0,1,2\}$7. With the same backbone and Adapters, $B \in \{0,1,2\}$8 yields steatosis $B \in \{0,1,2\}$9, ballooning $I \in \{0,1,2,3\}$0, and inflammation $I \in \{0,1,2,3\}$1; $I \in \{0,1,2,3\}$2 yields steatosis $I \in \{0,1,2,3\}$3, ballooning $I \in \{0,1,2,3\}$4, and inflammation $I \in \{0,1,2,3\}$5; $I \in \{0,1,2,3\}$6 yields steatosis $I \in \{0,1,2,3\}$7, ballooning $I \in \{0,1,2,3\}$8, and inflammation $I \in \{0,1,2,3\}$9; $<2$0 yields steatosis $<2$1, ballooning $<2$2, and inflammation $<2$3. These results quantify negative transfer at $<2$4 and show that moderate orthogonality mitigates it, whereas excessive orthogonality damages performance by over-decoupling.

A second ablation compares Adapter-only and LoRA-only variants with the backbone still frozen. Adapter-only yields steatosis $<2$5, ballooning $<2$6, and inflammation $<2$7, whereas LoRA-only yields steatosis $<2$8, ballooning $<2$9, and inflammation $<5\%$00. Both are parameter-efficient, but Adapters are reported as the default because they show stronger overall performance in this setting.

The qualitative analysis examines Adapter activations in the last stage and reports partially disjoint, pathology-consistent attention across tasks: steatosis Adapters focus on lipid vacuoles, ballooning Adapters on swollen hepatocytes, and inflammation Adapters on inflammatory clusters. This does not by itself prove causal disentanglement, but it is presented as complementary evidence for the subspace-decoupling effect induced by $<5\%$01.

7. Efficiency profile, limitations, and prospective extensions

The efficiency claim is tied directly to freezing the backbone. Per task and per inserted layer, an Adapter adds approximately $<5\%$02 parameters, so if $<5\%$03 Adapter blocks are inserted, then

$<5\%$04

and across $<5\%$05 tasks the total is approximately $<5\%$06. The paper states that this is a small fraction of full fine-tuning. It further states that Adapters add only two thin linear projections and a GELU per token in the last stage, so backbone FLOPs remain dominant and unchanged. Training and inference time are therefore close to backbone-only, with reduced optimizer state and memory because the backbone is frozen (Huang et al., 28 May 2026).

The paper identifies several limitations. First, large $<5\%$07 can erode beneficial sharing and harm performance, so the method is sensitive to the regularization strength and to the Adapter rank $<5\%$08. Second, the approach relies on high-quality patch-level labels, and label noise especially affects inflammation. Third, the reported results are patch-level and based on a mouse model; slide-level aggregation and extensive validation on human, multi-center cohorts are stated as necessary for clinical translation.

The reported future directions include integrating additional endpoints such as fibrosis and clinical outcomes, exploring cross-layer Adapter placements, and combining the subspace-decoupling strategy with gradient-based conflict mitigation. The paper also states that the code and curated dataset have been prepared and will be made publicly available upon acceptance to support reproducibility. Within the scope presented, the principal takeaway is that explicit orthogonality on task-specific Adapter weights offers a compact mechanism for mitigating negative transfer in correlated histological tasks while preserving a shared transformer backbone.