Parameter-Efficient Subspace Decoupling ViT for Mitigating Multi-Task Negative Transfer in Histological Scoring

Abstract: Histological scoring is essential for diagnosing Non-Alcoholic Fatty Liver Disease (NAFLD), yet its automation remains challenging due to the high annotation cost and negative transfer among the strongly correlated NAFLD Activity Score (NAS) indicators in multi-task learning. To address this issue, we propose a subspace-decoupled multi-task Vision Transformer (ViT) that integrates lightweight task-specific Adapters with orthogonality-based constraints. This design constructs independent feature subspaces for steatosis, ballooning, and inflammation, effectively reducing task interference while retaining shared representations. We further construct a curated multi-task mouse NAFLD histology dataset with expert annotations for all NAS components. Experimental results demonstrate that the proposed method improves multi-task stability and generalization with substantially reduced computational cost compared to training separate single-task models. The code and the curated dataset have been prepared and will be made publicly available upon acceptance to support reproducibility.

• The paper introduces a novel parameter-efficient ViT framework with task-specific Adapter modules that mitigate negative transfer in multi-task NAFLD scoring.
• It employs an orthogonality-based regularization and uncertainty-weighted multi-task loss to balance learning for steatosis, ballooning, and inflammation.
• Comprehensive experiments on a curated histology dataset demonstrate competitive accuracy and scalability compared to full fine-tuning approaches.

Parameter-Efficient Subspace Decoupling ViT for Multi-Task Histological Scoring of NAFLD

Motivation and Challenges in Multi-Task Histological Scoring

Automated histological scoring is critical for objective diagnosis and monitoring of Nonalcoholic Fatty Liver Disease (NAFLD), particularly through quantification of the Non-Alcoholic Steatohepatitis Activity Score (NAS) components: steatosis, hepatocellular ballooning, and lobular inflammation. Manual annotation of these features is costly, suffers from inter-observer variability, and impedes scalable computer-aided diagnosis. Multi-task learning (MTL) provides a framework for joint modeling of correlated NAS indicators, but naive shared representations induce negative transfer due to pairwise correlations and imbalanced label distributions. Specifically, easier NAS components dominate joint optimization, degrading performance on more challenging sub-tasks. Existing Vision Transformer (ViT) methods largely focus on single-task settings or full fine-tuning, lacking explicit mechanisms to mitigate task interference in MTL.

Figure 1: Teaser illustrating the motivation and design of the proposed parameter-efficient multi-task framework.

Parameter-Efficient Multi-Task Swin-ViT Architecture

The proposed framework leverages a shared, frozen Swin-T backbone with lightweight task-specific Adapter/LoRA branches. These branches are inserted in higher transformer layers, each predicting a different NAS component from H&E patches. Parameter-efficient fine-tuning is achieved by maintaining independent bottleneck Adapters for each task, dramatically reducing the number of trainable parameters compared to full fine-tuning. The design specifically encourages tasks to occupy distinct high-level semantic subspaces, addressing the problem of negative transfer while retaining shared low-level representations.

A novel orthogonality-based constraint is introduced to decouple task-specific Adapter subspaces. The orthogonal decoupling loss regularizes the mutual dot-products of Adapter projection matrices, enforcing geometric independence and minimizing cross-task interference. This subspace separation is complemented by uncertainty-weighted multi-task loss optimization, which balances gradient magnitudes according to task-specific predictive noise.

Figure 2: Overview of our parameter-efficient multi-task Swin-T framework for NAFLD histological scoring. A frozen Swin-T backbone with task-specific Adapter/LoRA branches predicts Ballooning, Steatosis, and Inflammation from augmented H&E patches, and is trained with uncertainty-weighted multi-task loss and orthogonal subspace regularization.

Dataset Construction and Statistical Analysis

A curated mouse NAFLD histology dataset was developed with expert annotation of all NAS components. Only morphologically consistent H&E patches with definitive scoring information were retained. Standard augmentation procedures expanded the dataset eight-fold, resulting in 3,192 qualified patches. The dataset exhibits pronounced class imbalance and high tissue coverage, reflecting realistic learning constraints.

Figure 3: Summary of NAS class distributions and patch-level quality statistics in our dataset.

Numerical Evaluation and Ablation Studies

Extensive experiments show that the parameter-efficient multi-task approach with subspace decoupling achieves competitive or improved accuracy across NAS components compared to both full fine-tuning baselines and traditional InceptionV3-based CNNs trained separately per task. The Swin-Tiny backbone, with Adapter-only configuration, maximizes accuracy on steatosis and inflammation; ballooning is nearly saturated across backbones.

Ablation on the orthogonal constraint weight $\lambda$ demonstrates that a moderate penalty ($\lambda=0.1$) significantly improves multi-task performance across all components, especially steatosis and inflammation. Over-regularization ($\lambda=1.0$) induces performance collapse, confirming the necessity of balancing shared and task-specific representations. Comparison between Adapter-only and LoRA-only variants indicates that both strategies are effective, though Adapter modules slightly outperform LoRA on aggregate accuracy.

Qualitative analysis of Adapter activations shows spatially distinct responses aligned with pathology-consistent regions for each NAS component, corroborating the efficacy of subspace decoupling in practice.

Implications and Future Directions

The proposed framework presents a scalable solution for high-fidelity, multi-task histological scoring with minimal computational cost and explicit mitigation of negative transfer. Practical implications include accelerated annotation workflows and more robust automated quantification of NAFLD/NASH indicators. The orthogonality-based regularizer and parameter-efficient fine-tuning scheme generalize beyond NAFLD, providing a methodological blueprint for multi-task learning in other correlated medical imaging tasks.

Theoretical implications stem from the geometric separation of task-specific subspaces, highlighting that negative transfer can be effectively suppressed without sacrificing low-level representational sharing. This approach may catalyze further investigation into adaptive subspace constraints and dynamic task-balancing in MTL.

Future developments could extend the framework to whole-slide aggregation, patient-level diagnosis, and incorporation of additional pathological or clinical endpoints. Validation on human, multi-center datasets and integration with slide-level inference modules will be essential for clinical translation.

Conclusion

This work introduces a parameter-efficient Subspace-Decoupling ViT framework for multi-task histological scoring, integrating lightweight task-specific Adapters, explicit orthogonality-based regularization, and uncertainty-weighted optimization. Experimental results substantiate improved multi-task stability, generalization, and computational efficiency. The curated NAFLD histology dataset and release plan facilitate reproducible research. Limitations include exclusive focus on patch-level mouse data and absence of slide-level aggregation. Future work should pursue comprehensive clinical validation and broader application to related multi-task medical image analysis settings.