Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation

Abstract: Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of segmentation foundation models for medical imaging. However, most federated LoRA methods adopt a uniform aggregation rule, which breaks under the encoder-decoder asymmetry in medical segmentation: the encoder is dominated by appearance shifts, while the decoder is dominated by supervision variations. This mismatch entangles shared anatomy with site-specific biases and harms generalization. To address this, we propose Inverse Asymmetric Tuning (IAT). IAT aligns adaptation with heterogeneity sources by personalizing module-specific components in the encoder to absorb appearance shifts and in the decoder to accommodate site-dependent supervision, while retaining a shared pathway for transferable consensus. However, structural separation alone is insufficient under LoRA's bilinear parameterization, where multiplicative coupling can still cause site-specific updates to leak into the shared direction. We therefore introduce a Subspace Orthogonality Regularizer that penalizes shared-local collinearity in the effective update space, mitigating leakage without extra communication. Experiments show consistent improvements over strong federated LoRA and parameter-efficient FL baselines.

• The paper introduces Inverse Asymmetric Tuning (IAT) to allocate personalized and shared LoRA factors between encoder and decoder, addressing cross-site heterogeneity.
• It applies a Subspace Orthogonality Regularizer to decouple shared and local updates, ensuring robust generalization across divergent medical imaging datasets.
• Experiments on histology and fundus benchmarks show superior Dice scores with minimal communication and parameter costs, enhancing federated learning efficiency.

Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation

Introduction and Motivation

The paper addresses the challenge of adapting foundation segmentation models (notably SAM) to diverse medical imaging datasets in federated settings, where data aggregation across institutions is restricted due to privacy constraints. The dominant approach for parameter-efficient adaptation in federated learning (FL) has been Low-Rank Adaptation (LoRA), which reduces communication costs by focusing updates on low-rank matrices. However, the existing FL LoRA strategies typically assume uniform aggregation of LoRA factors, disregarding architectural asymmetries in encoder-decoder models. In medical segmentation, the encoder is primarily impacted by covariate shifts (acquisition/appearance changes), whereas the decoder is driven by concept shifts (annotation/supervision differences), leading to entangled shared and site-specific representations and suboptimal generalization.

Methodological Framework

The authors introduce a structure-aware federated fine-tuning framework consisting of two main components:

• Inverse Asymmetric Tuning (IAT): This module differentiates parameter sharing strategies between encoder and decoder, following theoretical derivations of shift-dependent preference. For encoder layers, dominated by covariate shift, the input-side LoRA factor (A) is personalized per client, while the output-side factor (B) is shared. In contrast, the decoder adapts the output-side factor locally and shares the input-side factor. This explicit inversion aligns parameter optimization with the dominant heterogeneity sources for each submodule.
• Subspace Orthogonality Regularizer (SOR): Structural decoupling alone does not prevent site-specific optimization biases from leaking into the shared global model due to LoRA's intrinsic bilinear coupling. The SOR penalizes collinearity between shared and local update directions within the effective parameter space, maintaining functional independence and preventing optimization interference through a soft geometric constraint.

Formally, the method achieves an $O(T^{-1/2})$ convergence rate under standard smoothness and bounded variance assumptions, matching FedAvg in non-convex settings up to lower-order drift terms.

Experimental Validation

The framework is evaluated on two federated medical image segmentation benchmarks:

• Histology Nuclei Segmentation: Seven clients, spanning PanNuke tissue types and independent datasets (MoNuSeg, MoNuSAC2020, TNBC), exhibit severe heterogeneity in tissue origin, staining, and scanner acquisition.
• Fundus Photography Segmentation: Four decentralized clients (REFUGE, ORIGA-light, Drishti-GS1, G1020) manifest domain shifts due to different acquisition protocols and patient demographics.

The foundation backbone is SAM ViT-B; LoRA decomposition is injected into both encoder and decoder, validating the necessity of adapting both modules for pixel-wise segmentation. The main evaluation metric is Dice Similarity Coefficient (DSC), with baseline comparisons to state-of-the-art parameter-efficient federated methods: FedIT, FLoRA, FedSA, FFA-LoRA, FedDPA, LoRA-FAIR, FlexLoRA, and FRLoRA—with all LLM-oriented methods adapted for vision encoder-decoder architectures.

Results

• Accuracy: On Histology and Fundus benchmarks, the proposed method achieves highest average DSC (81.40% and 84.52% respectively at LoRA rank=8), consistently outperforming the second-best competitor by 1.31–1.48%, with notable robustness on the most divergent domain (e.g., Drishti-GS1: +27% vs. FedIT).
• Efficiency: The method achieves near-minimal communication and parameter size (0.39M trainable parameters, 0.55M per-round communication), substantially reducing storage and bandwidth requirements while preserving SOTA accuracy—contrasting with FLoRA and FedDPA, which incur multi-fold parameter cost.
• Ablations: Incorporating IAT alone mitigates performance degradation due to domain shift; adding SOR attains maximal DSC, validating the theoretical claim that structural and functional disentanglement are necessary for robust federated segmentation.

Theoretical and Practical Implications

The paper provides a formal framework for shift-dependent LoRA factor allocation, establishing that optimal parameter sharing in federated segmentation is structurally inverted between encoder and decoder. This theoretically grounded approach addresses the limitations of uniform splitting strategies, which fail under complex cross-site heterogeneity. The Subspace Orthogonality Regularizer ensures that knowledge transfer is maximally efficient and that site-specific factors do not contaminate globally shared anatomical representations.

Practically, this method demonstrates how federated learning with foundation models can achieve robust, generalizable medical segmentation without costly full-parameter fine-tuning or excessive communication. The framework is broadly applicable to other encoder-decoder tasks exhibiting shift asymmetry, and could be extended to diverse multi-institutional AI deployments with strict privacy regimes.

Speculation on Future Directions

Potential advancements include adaptive rank allocation based on heterogeneity magnitude, integration with privacy-preserving cryptographic protocols, and extension to multimodal medical data. The structural insights gained here may inform future protocols for parameter-efficient federated training across complex architectures (e.g., hierarchical multimodal encoders/decoders), and motivate cross-task generalization studies in foundation models beyond segmentation.

Conclusion

The paper establishes a principled dual-decoupling protocol for federated medical segmentation, leveraging theoretical analysis of heterogeneity shifts to inform LoRA allocation strategies. By coupling Inverse Asymmetric Tuning with subspace orthogonality regularization, it achieves superior segmentation accuracy, efficiency, and robustness under severe cross-site domain shift. The implications extend to both practical federated deployment in medical AI and theoretical understanding of modular adaptation in non-i.i.d. decentralized environments (2606.08687).