Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning

Abstract: As LLMs are increasingly deployed at the network edge to provide pervasive generative AI services, decentralized federated learning (DFL) provides a vital mechanism for privacy-preserving, domain-specific fine-tuning through peer-to-peer exchanges of parameter-efficient updates. However, the dynamic nature of practical decentralized edge networks, where devices may dynamically join or leave the collaborative training process, requires the system to continuously adapt to new data while selectively removing prior contributions. This correction process remains a significant bottleneck, as individual device updates become deeply entangled within the global fine-tuned parameters. To address this challenge, we propose a priority-aware learning-unlearning correction framework based on orthogonal LoRA that can enhance the knowledge evaluation through topology adjustment. Specifically, we first design an orthogonal LoRA mechanism that yields post-training contribution coordinates, enabling history-free projection addition and deletion in response to membership changes. We then analyze the correction bottleneck and develop a priority-aware policy that selects among topology refinement, local correction, proximal damping, and synchronization scheduling according to the dominant residual term. A resource allocation algorithm is further developed to allocate limited communication across layer groups, prioritizing the primary bottlenecks within per-round wireless constraints. Experiments demonstrate that the proposed framework achieves robust post-event correction for both device join and leave events and validate that different residual regimes necessitate distinct correction actions.

• The paper introduces a priority-aware mechanism using orthogonal LoRA projections to isolate device contributions, enabling history-free learning and unlearning in decentralized federated learning.
• It decomposes the correction gap into local, consensus, and heterogeneity residuals, employing empirical Fisher information and gradient energy as proxies for optimal resource allocation.
• Experimental results on Qwen-7B and ResNet-18 illustrate up to a 6.3% lower correction gap and improved retention, all achieved with zero gradient storage overhead.

Priority-Aware Correction in Dynamic Decentralized LoRA Fine-Tuning

Introduction

The paper "Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning" (2606.22878) addresses the inherent challenges of adapting LLMs at the network edge using LoRA in decentralized federated learning (DFL) environments with dynamic device participation. Device join/leave events in real-world DFL setups necessitate efficient post-event correction mechanisms capable of absorbing new device knowledge while precisely removing departing device contributions under communication constraints. Current decentralized LoRA solutions lack granular device-level contribution separation and efficient unlearning methodologies without retraining or storing historical information, making correction a significant bottleneck. This paper introduces an orthogonal LoRA projection mechanism alongside a priority-aware resource allocation framework to enable history-free subspace addition and deletion, achieving robust decentralized correction after membership changes.

Figure 1: Dynamic DFL system illustration, showing device join and leave events managed through orthogonal LoRA projections and priority-aware correction.

Orthogonal LoRA for History-Free Contribution Indexing

The authors formalize a dynamic decentralized LoRA architecture where each device initializes a frozen, randomly sampled orthogonal projection basis per layer. In this protocol:

• During device joining, the orthogonal basis ensures new adapters form non-interfering subspaces, facilitating seamless incorporation of knowledge.
• Upon device leaving, the orthogonal projection enables subspace deletion by locally projecting out the departing member's basis, with the component transfer bounded by the pairwise basis overlap.

This structure ensures device-level contribution isolation in the aggregated adapter and allows history-free removal and addition without retraining or device-specific gradient logs. Theoretical results demonstrate bounded leakage between device subspaces and establish that the post-event initialization gap is proportional to the squared parameter residual.

Figure 2: Comparison of update conflict between conventional and orthogonal LoRA, highlighting improved device-specific projection separation.

Correction Residual Decomposition and Resource Allocation

Leveraging decentralized gradient descent (DGD), the paper characterizes the correction gap after device events through a decomposition into local optimization, consensus, heterogeneity, and irreducible residuals:

• Local residual ($\mathcal{R}_{\mathrm{loc}}$): shrinks with increased local steps.
• Consensus residual ($\mathcal{R}_{\mathrm{con}}$): contracts under denser aggregation graphs and stronger mixing.
• Heterogeneity residual ($\mathcal{R}_{\mathrm{het}}$): forms a persistent floor determined by cross-device objective misalignment.
• Tradeoffs exist: increasing local steps lowers local residual but raises consensus disagreement.

For practical resource allocation, the authors employ empirical Fisher information and post-projection gradient energy as computable proxies for layer group curvature and initialization gap. Priority scores ($\mathcal{S}_g$) rank layer groups to align computation, proximal damping, and communication density with dominant residuals, orchestrating correction with minimal overall gap under bandwidth constraints. The resulting correction policy automatically concentrates effort on high-risk layer blocks.

Numerical Results

The experimental validation utilizes Qwen-7B and ResNet-18 (CIFAR-100), simulating multiple join/leave events under decentralized LoRA protocols. Late blocks display $227\%$ higher priority scores, prompting doubled local steps and additional proximal damping. The priority-aware policy outperforms uniform and scheduling-agnostic strategies, yielding:

• $6.3\%$ lower final event gap compared to naive removal.
• $22\%$ lower gap relative to KD-based unlearning.
• Comparable correction quality to FedEraser-D2D ($0.69\%$ improvement), but with zero gradient storage overhead.
• $1\%$ lower retain loss than Fisher-weighted and influence-based correction arms.

Ablation studies verify that combining local-step, retain-prox, and priority synchronization components achieves the lowest gap-cost Pareto frontier. Routing the priority signal solely to topology is suboptimal due to elevated communication cost without local residual reduction. Figure results consistently demonstrate that priority-aware correction stabilizes retain-forget tradeoff and accelerates convergence after membership changes.

Figure 3: Event gap $E_{FU}$ over correction rounds showcasing accelerated convergence for the priority-aware method versus baselines and ablations.

Figure 4: Relative improvement RI versus uniform allocation illustrating superior retain loss management during correction.

Figure 5: Retain loss trajectory under leave event, evidencing reduced damage to retained knowledge in priority-aware policy.

Figure 6: RI versus round $\mathcal{R}_{\mathrm{con}}$0 for ablation arms, highlighting the advantages of priority-aware local steps and proximal anchoring.

Discussion and Implications

The orthogonal basis mechanism enables robust device-level knowledge separation in a decentralized adapter, addressing GDPR-compliant data removal and modular edge knowledge incorporation. The finite-round priority-aware correction framework formalizes communication-computation tradeoffs in dynamic membership DFL, providing a principled, scalable resource allocation strategy that adapts to per-layer sensitivity. The results indicate that targeted correction reduces both event gap and cost, and avoids catastrophic interference during rapid topology changes.

From a theoretical perspective, decomposing the correction gap into local, consensus, and heterogeneity components allows explicit identification and mitigation of dominant error sources. Practically, the policy is compatible with realistic wireless constraints, requiring only one-pass observables and incurring minimal overhead.

Future directions may include hierarchical extension to multi-hop edge-cloud federations, integration with adapter sparsification for further efficiency, and exploration of privacy-resilient certified unlearning protocols using the proposed subspace operations across foundation models. The layer-group priority metric could be generalized to multimodal or continual learning settings where modality or temporal drifts dominate correction needs.

Conclusion

This work introduces a priority-aware learning-unlearning correction methodology for dynamic decentralized LoRA fine-tuning, utilizing orthogonal projection matrices to enable robust device join and leave operations without historical logs. The framework achieves fast, resource-efficient correction by aligning allocation with layer group residual sensitivities, outperforming conventional and scheduling-agnostic protocols in both correction gap and retain loss. This establishes a scalable foundation for dynamic edge intelligence, privacy-conscious DFL, and future federated foundation model adaptation under real-world constraints.