Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback

Abstract: Recent works have advanced feedback-based learning systems, whereby a foundation model is able to intake incoming feedback (e.g., a user) to self-improve, creating a self-loop system of training. However, existing works are limited in needing to consider an offline setup to allow for such feedback-based methods, and are further limited in the need of requiring privileged ground-truth contexts for training. Moreover, there is limited consideration of federated learning (FL), which is particularly well-suited for incorporating external feedback across large networks of end users, for example, but requires methods to be efficient for training on resource-constrained edge devices. Therefore, we introduce SPEAR (Self-Play Enhancement via Advantage-Weighted Refinement), an efficient online learning algorithm for federated LLM fine-tuning. SPEAR utilizes a feedback-guided self-play loop to construct naturally contrastive pairs per prompt which are utilized to be trained on (i) standard maximum likelihood on correct completions and (ii) confidence-weighted unlikelihood on tail tokens of incorrect completions. Without the need of expensive group generations and ground-truth contexts for training (i.e., only partial, non-answer feedback), in contrast with existing works, SPEAR can be trained both online and in a resource-efficient manner. We validate SPEAR across various benchmark datasets, demonstrating its superior performance in comparison to state-of-the-art baselines. The implementation code is publicly available at https://github.com/lee3296/SPEAR.

• The paper introduces SPEAR, a method that combines self-play interactions and contrastive unlikelihood loss in federated environments to significantly enhance LLM fine-tuning performance.
• The methodology leverages win/loss traces with confidence weighting to enable feedback-driven adaptation while mitigating the challenges of noisy, partial real-time interactions.
• Empirical results demonstrate that SPEAR outperforms RL-based and self-distillation baselines on complex reasoning tasks while reducing computational overhead.

Self-Play Enhancement via Advantage-Weighted Refinement (SPEAR) for Online Federated LLM Fine-Tuning

Introduction and Motivation

Large-scale LLM fine-tuning in federated environments is challenged by the scarcity of high-quality, ground-truth feedback, heterogeneity of client resources, and the inherent noisiness of real-time user interactions. Existing online, feedback-guided adaptation methods for LLMs typically require privileged information (such as true answers) or rely on computationally costly preference optimization schemas ill-suited to federated, resource-constrained environments. The paper "Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback" (2605.07977) introduces the SPEAR algorithm, an online, efficient FL protocol that facilitates fine-tuning LLMs using noisy, partial real-time feedback without privileged context, while providing robust accuracy and computational scalability.

SPEAR Methodology

SPEAR is structured in two sequential phases: feedback-driven interaction and contrastive training, executed within an FL protocol on edge clients.

During each federated round, selected clients locally execute the self-play protocol:

1. Interaction and Data Collection:
   • The model produces an initial output for a user-supplied prompt.
   • A feedback mechanism (e.g., user correction, partial hints) is elicited—crucially, with no ground-truth answers required.
   • If the initial response is correct, it's logged as a win; otherwise, the feedback is appended to form a revised context, and the model attempts up to $N$ revisions.
   • All attempts are categorized into win (corrected or initially correct) and lose (uncorrected after $N$ attempts), with confidence weights for incorrectness.
2. Loss Design and Optimization:
   • Training combines conventional maximum likelihood (MLE) on win traces and a confidence-gated, tail-focused unlikelihood loss on lose traces.
   • The unlikelihood component suppresses high-confidence tokens in incorrect completions, with careful hyperparameters for confidence threshold $\mu$ and the tail token span $\tau$.
   • Client updates are aggregated using federated averaging weighted by the number of win traces, prioritizing high-quality local learners.

Figure 1: The two phases of the SPEAR algorithm—interaction with feedback generation yielding win/loss traces, followed by federated MLE and unlikelihood contrastive training.

Theoretical Properties

SPEAR’s loss function guarantees an information-theoretic separation of likelihoods between correct and incorrect completions. Under full coverage of high-confidence errors in the lose set, minimizing the SPEAR loss strictly enforces a log-probability margin between win and (tail) lose traces. This separation is tunable via $\mu$ (confidence threshold), $\tau$ (targeted token tail span), and the loss weighting factors. The framework provides theoretical insight into the tradeoff between coverage and penalization strength, indicating stability and consistent separation guarantees regardless of the quality of feedback or model scale.

Empirical Evaluation

SPEAR was benchmarked against state-of-the-art feedback-based fine-tuning and RL-based baselines (GRPO, OPSD, RLTF-SD, and Feedback SFT) on Llama3.2-3B and Qwen2.5-1.5B models using ARC-Challenge, HellaSwag, MathMCQA, and StrategyQA datasets. Experimental settings prioritized FL-specific constraints: client numbers, non-i.i.d. data splits, LoRA adaptation, and batch-budgeted training.

Key results:

• SPEAR surpassed all baselines in accuracy on every dataset and model, with especially strong gains on complex multi-step reasoning tasks (e.g., MathMCQA: SPEAR 56.7 vs. GRPO 40.2 on Llama3.2-3B).
• Feedback SFT lagged SPEAR, highlighting the necessity of the contrastive unlikelihood component.
• Self-distillation (OPSD) collapsed in these settings, yielding near-zero accuracy where direct feedback was noisy or incomplete.
• SPEAR demonstrated consistently lower wall-clock training time, with minimal computational overhead compared to SFT despite additional loss computations and corrections. Group-based RL baselines suffered from pronounced memory and runtime bottlenecks in federated settings.

Performance stability of SPEAR under varying hyperparameters ($\mu$, $\tau$) and client heterogeneity was empirically validated, confirming robustness to a broad operational envelope.

ARC-Challenge Performance Visualization

Figure 2: SPEAR’s superior performance progression on the ARC-Challenge benchmark compared to alternative methods.

Figure 3: Iterative accuracy curves on ARC-Challenge showing the rapid convergence and stability of SPEAR.

Additional Empirical Analysis

Ablations confirmed:

• Lower unlikelihood confidence thresholds ($\mu$) benefit performance by expanding the active penalization set, but with diminishing returns and minimal over-tuning risk.
• Targeting only the tail tokens for unlikelihood (smaller $\tau$) was effective for tasks demanding complex output structures.
• SPEAR maintained advantage in classical FL and even in centralized settings, evidencing flexible applicability beyond its original FL motivation.

Practical and Theoretical Implications

Practically, SPEAR enables real-world systems to improve LLMs from user-mediated feedback at the edge without exposing sensitive data, achieving both privacy and efficiency. The computational design is compatible with small-scale LoRA adapters and modest memory budgets typical of federated, resource-constrained devices. Theoretically, the margin bounds established by SPEAR provide a principled foundation for future feedback-contrastive objectives in LLM optimization—allowing explicit control over model calibration with respect to both positive and negative examples.

SPEAR’s training architecture provides a tractable alternative to RL-based preference optimization in FL, where group-based rollouts are infeasible, and reward specification is ill-defined. It also mitigates failure modes of self-distillation by robustly filtering and contesting spurious feedback trajectories.

Outlook and Future Directions

Potential future developments include adaptation to additional downstream tasks (dialog, code, retrieval augmentation), further reducing reliance on informative feedback by incorporating self-consistency heuristics, and integrating ranking-based or multi-feedback aggregation mechanisms to extend SPEAR’s resilience against adversarial or irrelevant feedback. Extension to hierarchical, multi-level FL aggregation, and tighter integration with personalized federated approaches, are also promising.

Conclusion

SPEAR introduces an effective, theoretically justified protocol for online federated LLM fine-tuning with partial, noisy, unprivileged feedback. The method achieves both strong empirical performance and computational efficiency, significantly outperforming prevailing baselines on multiple reasoning and QA tasks. Its flexibility and principled loss structure enable both immediate practical deployment in privacy-preserving LLM adaptation and foundational advances in federated representation learning and feedback alignment.