Online Learning from Strategic Human Feedback in LLM Fine-Tuning (2412.16834v2)

Abstract: Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning LLMs to align them with human preferences. However, human labelers are selfish and have diverse preferences. They may strategically misreport their online feedback to influence the system's aggregation towards their own preferences. Current practice simply averages labelers' feedback per time and fails to identify the most accurate human labeler, leading to linear regret $\mathcal{O}(T)$ for $T$ time slots. To our best knowledge, we are the first to study online learning mechanisms against strategic human labelers in the LLM fine-tuning process. We formulate a new dynamic Bayesian game and dynamically adjust human labelers' weights in the preference aggregation, ensuring their truthful feedback and sublinear regret $\mathcal{O}(T^{1/2})$. Simulation results demonstrate our mechanism's great advantages over the existing benchmark schemes.

The paper "Online Learning from Strategic Human Feedback in LLM Fine-Tuning" tackles the challenge of aligning LLMs such as ChatGPT more effectively with human preferences through Reinforcement Learning from Human Feedback (RLHF). The authors identify a critical problem in the RLHF process: human labelers, driven by diverse and selfish preferences, tend to strategically misreport their feedback to skew the system's preference aggregation toward their desires. This misreporting leads to linear regret ($\mathcal{O}(T)$) over $T$ time slots in the current practice, where feedback is simply averaged out without distinguishing the accuracy of feedback provided by different labelers.

To address this, the authors introduce a novel online learning mechanism that dynamically adjusts the weights of human labelers' feedback according to their accuracy in a dynamic Bayesian game framework. The game is structured to ensure truthful feedback from labelers by correlating their long-term influence within the system to the accuracy of their feedback. The mechanism employs a non-monetary approach, avoiding complex billing issues associated with monetary incentive designs, and operates successfully in an online setting where human labelers have repeated interaction opportunities.

Key contributions and findings of the paper include:

1. Mechanism Design:
   • A weighted aggregation mechanism dynamically adjusts labelers' weights based on their reported preference accuracy.
   • Ensures truthful reporting and achieves sublinear regret of $\mathcal{O}(T^{1/2})$, significantly better than previous approaches, by optimizing feedback aggregation in each round.
   • Utilizes a step-size parameter $\alpha$, chosen as $\frac{2}{3}\sqrt{\frac{2\ln N}{T}}$, where $N$ is the number of labelers and $T$ is the number of time slots, for effective learning and regret minimization.
2. Theoretical Analysis:
   • The authors use analytical proofs to show that their method is both truthful and efficient, leading to low regret compared to traditional feedback aggregation methods.
   • They illustrate that dynamic weighting based on feedback accuracy is crucial for minimizing regret in LLM fine-tuning processes.
3. Simulation Results:
   • Extensive simulations validate the proposed mechanism's advantage over existing benchmarks, such as the average feedback aggregation and median aggregation schemes. These benchmarks yield non-vanishing regret, indicating inefficiency in truthfully and accurately aggregating feedback.

The methodological advancements proposed in this paper are demonstrated to provide considerable improvements in fine-tuning LLMs by effectively mitigating the impact of strategic misreporting from human labelers and aligning the models closer to truthful human preferences. This aligns with practical application needs where LLMs are expected to dynamically adapt to user feedback and preferences in real-time, establishing a novel approach to handling strategic behavior in human feedback loops.