Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits

Abstract: The recent advancement of LLMs offers new opportunities to generate user preference data to warm-start bandits. Recent studies on contextual bandits with LLM initialization (CBLI) have shown that these synthetic priors can significantly lower early regret. However, these findings assume that LLM-generated choices are reasonably aligned with actual user preferences. In this paper, we systematically examine how LLM-generated preferences perform when random and label-flipping noise is injected into the synthetic training data. For aligned domains, we find that warm-starting remains effective up to 30% corruption, loses its advantage around 40%, and degrades performance beyond 50%. When there is systematic misalignment, even without added noise, LLM-generated priors can lead to higher regret than a cold-start bandit. To explain these behaviors, we develop a theoretical analysis that decomposes the effect of random label noise and systematic misalignment on the prior error driving the bandit's regret, and derive a sufficient condition under which LLM-based warm starts are provably better than a cold-start bandit. We validate these results across multiple conjoint datasets and LLMs, showing that estimated alignment reliably tracks when warm-starting improves or degrades recommendation quality.

• The paper's main contribution is establishing a prior error metric that predicts when synthetic priors accelerate learning versus when they induce negative transfer in contextual bandit systems.
• It introduces the Noisy-CBLI framework to systematically analyze the effects of both random replacement and preference flipping noise on warm-started recommender algorithms.
• Empirical results demonstrate that while aligned LLM priors can substantially reduce cumulative regret, misalignment or high corruption levels can make warm-start approaches less effective than traditional cold-start methods.

Theoretical and Empirical Evaluation of LLM-Initialized Bandits under Corruption and Misalignment

Introduction

This work presents a rigorous analysis of using LLMs to generate synthetic preference data for warm-starting contextual multi-armed bandit (CB) algorithms in recommender systems. By leveraging the intrinsic world knowledge of LLMs, the goal is to overcome the cold-start problem, wherein traditional CBs perform essentially randomly until receiving enough real user feedback. While previous results have indicated that LLM-generated priors can significantly reduce early cumulative regret, these findings have relied on a latent assumption that LLM preferences are well-aligned with real user choices.

The authors introduce a comprehensive evaluation framework, termed Noisy-CBLI, that systematically interrogates the alignment assumption by injecting controlled random and systematic (label-flipping) noise into the LLM-simulated dataset. The methodology and theoretical developments delineate the regimes under which synthetic priors genuinely provide benefit versus when misalignment or corruption leads to negative transfer, even surpassing cold-start regret.

Figure 1: Overview of the Noisy-CBLI evaluation pipeline, integrating LLM-generated synthetic data, optional noise injection, and subsequent LinUCB warm-start and fine-tuning on real user data.

Methodology and Experimental Protocol

Dataset Construction and Task Formulation

Experiments span three real-world conjoint analysis datasets—COVID-19 vaccine acceptance, immigration attitudes, and leisure travel preferences—each encoded as contextual multi-armed bandit decision sequences. LLMs are prompted to simulate individual user responses to binary profile comparisons using carefully structured prompts. The corresponding LinUCB algorithm is then initialized with synthetic context-reward pairs and fine-tuned on real data. Cold-start performance is established by running LinUCB without pre-training.

Corruption Models

Two principal corruption models are examined:

• Random Replacement: A proportion $p$ of LLM-generated labels is replaced with random actions, simulating uninformative or arbitrary mistakes.
• Preference Flipping: A proportion $p$ of labels is inverted (for $K=2$, from A to B and vice versa), inducing systematic bias that can directly conflict with the LLM prior.

Warm-started LinUCB performance is assessed as a function of $p$ using cumulative regret relative to an oracle arm selection.

Theoretical Analysis

The prior-centered analysis reveals that all pretraining effects can be encapsulated by a single prior error term $\mathcal{B}_0 = \|\theta_0 - \theta^\star\|_{A_0}$, where $\theta_0$ is the ridge estimate from the synthetic prior and $\theta^\star$ is the (unknown) true parameter governing user preferences. This term quantifies the combined effect of random noise, label flipping, and systematic misalignment.

The authors provide explicit bias-variance decompositions showing:

• For aligned priors (LLM-simulated responses are close to user distributions), $\mathcal{B}_0$ remains small, and moderate corruption ($p < 0.3$) does not negate warm-start advantages.
• For misaligned priors (significant divergence between LLM and human preferences), the systematic bias dominates, and warm-start can be immediately deleterious, with cold-start outperforming at $p = 0$.
• Preference flipping induces a quadratic growth in the prior error, leading to a critical corruption threshold $p$0 at which warm-start ceases to be beneficial.

This analysis holds across linear contextual and sleeping bandit formulations, with the empirical estimation of $p$1 via ridge fits to synthetic and real datasets accurately predicting the observed warm-start–to–cold-start phase transition.

Empirical Results

Effect of Preference-Flipping Noise

On the COVID-19 Vaccine dataset, incremental increases in flipping noise gradually erode the regret reduction conferred by the LLM prior. Warm-start is robust up to approximately $p$2, with regret advantage vanishing near $p$3. For $p$4, the prior becomes antithetical to human preferences and induces negative transfer—final regret surpasses the cold-start baseline, especially as the scale of synthetic pretraining increases.

Figure 2: Mean cumulative regret under preference-flipping corruption for the COVID-19 vaccine task; advantage persists until $p$540% flipping, then reverses direction.

Effect of Random-Response Noise

With random replacement, the system demonstrates increased resilience; warm-start performance monotonically degrades but never significantly exceeds cold-start regret. This demonstrates that uninformative errors in synthetic priors are less hazardous than systematic errors that adversarially bias the prior.

Figure 3: Cumulative regret curves under random-response corruption for the COVID-19 vaccine dataset; warm-start remains competitive even at high random noise rates.

Cross-Domain and Cross-Model Effects

Analysis across the immigration and travel datasets shows consistent patterns: when LLM alignment is high, warm-start maintains an advantage up to moderate corruption. On misaligned tasks, warm-start fails outright, reinforcing the theoretical claim that alignment is the dominant failure mode—random noise alone is not sufficient to induce negative transfer. Furthermore, empirical results across different LLMs and version snapshots reveal that model drift and prompt sensitivity are material concerns; even small changes in LLM design or inference parameters alter the efficacy of the prior.

Prior Error as a Diagnostic

The empirical prior error $p$6 serves as a practical diagnostic: smaller values correspond to reliable improvement over cold-start; elevated values predict marginal or negative transfer. This supports the theoretical sufficiency condition derived for warm-start utility and provides actionable guidance for practitioners considering deployment of LLM-generated priors in the absence of ground-truth user data.

Implications and Future Directions

The findings delineate the practical utility and inherent risks of synthetic preference warm-starting in modern recommender and bandit systems. In domains where the LLM is known to be aligned with user preference distributions—and provided synthetic data is not heavily corrupted—LLM initialization yields substantial efficiency gains in early-stage learning. However, applications sensitive to demographic, ideological, or content-specific biases must proceed with extreme caution. Without task-specific checks on alignment, synthetic priors can amplify both systematic bias and cumulative regret, with model drift further compounding risk over time.

A natural extension is the development of lightweight, domain-agnostic alignment diagnostics or estimators of potential prior harm prior to deployment. Additionally, moving beyond linear bandit frameworks towards nonlinear or deep CBs is essential for next-generation applications, where context-dependent noise and expressivity exceed current analytic tractability.

Conclusion

This study establishes that LLM-initialized bandits are not universally beneficial; their value is contingent on the fidelity of the synthetic prior to the target user population and the absence of significant, structured corruption. Theoretical and empirical results converge on a prior error threshold that reliably predicts the transition from useful to harmful pretraining. These insights clarify the conditions for safe and effective application of synthetic preferences for cold-start amelioration and highlight open challenges for future research—particularly in diagnostics, robustness auditing, and ethical deployment in high-stakes domains.