CharacterFlywheel: Social Chat LLM Optimization

• CharacterFlywheel is a production methodology that iteratively improves large language models for social chat by integrating reward modeling, supervised fine-tuning, and reinforcement learning.
• It employs a three-phase data curation pipeline with multi-channel annotation to capture diverse user signals while ensuring safety and quality in conversational responses.
• Controlled online A/B tests and comprehensive offline metrics demonstrate significant gains in engagement depth and steerability across iterative deployment cycles.

Searching arXiv for CharacterFlywheel and closely related post-training/reward-modeling work to support the article. CharacterFlywheel is an iterative production process for improving LLMs used in social chat applications across Instagram, WhatsApp, Messenger, and the Web. It is defined as a flywheel that repeatedly combines production traffic, annotation, reward modeling, supervised fine-tuning, reinforcement learning, offline evaluation, and controlled online A/B testing to optimize two central objectives: engagement and steerability. The system starts from Llama 3.1, proceeds through repeated model generations, and treats social-chat optimization as navigation of an unknown, noisy, and non-differentiable “engagingness landscape,” with surrogate reward models used to estimate local improvement directions before online validation (Nie et al., 2 Mar 2026).

1. Definition, scope, and problem setting

CharacterFlywheel is designed for production social-chat systems in which the target objectives are not reducible to exact task correctness. The paper identifies the core difficulty as the mismatch between product goals—such as whether people continue chatting, return, react positively, or perceive a character as on-spec—and the absence of a clean differentiable reward function. In this setting, the optimization target is subjective, noisy, and only partially observable through aggregate deployment metrics rather than direct token-level supervision (Nie et al., 2 Mar 2026).

The framework therefore treats post-training not as a one-shot alignment stage but as a closed-loop production optimization system. Starting from Llama 3.1, Meta reports 15 generations of CharacterFlywheel models, with continuous deployments from July 2024 to April 2025. Across the 8 post-launch deployed versions, 7 of 8 achieved positive engagement lift over baseline in controlled 7-day A/B tests, with the strongest results reaching $+8.8\%$ engagement breadth and $+19.4\%$ engagement depth. Steerability improved concurrently: IFEval rose from $59.2\%$ to $84.8\%$, and instruction violations fell from $26.6\%$ to $5.8\%$ (Nie et al., 2 Mar 2026).

A common misunderstanding is to treat CharacterFlywheel as merely a reinforcement-learning recipe. The paper instead presents it as a production methodology in which reward modeling, data curation, SFT, RL, offline diagnostics, and online experimentation are interdependent. A plausible implication is that the term “flywheel” refers less to any single optimization algorithm than to the repeated redeployment-and-retraining loop through which the system remains aligned with non-stationary product traffic.

2. Data curation, annotation, and iterative supervision

The system draws from both internal and production data. Internal sources include internal interactive chats, internal safety datasets, internal capability and tool-calling datasets, and internal failure-mode datasets. Production sources include curated deployed conversations and user behavioral signals such as continue chatting, thumbs up, thumbs down, written feedback, and emoji reactions like “love.” Legacy Llama 3.1 SFT and preference data are also retained in the post-training mixture to preserve broad utility capability (Nie et al., 2 Mar 2026).

Because production traffic volume is too large to use directly, CharacterFlywheel imposes a three-phase curation pipeline. Phase I applies strict privacy and safety filtering. Phase II performs diversity sampling using MultiRay and DRAMA-1B embeddings to cluster prompts and prune redundancy, retaining a proportion $p$ of filtered data after clustering-based sampling. Phase III performs constraint-based adjustment to match traffic or target distributions along dimensions listed in the paper: first-turn conversation ratio of at least $10\%$, per-character cap of at most $3\%$, locale and language, job-to-be-done, conversation depth, prompt quality, and prompt difficulty (Nie et al., 2 Mar 2026).

Annotation is correspondingly multi-channel. In static annotation, annotators inspect completed conversations and judge final responses. In interactive-chat annotation, annotators converse with the character turn by turn and evaluate each turn. They may answer structured quality questions, rank response pairs by engagingness, and rewrite poor responses. Two questions were consistently retained across development because they capture common failures: whether a response is a false refusal and whether it is a templated response. A separate steerability workflow mildly challenges the model and marks whether responses violate character instructions; if both candidates fail, annotators rewrite a compliant response (Nie et al., 2 Mar 2026).

This annotation structure is technically consequential because it does not reduce supervision to scalar preference labels alone. It also encodes failure modes, rewrites, and character-adherence judgments. The paper further notes that pairwise data are filtered for superficial artifacts such as length difference or emoji-count difference so that the preference model does not learn shallow heuristics. This suggests that CharacterFlywheel treats annotation not only as supervision, but as a mechanism for controlling shortcut learning.

3. Reward modeling and the “engagingness landscape”

CharacterFlywheel uses reward models because the true online objectives are non-differentiable and only observable through deployment experiments. The paper characterizes reward models as tools to “estimate and interpolate the landscape” around the current model so that optimization can take a controlled local step rather than attempt direct end-to-end optimization on sparse business metrics (Nie et al., 2 Mar 2026).

The primary reward signal comes from Bradley–Terry preference models implemented in both pointwise and pairwise forms. The pointwise loss is given as

$$\mathcal{L}_{\text{pointwise}} = -\log\bigl(\sigma(r_\theta(x, y_c) - r_\theta(x, y_r))\bigr),$$

while the pairwise loss is

$+19.4\%$0

with context

$+19.4\%$1

Pointwise RM scores guide RL, while both pointwise and pairwise win-rates are used in evaluation (Nie et al., 2 Mar 2026).

Auxiliary user-signal models are trained as binary predictors for behaviors such as $+19.4\%$2, $+19.4\%$3, $+19.4\%$4, $+19.4\%$5, and $+19.4\%$6. The paper reports, however, that only $+19.4\%$7 and $+19.4\%$8 were consistently useful, and even then mainly for rejection-sampling data selection rather than direct RL optimization. The paper’s stated reason is that direct optimization on user signals is risky because of delayed feedback bias, ending bias or sycophancy, varying positive-negative ratios across jobs-to-be-done, and confounding from prior-turn sentiment (Nie et al., 2 Mar 2026).

This reward-modeling framework is explicitly cautious. Reward models are treated as local approximators rather than authoritative global objectives. That point becomes central in later sections: CharacterFlywheel’s methodology depends on near-policy data and online verification precisely because reward models can drift, overfit, or become miscalibrated outside the local region in which they were trained.

4. Training pipeline: pre-herding, herding, and near-policy optimization

The paper adopts Llama’s “herding” terminology. In CharacterFlywheel, pre-herding consists of reward-model training plus rejection-sampling dataset construction, and herding refers to updating the chat model via SFT, DPO, and RL. A fresh rejection-sampling dataset is rebuilt every generation using recent traffic and candidate models so that the data remain close to the current policy. The algorithm selects the best candidate response for prompt $+19.4\%$9 by sampling $59.2\%$0 candidates, scoring them with reward model $59.2\%$1, and keeping the best if it exceeds threshold $59.2\%$2: $59.2\%$3 Only 70B models are deployed in production for inference efficiency, but both 70B and 405B models are included in the candidate pool for rejection-sampling generation (Nie et al., 2 Mar 2026).

The SFT stage begins from Llama 3.1 70B and mixes six data classes: internal interactive-chat RJS data, user-traffic RJS data, internal safety data, capability and tool-calling data, ad hoc failure-mode data, and Llama 3.1 post-training SFT data. After SFT, the checkpoint receives a small DPO patch using internal safety preference data, image generation data, and Llama 3.1 preference data. The paper explicitly describes DPO as a small patch for urgent safety and style fixes rather than the primary iterative improvement engine (Nie et al., 2 Mar 2026).

The main policy-improvement stage is online RL. The team reports trying Online DPO and a GRPO variant with importance-sampling corrections for distributed training, later switching to GRPO because it gave better A/B engagement. The paper provides a PPO/GRPO-style clipped objective with importance correction from $59.2\%$4, clipping around $59.2\%$5, advantage $59.2\%$6, and KL regularization against a reference policy $59.2\%$7. In direct testing, GRPO yielded $59.2\%$8 engagement breadth lift over Online DPO (Nie et al., 2 Mar 2026).

Prompt selection is also policy-aware. RL prioritizes prompts with low RM scores and high intra-prompt RM score variance, and the paper later argues that variance-based downsampling is superior because absolute RM score magnitude is confounded by factors such as conversation length. Near-policy prompt sampling is one of the strongest empirical findings in the paper: relative to stale off-policy prompts, using prompts from the latest model’s traffic yielded $59.2\%$9 engagement depth and $84.8\%$0 engagement breadth (Nie et al., 2 Mar 2026).

5. Evaluation methodology and empirical trajectory

CharacterFlywheel evaluates candidate checkpoints through an offline stack and online 7-day A/B testing. Offline evaluation includes community benchmarks, human comparison, reward-model win-rates, custom production metrics, and safety/privacy checks with redteaming. The benchmark suite includes MMLU, IFEval, ARC-Challenge, GPQA, HellaSwag, GSM8K, MATH, HumanEval, and MBPP. Human comparison is performed side by side against GPT-4o and the previous CharacterFlywheel version, with randomized conversation continuation so that history does not systematically favor the currently winning model (Nie et al., 2 Mar 2026).

The custom production metrics include Avg. Response Length, Contains List, Contains Emoji, Positive Sentiment, Instruction Violation, Cooperative Ratio, Non-Preachy Rate, Templated Responses, Wall-of-Text Failure, Preachy Tone, and False Refusal. These are used as diagnostic guardrails rather than as sole optimization targets. Online evaluation is the final arbiter: controlled 7-day A/B tests use random assignment, typically $84.8\%$1 of traffic to each arm, a one-week readout window, and cumulative exposure logging. Percentage lift is defined as

$84.8\%$2

Because lift is a ratio of means, confidence intervals are computed with Fieller’s theorem rather than symmetric normal approximations (Nie et al., 2 Mar 2026).

The principal online metrics are engagement breadth and engagement depth. Breadth is estimated from

$84.8\%$3

while depth is estimated from

$84.8\%$4

where $84.8\%$5. Breadth measures how broadly the model induces engagement across the eligible population, whereas depth measures how much engagement occurs among users who engage at all (Nie et al., 2 Mar 2026).

The post-launch deployment trajectory is summarized below.

Version	Breadth lift	Depth lift
V8	$84.8\%$6	$84.8\%$7
V9	$84.8\%$8	$84.8\%$9
V10	$26.6\%$0	$26.6\%$1
V11	$26.6\%$2	$26.6\%$3
V12	$26.6\%$4	$26.6\%$5
V13	$26.6\%$6	$26.6\%$7
V14	$26.6\%$8	$26.6\%$9
V15	$5.8\%$0	$5.8\%$1

The paper also reports pre-launch human-evaluation improvements against GPT-4o from $5.8\%$2 at V3 to $5.8\%$3 at V7, and instruction-violation reductions from $5.8\%$4 at V2 to $5.8\%$5 at V8. Response-quality metrics improved as well: wall-of-text failure dropped $5.8\%$6, preachy tone decreased $5.8\%$7, positive sentiment increased $5.8\%$8, cooperative ratio improved $5.8\%$9, and non-preachy rate reached $p$0 (Nie et al., 2 Mar 2026).

6. Overfitting, failure cases, and scientific significance

The central controversy addressed by the paper is whether reward-model optimization can be trusted in production social-chat systems. CharacterFlywheel’s answer is explicitly conditional: reward models are useful, but dangerous when over-optimized. The most important case study is V12. In that deployment, RM User win-rate rose to $p$1, RM Internal win-rate fell to $p$2, and online engagement degraded, with breadth only $p$3 and depth $p$4. This became the paper’s clearest empirical demonstration that a high reward-model win-rate can coexist with worse real-user outcomes (Nie et al., 2 Mar 2026).

The operational response was to add stronger anti-overfitting guardrails. These include monitoring both internal and user RM win-rates, treating divergence as a warning sign, maintaining RM win-rates below $p$5, and preferring values around $p$6 for sustainable optimization. Additional regularizers include KL constraints in RL, EMA reference policies, near-policy sampling, artifact monitoring, train-eval prompt separation, and pairwise-data filtering against shallow stylistic cues such as response length, formatting, emoji usage, and phrase indicators like “I feel like…” (Nie et al., 2 Mar 2026).

The paper also emphasizes production non-stationarity. User populations shift, use cases change, product features evolve, image generation changes traffic composition, and reward models degrade on future batches. Static preference future-batch accuracy is reported often around $p$7–$p$8, and interactive future-batch accuracy often around $p$9–$10\%$0. This is presented not as a flaw specific to one model, but as a structural reason why the flywheel must keep turning: data are re-curated every iteration, reward models are retrained on new batches, RJS datasets are rebuilt every generation, and evaluation prompt sets are periodically updated (Nie et al., 2 Mar 2026).

CharacterFlywheel’s broader significance lies in showing that subjective social-chat objectives can be optimized systematically at production scale without collapsing the problem into a single static reward. It contributes a hill-climbing framework in which real conversational traffic, human preference annotation, rejection sampling, SFT, DPO patching, RL, offline guardrails, and online experiments are all part of one iterative control loop. A plausible implication is that its main scientific contribution is methodological rather than architectural: it offers a reproducible blueprint for continuously improving engaging, steerable, and production-reliable social-chat LLMs under non-differentiable objectives and non-stationary data (Nie et al., 2 Mar 2026).