What's In My Human Feedback? (WIMHF)

• WIMHF is a framework that distinguishes measurable preferences from expressed ones, formalizing human feedback in RLHF datasets.
• It employs sparse autoencoders to extract up to 32 interpretable latent features, achieving performance close to black-box reward models.
• The framework identifies systematic biases from human traits, interface quality, and aesthetic factors to enhance model safety and personalization.

What's In My Human Feedback? (WIMHF) refers to the empirical, algorithmic, and measurement frameworks for identifying, modeling, and interpreting the latent signals encoded in human-provided evaluative and preference data for machine learning systems. WIMHF research quantifies the axes along which human feedback varies—task-relevant and task-irrelevant—and develops analytical toolchains for extracting interpretable factors, modeling annotator biases, and mitigating feedback-driven pitfalls in reinforcement learning from human feedback (RLHF) and related domains.

1. Formalizing the Content of Human Feedback

WIMHF distinguishes between measurable preferences (the space of features along which options differ and can, in principle, be judged) and expressed preferences (the subset of features on which annotators’ feedback is actually conditioned) (Movva et al., 30 Oct 2025). In a canonical RLHF/preference dataset, each sample $(p, r_a, r_b, y)$ consists of a prompt, two candidate responses, and a binary label indicating preference. The formalism $$(p, r_a, r_b, y) \sim \Pr(p)\Pr(r_a, r_b|p)\Pr(y|r_a, r_b, p)$$ captures the stochastic dependence of feedback on both the data-generating process (which responses are shown) and annotator subjectivity.

Feedback modalities extend beyond explicit ratings to encompass scalar-valued judgments, pairwise/ternary comparisons, trajectory demonstrations, direct interventions, and implicit signals (facial expressions, timestamps, prosody, etc.) (Kaufmann et al., 2023, Cui et al., 2020). The precise semantics of these signals are context-dependent—ranging from direct reward proxies to latent, interpretable attributes.

2. Algorithmic Extraction of Interpretable Preferences

WIMHF operationalizes the extraction of latent preference dimensions via sparse feature learning on embedding differences. The method introduced in "What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data" employs a BatchTopK sparse autoencoder:

• Comparisons are represented as $e_\Delta = e_{r_a} - e_{r_b}$ in embedding space.
• A sparse linear encoder-decoder with a Matryoshka prefix ({8,32} dims) discovers up to $M = 32$ latent features with $K = 4$ non-zeros per example.
• The decoded features correspond to interpretable axes ("uses Markdown formatting", "emphasizes sustainability", etc.), and are mapped to linguistic descriptions via LLM annotation pipelines (Movva et al., 30 Oct 2025).

For each learned feature $z_j$, sparse logistic regression is used to model its predictive contribution to $y$: $$\Pr(y=1) = \sigma(\alpha + \beta_j z_j + \gamma \ell_\Delta)$$ with $\ell_\Delta$ controlling for length bias.

Empirical evaluations across seven major RLHF and LLM preference datasets reveal that 3–5 sparse features capture the majority of the explainable variance in human feedback, with AUCs achieving up to 84% of black-box reward model performance.

3. Sources of Systematic Bias and Implications for Modeling

Multiple lines of evidence have established human feedback as fundamentally non-stationary and systematically biased:

• Task-independent human factors: Human raters' backgrounds, "teacher" experience, and robot familiarity modulate feedback accuracy and precision. The CHARM framework models feedback as a nonlinear function of both task statistics ($\hat{r}$) and vectorized human traits $$(p, r_a, r_b, y) \sim \Pr(p)\Pr(r_a, r_b|p)\Pr(y|r_a, r_b, p)$$0, boosting prediction accuracy by +23 pp over reward-only models (Fang et al., 16 Jun 2025).
• Aesthetic attributes of agents: The appearance of a "cute" robot induces a statistically significant skew in positive/negative feedback (ratio $$(p, r_a, r_b, y) \sim \Pr(p)\Pr(r_a, r_b|p)\Pr(y|r_a, r_b, p)$$1 vs $$(p, r_a, r_b, y) \sim \Pr(p)\Pr(r_a, r_b|p)\Pr(y|r_a, r_b, p)$$2), necessitating algorithmic debiasing corrections as implemented in Stochastic TAMER (Eggers et al., 7 Feb 2025).
• Interface and prompt quality: Preferences in LLM dialogues reflect not just answer quality but prompt clarity, interface affordances, and user intent structure (Liu et al., 30 Jul 2025).
• Personality imprints: Feedback-driven finetuning encourages and discourages specific stylistic and personality traits (verbosity, structure, confidence vs. conciseness, uncertainty) (Findeis et al., 30 Sep 2025).

These findings highlight the necessity of modeling human-driven, context-dependent modulation in any serious analysis or deployment of feedback-based learning systems.

4. Modalities and Typologies of Feedback

WIMHF research adopts a multi-layered taxonomy of feedback modalities:

• Explicit preference signals: Pairwise comparisons, scalar ratings, K-way rankings.
• Implicit feedback: Facial action units, head gestures, voice prosody, typed interactions (Cui et al., 2020).
• Conversational feedback: Taxonomies ranging from binary to fine-grained 6-class systems ("Positive", "Rephrase", "MakeAwareNoCorr", "MakeAwareCorr", "Clarify", "NoFB") capture nuances in dialogue-based systems (Liu et al., 30 Jul 2025).
• Task structure: Detailed feedback typologies (FB-Bench) designate nine types (e.g., Pointing Out Errors, Raising Objections, Hinting Guidance, Misinformation) that correspond to model response trajectories and error-correction patterns (Li et al., 2024).

The integration of explicit and implicit signals, coupled with typology-aware modeling, expands the scope of WIMHF to operational environments with diverse user bases and multi-modal input streams.

5. Quantitative Impact, Downstream Applications, and Safety

Algorithmic advances in WIMHF have demonstrated substantial downstream benefits:

• Data curation and safety: Identifying features associated with unsafe biases (e.g., user votes against refusals and in favor of toxic content in LMArena) enables targeted relabeling—raising reward-model safety accuracy by +37 pp without loss of global performance (Movva et al., 30 Oct 2025).
• Personalization: Annotator-specific random-slopes models on subjective features allow fine-grained preference adaptation, improving held-out AUC by up to +1.1 pp with a few personalized samples.
• Model regression: Feedback forensic analysis surfaces rewarded/penalized personality traits, permitting intentional control of model stylistics and detection of sycophancy/overfitting (Findeis et al., 30 Sep 2025).
• Bias correction in RL: Protocols such as Stochastic TAMER inject negative feedback stochastically when annotator bias is detected, recovering high-return policies even under severe feedback skew (Eggers et al., 7 Feb 2025).

6. Practical Considerations, Methodological Limitations, and Open Problems

While WIMHF provides robust pipelines for extracting and mitigating key signals, several caveats remain:

• Reliance on rich embeddings and LLM interpretation pipelines: Feature extraction pipelines assume that pretrained embeddings and LLM annotators can reliably expose all relevant axes of variation; subtler or domain-specific features may evade detection (Movva et al., 30 Oct 2025).
• Noisy and context-dependent feedback: Both explicit and implicit streams are subject to noise, context interplay, under-specification, and annotator fatigue, necessitating continuous model validation and active query selection (Liu et al., 30 Jul 2025, Kaufmann et al., 2023).
• Dataset mixing risk: Many preference features flip sign across datasets (e.g., jokes/informality preferred on Reddit, disfavored in HH-RLHF), cautioning against naïve merging of preference data (Movva et al., 30 Oct 2025).
• Implicit feedback mapping constraints: While facial gestures and head cues encode task-relevant information, existing mappings are subject to observer context, sample diversity, and limited window scope (Cui et al., 2020).
• Personality drift and inconsistency: Human feedback pipelines must monitor and, if needed, constrain model personality drift due to overemphasis on particular traits, especially in open-ended or leaderboard-driven collection settings (Findeis et al., 30 Sep 2025).

In summary, WIMHF encompasses a suite of model-driven, interpretable, and bias-aware techniques enabling researchers and practitioners to systematically interrogate, interpret, and adapt the information contained in human feedback. These frameworks undergird improved RLHF reliability, greater safety in model deployment, and finer control of both behavioral and stylistic model outputs.