FaST: Feature-aware Sampling and Tuning

• FaST is a low-data personalization method (PPALLI) that tailors LLM responses using a parameter-efficient reward model called FaRM.
• It employs an iterative sampling-and-tuning loop on a fixed questionnaire to fine-tune responses based on user-selected annotations.
• Benchmark evaluations on DnD and ELIP show FaST’s superior accuracy and efficiency, even with as few as 16 user annotations.

FaST denotes Feature-aware Sampling and Tuning, a method for Personalized Preference Alignment with Limited Data (PPALLI), the setting in which an LLM-driven assistant is tailored to an individual user’s tastes from only a small number of preference annotations per user, typically a few dozen and in the stated formulation $\lesssim 100$ (Thonet et al., 6 Aug 2025). In PPALLI, all users answer a fixed questionnaire consisting of shared contexts and multiple candidate responses, and the resulting user-specific selections are used to fine-tune a model so that future generations align with that user’s latent preferences. FaST addresses this problem with a highly parameter-efficient reward model, termed FaRM, together with an iterative sampling-and-tuning loop for generation. The method is introduced alongside two questionnaire-based datasets, DnD and ELIP, and is reported to achieve the best overall performance among the benchmarked approaches in this low-data personalization regime (Thonet et al., 6 Aug 2025).

1. Problem formulation and scope

PPALLI addresses the practical scenario in which one wishes to tailor a large-language-model assistant to individual users’ tastes, yet can collect only a small number of preference annotations per user (Thonet et al., 6 Aug 2025). The setup assumes a fixed questionnaire: a shared set of $D$ contexts, each paired with $K$ candidate responses. For a new user, the user reviews this questionnaire and, for each context $i \in \{1 \ldots D\}$, selects exactly one preferred response $r_i$ among $\{r_{i,1}, \ldots, r_{i,K}\}$.

The resulting dataset is

$$D^{(u)} = \{(q_i, \{r_{i,k}\}_{k=1}^K, r^{(u)}_i)\}_{i=1}^D,$$

where $r^{(u)}_i$ is the user-chosen reply (Thonet et al., 6 Aug 2025). The goal is then to use $D^{(u)}$ to fine-tune an LLM so that it generates future answers aligned to that user’s latent tastes, despite having only a few dozen examples per user.

This formulation imposes two central constraints. First, all users are evaluated on the same user-agnostic questionnaire. Second, the personalization signal is sparse, since the available supervision consists of a single selected response per context. FaST is designed specifically for this regime. A plausible implication is that the method is less a generic preference-alignment framework than a structured solution to questionnaire-based personalization under extreme sample scarcity.

2. Benchmark datasets for PPALLI

To support research in PPALLI, the work introduces two high-quality, questionnaire-based datasets: DnD and ELIP (Thonet et al., 6 Aug 2025). Both share three stated properties: a fixed, user-agnostic questionnaire; per-user preference annotations over the same options; and interpretable profile structure.

Dataset	Shared questionnaire	User structure
DnD	129 in-game situations, each with $K=3$ alternative actions	10 distinct characters defined by race, class, moral alignment, background, and personality
ELIP	100 diverse ELI5 questions, each with $D$0 GPT-4o–generated answers	All 8 combinations of Expertise, Informativeness, and Tone

DnD (“Dungeons and Dragons”) simulates role-playing. It comprises 129 in-game situations and, for each, $D$1 alternative actions. Ten distinct characters are defined by race, class, moral alignment, background, and personality, with the paper giving “Grog,” a Chaotic-Neutral half-orc barbarian, as an example (Thonet et al., 6 Aug 2025). GPT-4o generated situations, actions, and, by role-playing each character, the preferred action. The dataset is intended to exhibit richly varying traits such as aggression versus caution.

ELIP (“Explain Like I Prefer”) personalizes an open-ended QA assistant. It contains one hundred diverse ELI5 questions, each paired with $D$2 GPT-4o–generated answers (Thonet et al., 6 Aug 2025). User profiles are constructed from all 8 combinations of three binary dimensions: Expertise, Informativeness, and Tone. These dimensions are instantiated as child-friendly versus expert, concise versus detailed, and friendly-humorous versus impersonal. GPT-4o simulated each user’s preferred answer.

The two datasets serve complementary roles. DnD captures preference alignment in a fantasy role-play setting, whereas ELIP targets conversational QA. Because both use a shared questionnaire across users, they are directly compatible with the PPALLI formulation and with FaST’s feature-discovery stage.

3. Feature-aware Reward Model (FaRM)

At the heart of FaST is FaRM, a highly parameter-efficient reward model that decomposes user preferences onto a small set of interpretable features (Thonet et al., 6 Aug 2025). The model operates in three stages: feature discovery, feature scoring, and per-user weight learning.

Feature discovery

Because PPALLI questionnaires are small, with fewer than 100 contexts, FaST prompts a powerful LLM, specifically GPT-4o, with the entire list of context-response sets and asks it to propose $D$3 global features that distinguish among the alternative responses (Thonet et al., 6 Aug 2025). These features are domain-agnostic. The examples given include, for DnD, “risk assessment,” “direct action,” and “stealth,” and for ELIP, “metaphorical analogy,” “humorous tone,” and “technical jargon.”

Each discovered feature is described by an attribute description, an attr_min phrase corresponding to score $D$4, and an attr_max phrase corresponding to score $D$5 (Thonet et al., 6 Aug 2025). This design makes the features interpretable and explicitly ordinal.

Feature functions

For each discovered feature $D$6, a lightweight LLM scorer $D$7 returns a real-valued score in $D$8 for response $D$9 in context $K$0 (Thonet et al., 6 Aug 2025). To capture uncertainty and maximize efficiency, $K$1 is prompted so that the next token is the numeric score, and FaST computes a probability-weighted average over the token distribution rather than a single argmax, described in the paper as being “à la G-Eval.”

Collecting all feature scores yields $K$2. This feature vector becomes the basis for user-specific reward estimation.

Per-user weight learning

FaST then fits a per-user weight vector $K$3 by maximizing the conditional log-likelihood of the observed choices under a McFadden, or multinomial Bradley-Terry, model:

$K$4

with

$K$5

and

$K$6

The objective is stated to be convex in $K$7, so $K$8 can be found efficiently by gradient descent (Thonet et al., 6 Aug 2025). Crucially, FaRM learns only $K$9 parameters per user, with the paper giving $i \in \{1 \ldots D\}$0 as an example, rather than the millions or billions associated with full fine-tuning.

This decomposition is central to FaST’s parameter efficiency. A plausible implication is that the method makes personalization tractable precisely because it separates representation of response attributes from estimation of user-specific trade-offs among those attributes.

4. Sampling-and-tuning personalization loop

Once FaRM is learned, FaST personalizes a generation model by iteratively sampling candidate replies, scoring them with FaRM, and updating the policy via a sampling-and-tuning loop (Thonet et al., 6 Aug 2025). The loop has three stated steps.

First, for each training context $i \in \{1 \ldots D\}$1 in $i \in \{1 \ldots D\}$2, the current LLM policy $i \in \{1 \ldots D\}$3 samples $i \in \{1 \ldots D\}$4 candidate responses. Second, these candidates are scored and ranked by the user-specific reward $i \in \{1 \ldots D\}$5. Third, the model parameters $i \in \{1 \ldots D\}$6 are updated by fine-tuning on the newly ranked context-response pairs.

The update can use either Supervised Fine-Tuning (SFT), yielding the RFT variant, or Direct Preference Optimization (DPO), yielding the Online-DPO variant (Thonet et al., 6 Aug 2025). The method also includes the best reply from the previous iteration in the next sampling step, which is stated to enforce monotonic reward improvement.

FaST is contrasted with Proximal Policy Optimization (PPO). The paper states that, unlike PPO, sampling-and-tuning requires only a ranking and no value network, is more stable, and has far fewer hyperparameters, which are identified as advantages in low-data, per-user fine-tuning (Thonet et al., 6 Aug 2025). This suggests that FaST is designed not merely to improve alignment quality, but also to reduce the operational complexity of personalization.

The overall architecture therefore combines a reward-modeling stage that is low-dimensional and interpretable with a generation fine-tuning stage that remains lightweight enough for individualized deployment.

5. Experimental protocol and empirical results

The empirical evaluation distinguishes preferred response prediction from personalized generation (Thonet et al., 6 Aug 2025).

Preferred response prediction

To validate FaRM itself, the authors held out 25% of contexts for validation and test using 50/25/25 splits and 5 random seeds. Baselines included a random baseline, Many-Shot In-Context Learning, Manyshot-CoT, a full-model reward model, RM-LoRA, the Compositional Preference Model, and FaRM with $i \in \{1 \ldots D\}$7 features using either LLaMA-3.2-3B-Instruct or Phi-4-Mini-Instruct backbones for $i \in \{1 \ldots D\}$8 (Thonet et al., 6 Aug 2025). Performance was measured by accuracy in predicting the user-chosen response.

On DnD test, FaRM reached approximately 63.9–69.4% accuracy, compared with 58–63% for RM-LoRA and CPM (Thonet et al., 6 Aug 2025). On ELIP test, FaRM reached approximately 71.0–75.3%, compared with 71.0–72.8% for CPM and 60.2–71.4% for RM-LoRA. The paper further reports that even with only 16 training examples, FaRM outperformed the full reward model.

Personalized generation

For unseen contexts, the evaluation measured how well a personalized LLaMA-3.2-3B-Instruct generated user-tailored replies (Thonet et al., 6 Aug 2025). Baselines included Zeroshot, Retrieval-Augmented Generation, Manyshot, Manyshot-CoT, SFT, DPO, and reward-model-based combinations such as RM paired with Best-of-N, PPO, Online-DPO, and RFT. FaST was paired with Best-of-N, Online-DPO, and RFT. Two oracle conditions were also included: Oracle-chosen and Oracle-gen with explicit profile access via GPT-4o.

Two LLM-judge metrics were used: a 0–5 rubric-based score and winrate in pairwise comparisons against Zeroshot and Oracle-chosen. Elo rankings and score improvements over Zeroshot were also reported.

The main reported generation result is that FaST + RFT improved personalization scores over Zeroshot by +0.46 on DnD and +0.57 on ELIP, surpassing RM + RFT at +0.37 and +0.49, and DPO at +0.32 and +0.54 (Thonet et al., 6 Aug 2025). The winrates of FaST + RFT versus Zeroshot were approximately 80.5% on DnD and 73.8% on ELIP. Against Oracle-chosen, FaST + RFT still won approximately 58.9% on DnD and 47.3% on ELIP. The Elo rankings placed FaST + Online-DPO and FaST + RFT among the very top non-oracle methods for both datasets.

The method is also reported to remain strong with only 16 annotations and when fine-tuning via LoRA, which the authors present as further evidence of parameter efficiency (Thonet et al., 6 Aug 2025).

6. Limitations, interpretation, and future directions

The authors identify three principal limitations (Thonet et al., 6 Aug 2025). The first concerns diversity of candidates: sampling-and-tuning depends on the base LLM’s ability to propose varied replies. The paper suggests that future work may explore prompts that jointly generate diverse lists. The second concerns LLM-judge evaluation: although rubric-based scores and winrates align, human studies are described as essential for validating real-user satisfaction. The third concerns simplified profiles: ELIP’s binary-dimension users enable controlled testing but may not capture the full breadth of genuine preferences.

These limitations define the current scope of FaST. The framework is optimized for settings in which users can answer a shared questionnaire, preferences can be decomposed into high-level features, and evaluation is feasible through synthetic or judge-based proxies. The authors propose future user-in-the-loop studies to collect real annotations and ratings (Thonet et al., 6 Aug 2025).

Within that scope, FaST’s distinguishing properties are clear. It discovers high-level, interpretable features automatically from the questionnaire; learns only $i \in \{1 \ldots D\}$9 parameters per user; and uses an iterative ranking-based fine-tuning loop rather than heavier RL machinery (Thonet et al., 6 Aug 2025). This suggests a broader methodological point: in low-data personalization, explicit decomposition of preferences into shared latent attributes may be more effective than directly fitting a large per-user reward model.