Heterogeneous Knowledge Preference Tuning

• Heterogeneous knowledge preference tuning is a framework for addressing fine-tuning and alignment challenges when models must resolve conflicts among multiple knowledge sources and diverse user preferences.
• It employs methods like direct policy alignment, hierarchical ordering, and retrieval control, with studies reporting significant improvements in metrics such as F1 and EM scores.
• Various approaches integrate explicit preference hierarchies and multi-model fusion techniques to ensure fair, consistent, and effective aggregation of heterogeneous information.

Searching arXiv for papers on heterogeneous knowledge preference tuning and closely related formulations. Heterogeneous knowledge preference tuning denotes a family of fine-tuning and alignment problems in which a model must resolve competition among multiple knowledge sources, multiple knowledge types, or multiple preference populations rather than assuming a single homogeneous reward or evidence source. In recent work, closely related formulations include direct alignment with heterogeneous user types and population-weighted rewards (Shirali et al., 22 Feb 2025), explicit preference hierarchies among instruction knowledge, contextual knowledge, and parametric knowledge (Zhou et al., 2024), controllable knowledge selection in retrieval-augmented generation (Zhang et al., 2024), and task-specific tuning procedures that inject, align, or fuse heterogeneous evidence such as emotional, causal, dictionary, graph, and retrieval signals (Mu et al., 20 Jul 2025, Li et al., 2022, Min et al., 2024). Across these formulations, the central technical question is how preference should be represented, observed, optimized, and aggregated when the underlying knowledge is heterogeneous.

1. Conceptual scope and recurring problem structure

Recent papers treat heterogeneity along several distinct axes. One axis is user heterogeneity, where different annotator types have different latent rewards over the same prompt–response pair (Shirali et al., 22 Feb 2025). A second axis is knowledge-source heterogeneity, where a LLM must choose among instruction knowledge, contextual knowledge, and parametric knowledge under an explicit total order (Zhou et al., 2024). A third axis is evidence-type heterogeneity, where the available support spans different modalities or structures, such as Text, Info, Table, and KG in heterogeneous retrieval, or phonetic, visual, and definition knowledge in Chinese Spell Checking (Min et al., 2024, Li et al., 2022). A fourth axis is model-source heterogeneity, where preferences are distilled from multiple source LLMs into a single target policy (Yang et al., 6 Mar 2025, Zhong et al., 9 Apr 2025).

Regime	Heterogeneous elements	Representative formalism
Direct alignment	User types $U$, rewards $r_u$, population prior $p(u)$	Average-reward objective $J_{\rm avg}$
Knowledge hierarchy	Instruction, context, parametric knowledge	$I \succ C \succ P$
Retrieval control	Text, Info, Table, KG	Unified retrieval space with type-balanced or type-preferred loss
Task-specific injection	Emotional/causal, phonetic/visual/definition, visual/text graph structure	Data mixing, contrastive tuning, graph adaptation
Multi-model fusion	Multiple source LLMs with distinct strengths	SFT plus preference optimization over pooled or weighted pairs

This recurring structure makes “preference” broader than pairwise preference labels alone. In some papers, preference is expressed as a reward functional over user types; in others, it is encoded as a hierarchy over knowledge sources, as a negative-sampling policy in retrieval, or as a weighting over multiple source-model outputs. This suggests that the topic is best understood as a generalization of alignment and fine-tuning under non-homogeneous information.

2. Direct policy alignment under heterogeneous user preferences

A precise formalization appears in the direct alignment literature. The setting assumes a finite set of user types $U$, where each type $u \in U$ has a latent reward function $r_u(x,a)$, a population prior $p(u)$, and a single stochastic policy $\pi_\theta(a \mid x)$ that is unaware of the user’s type at test time. The canonical objective is the population-weighted average reward

$r_u$0

The same work distinguishes three annotator-information settings: minimal feedback or “anonymous” feedback, paired-preferences where two labels are known to come from the same user, and full-feedback where every prompt–pair is judged by one representative of each type (Shirali et al., 22 Feb 2025).

The direct losses differ sharply across these settings. Anonymous-DPO corresponds to standard DPO on mixed labels. In the paired-preferences setting, a first-order correction estimates a variance term $r_u$1 and uses a corrected likelihood. In the full-feedback setting, Proposition 5 shows that a consistent DPO-style loss exists if one trains only on the unanimous-agreement subset, yielding the agreement loss. All three losses are differentiable and are optimized by forming mini-batches of the appropriate tuples and running gradient descent such as Adam (Shirali et al., 22 Feb 2025).

The theoretical results are restrictive. Theorem 3.3 states that if $r_u$2, no algorithm on anonymous preferences can learn the ranking induced by $r_u$3. Theorem 5.5 states that when $r_u$4 and preferences follow BT, any direct loss that is consistent for $r_u$5 must discard all samples on which not every annotator agrees. The paper therefore identifies a fundamental tension: consistency requires a loss whose minimizer is exactly the policy maximizing the true average reward, while sample efficiency requires using every preference label; under the stated conditions, both cannot be achieved simultaneously (Shirali et al., 22 Feb 2025).

The practical implications are equally direct. Without annotator metadata, standard DPO or RLHF will approximately maximize Borda count rather than the true average reward, and may produce majority-biased outputs. Tracking annotator IDs allows variance estimation and first-order correction. Collecting full feedback permits consistent recovery of the average-reward optimum, but only through a sample-inefficient unanimous-agreement loss. The same paper also recommends an indirect route: train separate reward models $r_u$6 for each type, form the empirical average $r_u$7, relabel anonymous data with $r_u$8, and then apply ordinary DPO (Shirali et al., 22 Feb 2025).

A common misconception is that heterogeneous preferences can be handled simply by averaging observed binary labels with no additional structure. The direct alignment results reject that view: minimal information can yield first-order improvements, but anonymous preferences are insufficient for learning the ranking induced by the true average-reward objective.

3. Preference hierarchies over parametric, contextual, and instruction knowledge

A distinct line of work defines knowledge preference as an explicit hierarchy among inference-time knowledge sources. One formulation partitions available knowledge into parametric knowledge $r_u$9, contextual knowledge $p(u)$0, and instruction knowledge $p(u)$1, and imposes the order

$p(u)$2

The associated inference rule is: if instruction specifies a fact, answer with the instruction-derived fact; else if context specifies a conflicting fact, answer with the context-derived fact; else answer with the parametric fact. The same paper synthesizes approximately $p(u)$3K hierarchical-preference examples in two streams—counterfactual and conflicting-factual—and fine-tunes Mistral-v0.3-7B with LoRA on the union of $p(u)$4K Alpaca instructions and $p(u)$5K HierPref data (Zhou et al., 2024).

The empirical gains on benchmark suites are large. On IfQA with gold passages, Mistral with Alpaca 0-shot scores $p(u)$6 and $p(u)$7, whereas Mistral with HierPref 0-shot scores $p(u)$8 and $p(u)$9. On MRQA with gold contexts, the scores move from $J_{\rm avg}$0, $J_{\rm avg}$1 to $J_{\rm avg}$2, $J_{\rm avg}$3. On CounterMemoryMRQA, the probability of updating to the context answer rises from approximately $J_{\rm avg}$4 to $J_{\rm avg}$5–$J_{\rm avg}$6, while incorrect updating falls from approximately $J_{\rm avg}$7 to $J_{\rm avg}$8–$J_{\rm avg}$9. Although the paper defines a primary loss and a contrastive penalty, it reports that in practice a single-term cross-entropy with carefully synthesized examples suffices (Zhou et al., 2024).

KnowPO addresses a narrower but operationally important case: retrieval-augmented generation under knowledge conflicts. It distinguishes two failure modes. In Contextual Ignorance, the model ignores correct, relevant retrieved evidence and reverts to its parametric answer. In Contextual Overinclusion, it hallucinates or misuses irrelevant or conflicting passages. The method constructs balanced preference pairs over these scenarios, rewrites samples into a unified template to mitigate length imbalance, and keeps the error-type ratio near $I \succ C \succ P$0. On SQuAD2.0-Eval with Baichuan2-7B-Chat, the base model has $I \succ C \succ P$1, KAFT has $I \succ C \succ P$2, and KaPO reaches $I \succ C \succ P$3. The same paper reports robust gains on RGB, KNOT, and CMB, including $I \succ C \succ P$4 on RGB and $I \succ C \succ P$5 on KNOT (Zhang et al., 2024).

Together, these works establish that preference over knowledge sources is not reducible to prompt wording alone. One paper shows that hierarchy can be instilled through synthesized conflict episodes and ordinary sequence-level supervision; the other argues that instruction tuning without explicit negative signals can still leave the model vulnerable to contextual ignorance and contextual overinclusion. This suggests that the relevant design choice is not merely “use retrieval” or “use instruction tuning,” but how conflict cases are constructed and how lower-priority knowledge is demoted.

4. Instruction-aware retrieval over heterogeneous evidence spaces

In heterogeneous retrieval, preference is implemented at the retriever level rather than solely at answer generation. UniHGKR defines a unified retrieval space over four evidence types,

$I \succ C \succ P$6

and uses a shared encoder for both queries and evidence. The training pipeline has three stages: heterogeneous self-supervised pretraining with a masked auto-encoder objective on data–text pairs, text-anchored heterogeneous embedding alignment with an InfoNCE loss, and instruction-aware retriever fine-tuning with hard negatives and two losses tailored to two retrieval scenarios (Min et al., 2024).

The final fine-tuning stage distinguishes an all-sources scenario and a type-specific scenario. For all-sources retrieval, the type-balanced loss samples roughly equal negatives from each type. For type-specific retrieval, the type-preferred loss drastically reduces negatives of the requested type while keeping ample negatives from other types, thereby biasing the retriever toward the instructed evidence type. Instructions are explicit, with forms such as retrieving from all knowledge sources or retrieving only from Tables (Min et al., 2024).

The benchmark introduced with the retriever, CompMix-IR, contains approximately $I \succ C \succ P$7M entries: $I \succ C \succ P$8M Text pieces, $I \succ C \succ P$9M KG triples, $U$0M Tables, and $U$1M Infoboxes. The QA set contains $U$2 queries split into $U$3 train, $U$4 dev, and $U$5 test examples, with five instruction sets and twenty paraphrases per set. In the all-sources scenario, UniHGKR-base improves MRR@100 from $U$6 for the best prior system to $U$7, with Hit@5 rising from $U$8 to $U$9. In the type-specific scenario, Table-Hit improves from $u \in U$0 to $u \in U$1, which the paper reports as a $u \in U$2 relative improvement. When used as the retriever in a FiD reader on ConvMix, UniHGKR-7B reaches $u \in U$3 and $u \in U$4 (Min et al., 2024).

This retrieval-centered formulation is important because it relocates preference control upstream. Instead of asking the generator to arbitrate among already retrieved heterogeneous evidence, the retriever itself is trained to obey user instructions about evidence type. A plausible implication is that heterogeneous knowledge preference tuning can be decomposed into multiple control layers: retrieval, conditioning, and response generation.

5. Task-specific heterogeneous knowledge injection and representation shaping

Several task-specific systems implement heterogeneous knowledge preference tuning by selecting, mixing, or structurally encoding auxiliary knowledge during fine-tuning. MEKiT for Emotion-Cause Pair Extraction injects two heterogeneous knowledge types: internal emotional knowledge and external causal knowledge. Internal emotional knowledge is obtained from COMET-BART trained on ATOMIC$u \in U$5 via the xReact relation, then mapped to the seven fine-grained emotion labels with SBERT cosine similarities; when COMET returns None, which occurs approximately $u \in U$6 of the time, the system falls back to a POSITIVE/NEGATIVE polarity classifier. External causal knowledge is drawn from a subset of the FLAN instruction-tuning corpus selected by semantic similarity to ECPE documents. The two sources are unified by instruction templates and a data-mixing strategy over $u \in U$7 and $u \in U$8, with $u \in U$9. The best reported setting is $r_u(x,a)$0, corresponding to ratio $r_u(x,a)$1, which yields $r_u(x,a)$2, $r_u(x,a)$3, and $r_u(x,a)$4. Ablations show that removing emotional knowledge drops $r_u(x,a)$5 by approximately $r_u(x,a)$6, removing causal knowledge drops $r_u(x,a)$7 by approximately $r_u(x,a)$8, and removing both drops approximately $r_u(x,a)$9 (Mu et al., 20 Jul 2025).

LEaD for Chinese Spell Checking uses a different mechanism: it mines heterogeneous dictionary knowledge in three modalities—phonetic, visual, and definition—and converts each modality into modality-specific positive and negative samples. Training uses a unified contrastive objective with InfoNCE temperature $p(u)$0, combined with the standard CSC cross-entropy loss. The CSC encoder $p(u)$1 is initialized from Chinese BERT-BASE, while the modality-specific encoders are frozen. At inference time, no auxiliary encoders are used; only the fine-tuned $p(u)$2 and CSC head remain. On sentence-level correction $p(u)$3, vanilla BERT scores $p(u)$4 on SIGHAN15, whereas the full $p(u)$5 system reaches $p(u)$6; on SIGHAN13 and SIGHAN14, the gains are from $p(u)$7 to $p(u)$8 and from $p(u)$9 to $\pi_\theta(a \mid x)$0, respectively (Li et al., 2022).

HeGraphAdapter applies the same general idea to few-shot VLM tuning. It constructs a unified heterogeneous graph $\pi_\theta(a \mid x)$1 with positive-text nodes, negative-text nodes, and visual nodes, together with edge types for intra-modality, inter-modality, and inter-class relations. A heterogeneous graph neural network performs negative-text aggregation, positive-text aggregation, and visual aggregation, after which the system trains both a text-based classifier and a visual-based classifier with joint loss $\pi_\theta(a \mid x)$2. The “knowledge preference” signal is encoded through positive nodes representing what the class is and negative nodes representing what the class is not. On eleven benchmark datasets with CLIP ResNet-50, the reported average improvement over GraphAdapter is $\pi_\theta(a \mid x)$3 in 1-shot, $\pi_\theta(a \mid x)$4 in 2-shot, $\pi_\theta(a \mid x)$5 in 4-shot, and $\pi_\theta(a \mid x)$6 in 16-shot settings; for domain generalization from ImageNet 16-shot, average target accuracy rises from $\pi_\theta(a \mid x)$7 to $\pi_\theta(a \mid x)$8 (Zhao et al., 2024).

HKFR for recommendation illustrates a text-only fusion variant. User heterogeneous behaviors are templated into behavior strings, passed through ChatGPT with a fusion prompt to produce a natural-language heterogeneous knowledge text $\pi_\theta(a \mid x)$9, and then used to fine-tune ChatGLM-6B with LoRA for recommendation. On the category task, HKFR reaches $r_u$00, $r_u$01, $r_u$02, and $r_u$03, compared with $r_u$04, $r_u$05, $r_u$06, and $r_u$07 for P5. In online A/B testing on cold-start users, the system reports $r_u$08 CTR and $r_u$09 GMV (Yin et al., 2023).

These systems do not share a single loss family, but they share an architectural stance: heterogeneous auxiliary knowledge is treated as a controllable training signal rather than as a static feature concatenation step. Depending on the task, that control appears as data ratios, contrastive pairs, graph edges, or natural-language summaries.

6. Fusion, aggregation, identifiability, and broader interpretations

A separate strand studies heterogeneous preference tuning when the heterogeneity comes from multiple source models rather than multiple knowledge documents. FuseChat-3.0 constructs a target model by combining responses from four source LLMs—Gemma-2-27B-it, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct—through two stages: supervised fine-tuning on selected high-quality responses and DPO on pooled intra-model preference pairs. Using Llama-3.1-8B-Instruct as target, the average score over fourteen benchmarks rises from $r_u$10 for the base model to $r_u$11 after SFT and $r_u$12 after LN-DPO. The instruction-following gains are especially large: AlpacaEval-2 moves from $r_u$13 to $r_u$14, and Arena-Hard from $r_u$15 to $r_u$16 (Yang et al., 6 Mar 2025).

FuseRL makes the fusion signal denser. For each prompt and source model, it selects the highest-reward response, computes a softmax-based model weight $r_u$17, and uses these weights in both a weighted SFT stage, FuseSFT, and a weighted preference-optimization stage, FusePO. FusePO is designed to work with RLOO, DPO, and SimPO by replacing the usual single-pair objective with a weighted sum over many source-specific winning–losing pairs. On AlpacaEval-2 with Llama-3.1-8B-Instruct, $r_u$18 rises from $r_u$19 to $r_u$20 for RLOO, from $r_u$21 to $r_u$22 for SimPO, and from $r_u$23 to $r_u$24 for DPO. The paper also reports that increasing the number of source models from $r_u$25 to $r_u$26 yields another $r_u$27 points in $r_u$28 (Zhong et al., 9 Apr 2025).

When heterogeneous preferences come from humans rather than source LLMs, aggregation and personalization become central. The RLHF framework based on heterogeneous feedback proposes a personalization-based route using representation learning or clustering to learn multiple reward models, and an aggregation-based route using utilitarian or Leximin reward aggregation, probabilistic-opinion aggregation, and a mechanism-design approach that makes truthful reporting a dominant strategy while maximizing social welfare (Park et al., 2024). This work is notable because it formalizes the bias–variance trade-off of personalization and introduces aggregation rules that are explicitly fairness-aware.

A stronger identifiability result appears in later work on unobserved preference heterogeneity. That paper connects RLHF preference learning to random-coefficients logit models in econometrics and proves that binary comparisons are insufficient for identifying latent user preferences, while rankings over three or more responses ensure identifiability under stated conditions. It then introduces EM-DPO, which alternates between computing responsibilities over latent annotator types and performing weighted DPO updates, and proposes a min–max regret aggregation criterion for producing a single fair policy when user type is unknown at inference time (Chidambaram et al., 17 Oct 2025). This result directly challenges the widespread reliance on binary pairwise feedback as a sufficient basis for heterogeneous alignment.

A broader interpretation of heterogeneous tuning appears in knowledge graph embeddings. Bamler, Salehi, and Mandt introduce per-entity and per-relation regularization strengths and learn thousands of such hyperparameters with variational EM. This is not response-level preference optimization, but it is a form of heterogeneous tuning in which different entities and relations receive different regularization “preferences.” On FB15K with DistMult, the paper reports MRR improving from $r_u$29 to $r_u$30 and Hits@10 from $r_u$31 to $r_u$32 (Bamler et al., 2019).

Taken together, these results indicate a consistent shift away from the homogeneity assumption. In the current literature, heterogeneous knowledge preference tuning is not a single algorithmic recipe but a research program spanning direct alignment, retrieval control, knowledge injection, model fusion, fairness-aware aggregation, and identifiability theory. The common conclusion is that once knowledge or preferences are heterogeneous, the training objective, the data-collection protocol, and the aggregation rule must be designed with that heterogeneity as a first-class object rather than treated as noise.