Preference Propagation in Multi-Agent Systems

• Preference propagation is the dynamic transmission and integration of preferences among agents or states, enhancing collective decision-making in domains like imitation learning and recommender systems.
• Algorithmic implementations such as Predictive Preference Learning and agentic recommender systems use bootstrapping, message passing, and Markov chain models to effectively propagate preferences.
• Empirical studies show that preference propagation reduces human interventions and boosts performance in safety-critical scenarios, leading to improved convergence and adaptive consensus.

Preference propagation refers to the dynamic transmission, inference, and integration of preferences across agents, states, or network nodes, enabling the influence of one entity’s expressed or learned preferences to affect decision-making or optimization in other contexts. Preference propagation arises in machine learning, recommender systems, graph-based inference, voting, and imitation learning, where it underlies collective adaptation, efficient credit assignment, and robust consensus formation.

1. Formal Mechanisms of Preference Propagation

In imitation and interactive learning systems, preference propagation operationalizes the extension of a single human intervention’s feedback beyond the present state to an imagined or predicted rollout of future states. For example, Predictive Preference Learning (PPL) introduces a mechanism in which an expert’s corrective action at a current state $s$ signals a preference not just at $s$, but also propagates to each of the next $L$ predicted states $\{\tilde s_1, \ldots, \tilde s_L\}$, termed the "preference horizon." Each such intervention is bootstrapped into $L$ tuples $(\tilde s_i, a^+, a^-)$ indicating that the expert’s action $a^+$ is preferred to the agent’s action $a^-$ at all $\tilde s_i$. These are persisted in a preference buffer, greatly amplifying the impact of sparse supervision and efficiently expanding exploration coverage in safety-critical regions (Cai et al., 2 Oct 2025).

Agentic recommender systems (e.g., RecNet) generalize preference propagation to interactive networks of users and items. Here, real-time user or item profile updates are routed through a network of "router agents" which aggregate and integrate preference changes, dynamically disseminating them to the most relevant downstream agents. A personalized reception mechanism, combining message buffers and filter memories, integrates incoming propagated preferences conditionally based on learned rules, enabling accurate, continual adaptation of user and item profiles (Li et al., 29 Jan 2026).

In collective decision-making, such as proxy voting, preference propagation is modeled via fractional delegation networks, represented as absorbing Markov chains. Voters distribute fractional weights among other voters and directly to proposals. Through Markovian propagation, each voter’s preference is recursively delegated, with final consensus determined by the absorption probabilities into proposal nodes (Sakai et al., 18 Apr 2025).

2. Mathematical Foundations and Objective Formulations

Preference propagation mechanisms are grounded in rigorous mathematical frameworks depending on context.

Imitation Learning: In PPL, the joint policy $\pi_n$ is trained on a combined behavioral cloning loss and a contrastive preference loss: $$\mathcal{L}(\pi_\theta) = \mathcal{L}_{\rm BC}(\pi_\theta) + \mathcal{L}_{\rm pref}(\pi_\theta)$$ where the contrastive preference loss enforces the propagated expert preference over $L$ imagined states, enabling preference signals to regularize and accelerate policy optimization (Cai et al., 2 Oct 2025).

Graph-based Propagation: In stochastic blockmodel-based community detection, the label propagation process is formally connected to a Gaussian stochastic blockmodel. Node "preferences" $p_i$ correspond to intra-community eigenvector centrality, and label propagation with preference is equivalent to maximizing a likelihood that integrates these propagated nodal influences (Zhang et al., 2014).

Proxy Voting: In propagational proxy voting, the transition matrix $W$ between voters and proposals is used to formulate the Markov chain. The fundamental matrix $N=(I_n-Q)^{-1}$ quantifies expected propagation, and the aggregated preference (consensus) for proposals is given by

$\mathbf{c} = \mathbf{1}_n^\top B$

with $B = N R$, where $Q$ describes intra-voter transitions and $R$ encodes direct voter-to-proposal preferences (Sakai et al., 18 Apr 2025).

3. Algorithmic Implementations

A range of algorithmic strategies instantiate preference propagation:

PPL (Predictive Preference Learning): The algorithm cycles over environment steps with the following workflow:

1. For each step, predict a short horizon rollout under the agent's proposed action.
2. Present the predicted trajectory to the human; if intervention occurs, the correction is propagated as preference labels to the next $L$ predicted states.
3. Update both the behavioral cloning and the contrastive preference objectives and iterate (Cai et al., 2 Oct 2025).

RecNet (Agentic Recommender Systems): RecNet’s two-phase pipeline alternates:

• Forward phase: Router agents aggregate and broadcast preference updates; client agents assimilate updates using buffers and filter memories.
• Backward phase: Real user feedback triggers credit assignment and natural language module refinement via LLMs, enabling continuous self-optimization of the propagation network (Li et al., 29 Jan 2026).

Proxy Voting: Given a voting matrix, the consensus and node influence are efficiently computed by partitioning the Markov transition matrix, computing the fundamental matrix and absorption probabilities, and aggregating the total influence for each node (Sakai et al., 18 Apr 2025).

Domain	Formalism	Propagation Mechanism
Imitation Learning	Preference horizon, contrastive loss	Bootstrapping interventions to future rollout
Recommender Systems	Router/message-passing, agent networks	Attribute and rule-based dissemination
Proxy Voting	Absorbing Markov chains	Fractional delegation and absorption

4. Theoretical Analysis of Propagation Effects

In PPL, the ability of preference propagation to reduce human workload and improve safety-coverage is theoretically justified. The optimality gap between learner and expert is bounded in terms of (i) optimization error, (ii) misalignment of propagated preferences, and (iii) distributional mismatch. As propagation horizon $L$ increases, on-policy state coverage improves, but label fidelity may degrade, necessitating trade-off calibration (Cai et al., 2 Oct 2025).

For proxy voting, propagation is represented via matrix inversion, ensuring that preferences are both recursively delegatable and resilient to subgraph structure. Influence scores derived from the fundamental matrix reflect the centrality and impact of participants in consensus formation (Sakai et al., 18 Apr 2025).

In network community detection, node preference propagation (through eigenvector-based centrality) is grounded in maximum-likelihood estimation, guaranteeing statistical efficiency for label propagation algorithms and resistance to limitations such as the resolution limit (Zhang et al., 2014).

5. Empirical Validation Across Domains

Preference propagation frameworks have demonstrated significant empirical benefits:

• Imitation Learning: In both autonomous driving (MetaDrive) and robotic manipulation (Robosuite), PPL achieved target performance (∼80% success) in fewer environment steps and with ∼30% fewer human interventions compared to state-of-the-art interactive imitation learning baselines. Ablations removing preference loss or propagation severely degrade learning outcomes (Cai et al., 2 Oct 2025).
• Recommender Systems: RecNet achieves 5–15 NDCG-point improvements over leading LLM-based and traditional baselines. Ablations confirm key gains arise from both the routing architecture and personalized reception/optimization modules. Router propagation also improves cold-start performance (Li et al., 29 Jan 2026).
• Proxy Voting: In experiments, the consensus results and influence scores effectively capture nuanced preference distribution and centrality, with algorithmic complexity scaling with network size but remaining practical for real-world participatory settings (Sakai et al., 18 Apr 2025).
• Community Detection: Preference-based label propagation (via eigenvector centrality) matches or exceeds vanilla LPA, LPA-RandomWalk, and variational Bayes SBM on benchmarks including LFR and ER graphs, as well as real-world networks (Zhang et al., 2014).

6. Extension to Related Paradigms

Preference propagation unifies and generalizes various signal transmission methods in multi-agent systems, graph inference, and collective optimization:

• In imitation learning, it stands as an alternative to episodic aggregation, extending local supervision to global operational safety.
• In recommender systems, it incorporates network-structured, real-time feedback, transcending point-to-point embeddings and pointwise optimization.
• In voting, it provides a rigorous probabilistic semantics for delegation and influence, capturing distributed agency in consensus mechanisms.

A plausible implication is that preference propagation mechanisms, by enhancing the expressiveness and effect range of sparse or noisy supervision, can serve as architectural primitives for scalable, efficient, and adaptive collective intelligence systems.