Preference elicitation and alignment among strategic agents in MOMARL

Develop mechanisms and learning frameworks for multi-objective multi-agent reinforcement learning in which agents concurrently learn their users’ utility functions and optimal policies despite misaligned incentives and strategic behaviour (including hiding or misrepresenting preferences). Specifically, design negotiation, communication, or social-contract protocols and accompanying algorithms that elicit and align preferences across agents and stakeholders in individual-utility settings, ensuring robust performance when agents may benefit from not sharing preferences openly.

Background

In single-agent settings, interactive preference elicitation can align policies with user trade-offs over objectives. In multi-agent settings with individual utilities, however, agents may have incentives not to reveal their preferences or may benefit from actively hiding them, complicating preference elicitation and coordination.

The paper highlights that interactive MOMARL—where agents simultaneously learn how to act and model users’ preferences—has not yet been explored. Addressing misalignment and strategic concealment calls for new protocols and algorithms (e.g., negotiations or social contracts) that can reliably elicit preferences and enable aligned multi-agent behaviour.

References

Overcoming the difficulties posed by misalignment of preferences, as well as the fact that it might no longer be in the agents' best interest to share their preferences openly (on the contrary, it might even be better to actively hide this information) are still very much open challenges.

— MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning  (2407.16312 - Felten et al., 2024) in Section 7.2 (Utility Modelling and Preference Elicitation)

Beyond these procedural design questions, one important set of complex open questions in deliberative democracy arises from the fundamental fact that the space of policy options is massive. This makes deliberation categorically different from ordinary voting problems. The alternatives are not fixed in advance, but are generated and refined during the process itself. A mechanism must therefore explore the space of possible proposals while simultaneously eliciting and aggregating preferences over them.

— The New Mathematics of Democracy  (2608.16869 - Flanigan et al., 17 Aug 2026) in Section 4, subsection “Frontier work and open questions,” paragraphs beginning “Beyond these procedural design questions”