Preference elicitation and alignment among strategic agents in MOMARL

Develop mechanisms and learning frameworks for multi-objective multi-agent reinforcement learning in which agents concurrently learn their users’ utility functions and optimal policies despite misaligned incentives and strategic behaviour (including hiding or misrepresenting preferences). Specifically, design negotiation, communication, or social-contract protocols and accompanying algorithms that elicit and align preferences across agents and stakeholders in individual-utility settings, ensuring robust performance when agents may benefit from not sharing preferences openly.

Background

In single-agent settings, interactive preference elicitation can align policies with user trade-offs over objectives. In multi-agent settings with individual utilities, however, agents may have incentives not to reveal their preferences or may benefit from actively hiding them, complicating preference elicitation and coordination.

The paper highlights that interactive MOMARL—where agents simultaneously learn how to act and model users’ preferences—has not yet been explored. Addressing misalignment and strategic concealment calls for new protocols and algorithms (e.g., negotiations or social contracts) that can reliably elicit preferences and enable aligned multi-agent behaviour.

References

Overcoming the difficulties posed by misalignment of preferences, as well as the fact that it might no longer be in the agents' best interest to share their preferences openly (on the contrary, it might even be better to actively hide this information) are still very much open challenges.

— MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning  (2407.16312 - Felten et al., 2024) in Section 7.2 (Utility Modelling and Preference Elicitation)

Several further strategic issues remain open: coordinated misreporting by multiple agents, platform-side manipulation of feature reports $f_j$, and adaptive learning of $v_1$ under repeated play.

— Multi-Dimensional Matching  (2609.29958 - Aldridge, 24 Sep 2026) in Section 7.2, “Strategic Considerations” (Section \ref{sec:limitations})

Beyond these procedural design questions, one important set of complex open questions in deliberative democracy arises from the fundamental fact that the space of policy options is massive. This makes deliberation categorically different from ordinary voting problems. The alternatives are not fixed in advance, but are generated and refined during the process itself. A mechanism must therefore explore the space of possible proposals while simultaneously eliciting and aggregating preferences over them.

— The New Mathematics of Democracy  (2608.16869 - Flanigan et al., 17 Aug 2026) in Section 4, subsection “Frontier work and open questions,” paragraphs beginning “Beyond these procedural design questions”

Regardless of the implementation, key research questions remain: How can subjective quality preferences be elicited, formalized, and kept up to date as projects evolve? What is the right level of granularity for quality specifications? How should conflicts between different stakeholders' preferences be resolved within such an evaluation framework?

— Continuous Autonomous Refactoring: A Research Roadmap for AI-Driven Code Quality Maintenance  (2609.01236 - Sun et al., 1 Sep 2026) in Section 3.2, The Problem of Definition and Evaluation

We describe an architecture rather than an evaluated system, and several questions remain open. The preference extension and the explanation feedback loop are proposed: the extension and the assistants are not yet built, and the loop has been exercised only on the OWL path. A first evaluation would target the acquisition loop rather than schedule quality: how often the static check catches a preference that could never be satisfied, how many renegotiation turns follow, and whether checking and solver encoding stay within interactive time. A first design question is where the preference semantics belong, the ConfigVoc-versus-ConfigDL choice detailed in Section~\ref{sec:loop}; because both paths supply inconsistency justifications, it governs how preferences are checked, not whether the loop can run. Modeling preferences as weighted soft constraints further raises the questions of how to set and compare priorities across stakeholder groups, and how to handle multiple competing soft constraints (multi-objective optimization); because the groups are self-interested, priority elicitation also carries a social-choice dimension, with the risk of strategic misreporting.

— Ontology-Mediated Neurosymbolic Constraint Acquisition from Multiple Stakeholders  (2609.29876 - Bischof et al., 24 Sep 2026) in Section 6, paragraph “Limitations and open questions”

This decision becomes increasingly challenging as the number of objectives grows and remains an open problem; addressing it via human-in-the-loop methods for multi-objective evaluation and selection is a key direction for future work.

— Learning Pareto Stationary Fronts via Single-Pass Backpropagation  (2610.06397 - Celik et al., 5 Oct 2026) in Section 6, “Limitations and future work”