Preference Estimation via Opponent Modeling in Multi-Agent Negotiation

Abstract: Automated negotiation in complex, multi-party and multi-issue settings critically depends on accurate opponent modeling. However, conventional numerical-only approaches fail to capture the qualitative information embedded in natural language interactions, resulting in unstable and incomplete preference estimation. Although LLMs enable rich semantic understanding of utterances, it remains challenging to quantitatively incorporate such information into a consistent opponent modeling. To tackle this issue, we propose a novel preference estimation method integrating natural language information into a structured Bayesian opponent modeling framework. Our approach leverages LLMs to extract qualitative cues from utterances and converts them into probabilistic formats for dynamic belief tracking. Experimental results on a multi-party benchmark demonstrate that our framework improves the full agreement rate and preference estimation accuracy by integrating probabilistic reasoning with natural language understanding.

• The paper presents a dual-channel Bayesian framework that fuses numerical and linguistic data to estimate agents' private preferences.
• It demonstrates improved negotiation outcomes with a high full agreement rate (0.62) and reduced estimation errors compared to baseline methods.
• The approach enhances interpretability and robustness in automated multi-agent negotiation, highlighting practical and theoretical advances.

Bayesian Opponent Modeling with Natural Language Preference Estimation in Multi-Agent Negotiation

Introduction

Automated negotiation in multi-party, multi-issue environments demands the accurate modeling of agents' private preferences. Traditional opponent modeling techniques depend predominantly on observing numerical bids and inferring utility surfaces, inherently underutilizing the qualitative cues present in agent utterances during negotiation. The referenced paper, "Preference Estimation via Opponent Modeling in Multi-Agent Negotiation" (2604.15687), proposes an integrated Bayesian framework that systematically incorporates qualitative signals from agents' natural language into probabilistic preference estimation. The approach extends established numerical opponent modeling with direct extraction of preferences from negotiation dialogues using LLMs.

Problem Setting and Motivation

Conventional Bayesian or RL-based negotiation frameworks operate effectively under the assumption that all relevant information is embedded in the numerical evolution of bids [opponent-modeling, he2016deep]. This is limiting in practical settings where agents often communicate partial, indirect, or soft constraints through language. While LLMs have recently demonstrated significant capabilities in intent recognition, Theory of Mind (ToM), and context interpretation [Kosinski2024tom], attempts to leverage their generative reasoning directly for decision making in negotiation have shown limited strategic consistency and unstable preference tracking [negotiationtom, zhao-etal-2025-large].

To address these gaps, the proposed method formalizes a dual-channel Bayesian opponent modeling system, where both deal history and language utterances are mapped into likelihoods over a hypothesis space of opponent preference profiles. The key technical contribution is a principled mechanism for combining numerical and linguistically-derived evidence in a tractable inference process.

Methodology

Negotiation Environment and Preference Representation

The environment comprises $N$ negotiating agents and $M$ issues, with each issue $i_m$ offering $K_m$ options. Agents' utility functions are additive sums over issue-option value functions. Each agent possesses a private utility function and a reservation utility (BATNA threshold). Negotiation proceeds in rounds, with each round involving the proposal of a deal and an associated natural language utterance.

Bayesian Opponent Modeling Architecture

The approach adopts a Bayesian estimation process over a finite hypothesis space $H = \{h_1, \dots, h_K\}$, with each hypothesis encoding a possible weight vector over issues and a set of option evaluation functions. The estimated utility of a deal for a given hypothesis follows an additive form:

$$\hat{U}(d_t; h_k) = \sum_{m=1}^{M} w_m^{(k)} v_m^{(k)}(o_t^m)$$

A likelihood over hypotheses is maintained and updated using both numerical and linguistic signals.

Numerical Likelihood

Via a concession-based behavioral assumption, the likelihood of observing a specific deal $d_t$ under hypothesis $h_k$ is modeled as a Gaussian centered on the conjectured concession curve:

$$P(d_t \mid h_k) \propto \exp\left( -\frac{(\hat{U}(d_t; h_k) - u'(t))^2}{2\sigma^2} \right)$$

Here, $u'(t)$ denotes the assumed target utility for round $M$0.

Linguistic Likelihood Extraction

Utterance $M$1 is parsed through a prompted LLM (specifically GPT-4.1) to produce a structured preference signal $M$2 consisting of a target (issue/option or comparison) and a stance (prefer/oppose). Example outputs include signals such as "prefer issue B" or "oppose option D4." (Figure 1)

Figure 1: Schematic of the negotiation protocol and the Bayesian opponent modeling process incorporating deal and natural language analysis.

The likelihood of a linguistic signal under a hypothesis is computed using Luce's Choice Axiom. For an expressed preference for issue $M$3:

$M$4

Likewise, comparisons and option-level stances leverage weight and evaluation vector ratios.

Bayesian Fusion for Preference Update

Assuming conditional independence, the posterior over preference hypotheses after observing both the deal $M$5 and the linguistic signal $M$6 is

$M$7

This process is iterated at each negotiation round, dynamically refining the belief over opponent preference profiles.

Experimental Evaluation

The method is benchmarked in a 6-agent, 5-issue multi-party negotiation scenario modeled after the "Harbour Sport Park" problem, which is characterized by extreme sparsity in the agreement space—only 0.4% of deals permit full consensus. Comparative baselines include:

• Base-LLM: Direct LLM-based negotiation without explicit opponent modeling.
• Base-OM: Standard Bayesian update using only deal history.
• LLM-PE: LLM infers opponent scores directly without probabilistic belief update.
• Proposed (p1/all): The proposed Bayesian method with preference estimation by either only the leader or all agents.

Quantitative Results

The proposed approach, when all agents engaged in mutual opponent modeling ("all"), achieved the highest Full Agreement Rate (FAR) at 0.62 and competitive Partial Agreement Rate (PAR) at 0.89. Notably, preference estimation error (measured in MSE between estimated and true score functions) was lowest for the proposed method among all Bayesian approaches. The use of linguistic likelihoods leads to more balanced error across agent types.

Discussion of Claims and Implications

Strong empirical results: The method achieves higher rates of full agreement and improved preference estimation accuracy compared to both numerical-only and LLM-only baselines. The explicit fusion of LLM-extracted qualitative cues with Bayesian updating is argued to offer both improved accuracy and more stable strategic behavior, particularly in complex, low-information, multi-agent environments.

Contradictory to naive LLM use: Directly prompting LLMs for numerical preference inference (LLM-PE) underperforms, demonstrating the necessity of structured probabilistic reasoning atop LLM-extracted signals. This challenges the utility of purely generative ToM-style inference in practical negotiation.

Practical implications: For practical deployment in automated multi-agent systems and organizational consensus-building, the approach combines model interpretability (explicit beliefs over hypotheses) with improved robustness under information scarcity and ambiguity.

Theoretical implications: The work bridges symbolic probabilistic modeling and neural natural language understanding, highlighting a compositional paradigm for multi-agent reasoning systems. The Bayesian fusion mechanism allows principled quantification of uncertainty and incremental information integration, a key advantage over monolithic neural approaches.

Limitations and Future Directions

• Assumption of Sincerity: The current realization assumes broadly truthful signaling in dialogue. Modeling strategic deception or unreliable utterances is an open dimension, potentially addressed by incorporating a reliability parameter or an adversarial linguistic channel.
• Scalability: Computational cost scales factorially with issue and option cardinality, though known approximation schemes from prior Bayesian opponent modeling can be adopted to mitigate this.
• Generalizability: Empirical validation focuses on a single, high-difficulty negotiation scenario. Broader experiments across different utility structures and agent population sizes will inform the robustness of the framework.
• Joint BATNA Inference: Extending the method to infer opponents' reservation thresholds (BATNAs) online could further improve the proposal quality and agreement rate, especially in highly ambiguous settings.

Conclusion

This paper demonstrates that integrating LLM-based qualitative utterance parsing with a structured Bayesian inference framework materially improves negotiated outcomes and preference estimation accuracy in complex multi-agent, multi-issue negotiation. The approach provides a principled pathway toward more interpretable, robust, and sample-efficient autonomous negotiation agents, with implications for organizational AI deployment and the development of neuro-symbolic multi-agent systems.

---

References:

"Preference Estimation via Opponent Modeling in Multi-Agent Negotiation" (2604.15687) Other citations as referenced in the main paper.