ACE: LLM Negotiation Coaching System

• The paper introduces ACE, a novel LLM-based negotiation coaching system that integrates realistic simulation with live, error-aware feedback.
• It employs a rigorous annotation schema on negotiation transcripts, achieving high inter-annotator reliability and significant user performance improvements.
• The system leverages GPT-4 and GPT-4o in a modular architecture, enabling scalable, on-demand coaching for underrepresented groups.

The Assistant for Coaching nEgotiation (ACE) is a LLM-based negotiation coaching system designed to democratize access to strategic bargaining education by providing users with interactive negotiation practice and structured, targeted feedback. Distinct from prior LLM tutoring systems, ACE integrates negotiation partner simulation and real-time diagnostic feedback within a principled annotation framework grounded in negotiation pedagogy, with a specific emphasis on supporting underrepresented groups lacking access to traditional bargaining instruction (Shea et al., 2024).

1. System Motivation and Core Functionalities

The primary goals of ACE are to (i) provide scalable, on-demand negotiation coaching for distributive bargaining scenarios and (ii) address the limitations of existing AI tutors, which focus mainly on factual domains and lack facility for real-time strategic decision-making. Unlike "canned" or reactive negotiation bots, ACE operates as both (a) a realistic LLM-based negotiation partner—termed "chat agent"—and (b) an automated feedback provider that diagnoses preparation, linguistic, or tactical mistakes and delivers actionable, targeted suggestions for improvement. These functionalities are intended to counteract the evidence that unstructured role-play alone does not foster negotiation skill acquisition and to fill gaps in accessibility for learners otherwise excluded from expert-led seminars [(Shea et al., 2024), §1–2].

2. Data Collection and Annotation Schema

2.1 Negotiation Transcript Corpus

ACE leverages a data corpus composed of negotiation transcripts from 50 MBA students engaged in three successive single-issue distributive bargaining exercises—predominantly automobile transactions (Honda Accord, other used cars). After preprocessing (audio via Whisper API, manual correction), 40 filtered dialogues remain, covering 823 turns with an average of 34 tokens per turn. Scenario continuity yields a 75% deal closure rate [(Shea et al., 2024), §4, Table 1].

2.2 Error Annotation Scheme

An expert-driven binary annotation schema encompasses eight error categories split into preparation and negotiation (per-turn) errors:

• Preparation: Strategic walk-away point, strategic target price.
• Negotiation: Breaking the ice, giving the first offer, ambitious opening point, strong counteroffer, inclusion of rationale, and strategic closing behavior.

These categories are instantiated via formal definitions and formulaic cutoffs. For instance, the "ambitious opening point" for buyers is flagged if the opening offer $O_1$ satisfies $O_1 \leq 0.9 T$, with $T$ the buyer's strategic target price. "Strong counteroffer" requires $O_t < (O_{t-1} + \min(S,\,W))/2$ where $S$ is the seller's offer and $W$ the walk-away [(Shea et al., 2024), §5, Table 2–3].

2.3 Inter-Annotator Agreement and Error Prevalence

Annotation reliability yields Cohen’s $\kappa=0.87$ over 288 turns. Strategic closing and strong counteroffer are the most prevalent errors, flagged in 36/40 dialogues and 34/73 applicable turns, respectively [(Shea et al., 2024), Table 3]. Standard precision, recall, and $F_1$ score definitions are used for error classification performance assessment.

3. Model Architecture and Error Detection Integration

3.1 LLM and Prompt Engineering Design

No supervised fine-tuning is employed; instead, ACE orchestrates prompt-based inference with OpenAI GPT-4 and GPT-4o via API calls. The negotiation agent adopts a constrained instructional prompt specifying reservation price, scenario and market details, and a "subjective limit" heuristic. Error detection is implemented through zero-shot or few-shot GPT-4o classifiers tailored per error type and price-extraction prompts for quantitative errors [(Shea et al., 2024), §6.2–6.4, Tables 11–16].

3.2 Annotation Pipeline Integration

Price-based errors leverage explicit formulaic cutoffs applied to extracted offer values. Language-based errors employ three classifier modules per category, each provided with a definition and in-context expert examples. There is no explicit chain-of-thought requirement; stepwise inference is dictated by prompt engineering strategies.

3.3 Training Paradigm

All models rely exclusively on in-context learning; no gradient-based objectives (e.g., cross-entropy) or explicit loss functions are optimized. The annotation scheme’s logical predicates dictate labeling and feedback structure [(Shea et al., 2024), §6].

4. Feedback Generation and Delivery

4.1 Preparation Feedback

Numeric user responses to preparatory questions are checked against formal criteria. Feedback is generated either through hardcoded messages or via prompt-based contextualization using GPT-4o with exemplar feedback responses [(Shea et al., 2024), §6.3, Appendix E].

4.2 Turn-Based Feedback Loop

The primary pipeline comprises:

1. Error Identification: Sequentially applies price extraction and language classifiers to each turn.
2. Direct Feedback: Each detected error triggers a GPT-4o prompt with error context, category definition, and expert examples to generate targeted advice.
3. Utterance Revision: The same context is used to prompt GPT-4o to rewrite the user’s utterance, correcting flagged errors with congruent illustration [(Shea et al., 2024), §6.4, Fig. 3, Tables 13–14].

4.3 Holistic and Aggregate Feedback

A separate holistic feedback module prompts GPT-4o for global conversational attributes—formality, firmness, and linguistic complexity—with reference to specific utterances in-session [(Shea et al., 2024), Table 15].

4.4 Latency and Engineering Considerations

Average per-turn latency for error classification and feedback generation is 1–2 seconds each, with GPT-4o selected for performance in error modules and GPT-4 for negotiation quality [(Shea et al., 2024), §6, Appendix F].

5. Experimental Validation and Quantitative Outcomes

5.1 User Study Protocol

A controlled user experiment (N=390, 374 valid; 52% female; mean age 34.5) tested ACE across two negotiation tasks. Subjects were randomly assigned to three conditions: ACE feedback (N=119), Other-feedback (zero-shot GPT-4, N=129), and no-feedback (N=126). Pre- and post-task surveys measured motivation, process, and subjective reflection [(Shea et al., 2024), §7, Tables 6–8].

5.2 Performance Metrics and Statistical Analysis

Objective performance (final deal price; lower better for buyers) and subjective improvement were measured. In Trial 2 (sublease), only the ACE arm demonstrated significant negotiation performance improvement ($t$-test $t(118) = -2.97,\,p=0.003,\,d=0.38$); Other-feedback and No-feedback were non-significant. A two-way mixed ANOVA confirmed ACE’s improvement over other arms [$O_1 \leq 0.9 T$0], with pairwise tests indicating ACE > No-feedback and ACE > Other-feedback; Other ≈ No [(Shea et al., 2024), §7, Table 5].

Subjective perceived improvement was highest in ACE (mean 4.34) versus alternatives (4.19, 4.17). Error identification modules reached ≥0.90 accuracy, though $O_1 \leq 0.9 T$1 ranged down to 0.54 for strategic closing [(Shea et al., 2024), Table 4].

6. Implementation, System Integration, and Deployment

The ACE backend consists of a Python/Flask orchestration layer interfacing with OpenAI API endpoints for GPT-4 and GPT-4o. Its architecture exposes modular REST endpoints for simulation agent, error detection, and feedback mechanisms, enabling integration with web-based negotiation simulators or learning management systems via JavaScript widgets or iframes [(Shea et al., 2024), §6, Appendix F].

The inference pipeline is structured as: preparatory phase → numeric/error checking → dialog loop (user input → price extraction/classification → error detection → feedback prompt → revision prompt → final feedback delivery).

Latency optimization and modularization are achieved through the separation of negotiation quality (GPT-4) and feedback responsiveness (GPT-4o).

7. Limitations and Prospective Developments

ACE exhibits several limitations:

• LLM Softness: Model tendency toward over-concession is only partly mitigated by the "subjective limit" heuristic.
• Session Isolation: Lack of persistent user modeling or longitudinal memory restricts adaptive feedback.
• Culture-Bound Normativity: The annotation and feedback scheme reflect American distributive-bargaining conventions, potentially reducing international or domain generalizability [(Shea et al., 2024), §8].

Proposed future directions include extension to multi-party and multi-issue negotiation scenarios, development of real-time scaffolding to guide users mid-dialogue, personalization via persistent user histories, and the adaptation of the feedback pipeline for diverse negotiation cultures and styles.

---

ACE represents, to date, the first large-scale synthesis of LLM-based interactive negotiation partnership and error-aware feedback consistent with established business-school teaching paradigms, empirically validated via large-scale randomized user studies and rigorous annotation methodology (Shea et al., 2024).