AgenticPay: Benchmark for Language-Based Negotiation

• AgenticPay is a benchmark and simulation framework for multi-agent natural language negotiation that evaluates strategic convergence and balanced economic surplus.
• It models buyer–seller markets with diverse tasks and realistic business scenarios, integrating private information and structured dialogue into economic transactions.
• Empirical results show that effective negotiation depends on both fluent language generation and strategic economic behavior, revealing biases and convergence challenges in LLMs.

Searching arXiv for AgenticPay and closely related agentic payment papers to ground the article in current literature. I’m going to use the arXiv search tool to confirm the primary AgenticPay paper and a few adjacent works that define its research context. AgenticPay is a benchmark and simulation framework for multi-agent buyer–seller negotiation driven by natural language, introduced to evaluate LLM agents as economic actors operating under private constraints, product-dependent valuations, and multi-round linguistic bargaining rather than numeric bidding alone (Liu et al., 5 Feb 2026). It grounds dialogue into structured negotiation actions, measures feasibility, efficiency, and welfare, and shows that fluent language generation is not equivalent to competent economic behavior. Subsequent literature uses AgenticPay as an adapted environment for execution-boundary governance (Shi et al., 3 Jun 2026). This suggests that the term now denotes both a specific benchmark and a broader research locus for agentic commerce, where negotiation, authorization, compliance, clearing, and settlement must be studied as a connected stack.

1. Origin, scope, and research role

AgenticPay was introduced to fill a gap in prior evaluation settings for LLM agents. Earlier benchmarks, as described in the original paper, tended to emphasize single-agent reasoning, tool use, preference following, auction-style bidding, or simplified bilateral bargaining, while omitting the properties that dominate real transactions: private reservation values, heterogeneous products, multi-round negotiation, competition among multiple buyers and sellers, and agreement through language rather than scalar bids alone (Liu et al., 5 Feb 2026). AgenticPay therefore treats negotiation as an economic interaction problem rather than as a pure dialogue task.

The framework contains 111 negotiation tasks, spanning 31 basic tasks and 80 realistic tasks across 8 multi-agent configurations and 10 realistic business scenarios. Those scenarios cover Daily Life, Professional Services, Business Procurement, and Financial Assets, with price ranges from \$350 to \$120k (Liu et al., 5 Feb 2026). This scale matters because the benchmark is not a single bargaining toy environment; it is a task family that varies buyer multiplicity, seller multiplicity, product multiplicity, interaction mode, and domain semantics.

A recurrent misconception is that language-mediated commerce is merely an application layer over ordinary dialogue evaluation. AgenticPay rejects that premise. Its core unit is an economically consequential exchange in which agents must preserve private information, negotiate strategically, converge within a horizon, and satisfy feasibility constraints. In that sense, AgenticPay functions less like a conversational benchmark and more like a structured market simulator with natural language as the strategic medium.

2. Market model and negotiation protocol

The formal setting is a buyer–seller market with buyers $\mathcal{B}=\{1,\ldots,N_B\}$ and sellers $\mathcal{S}=\{1,\ldots,N_S\}$. Each buyer $i$ has private state $b_i$, each seller $j$ has private state $\sigma_j$, each offered product is represented by a public feature vector $v_j \in \mathcal{V}$, and all agents observe a shared market context $x \in \mathcal{X}$ (Liu et al., 5 Feb 2026). In the concrete instantiation emphasized by the paper, buyers hold a confidential maximum willingness-to-pay $p^{\max}$, while sellers hold a confidential minimum acceptable price $\mathcal{B}=\{1,\ldots,N_B\}$0. These reservation values are injected into system prompts and explicitly withheld from shared dialogue.

Negotiation is modeled as a finite-horizon language game. At round $\mathcal{B}=\{1,\ldots,N_B\}$1, buyer and seller messages are sampled from role-specific policies: $\mathcal{B}=\{1,\ldots,N_B\}$2 where $\mathcal{B}=\{1,\ldots,N_B\}$3 is public dialogue history (Liu et al., 5 Feb 2026). The benchmark then grounds these free-form messages through a parser $\mathcal{B}=\{1,\ldots,N_B\}$4: $\mathcal{B}=\{1,\ldots,N_B\}$5 extracting structured actions, especially transaction-price proposals $\mathcal{B}=\{1,\ldots,N_B\}$6.

The protocol is deliberately hybrid. Every turn must contain exactly one machine-readable price offer, using ### BUYER_PRICE($\mathcal{B}=\{1,\ldots,N_B\}$7X) ### for the seller, and explicit commitment is signaled with MAKE_DEAL (Liu et al., 5 Feb 2026). This design preserves natural-language expressiveness while making the outcome executable and measurable. A deal is reached when both parties propose the same price, and a valid transaction additionally requires that the price lie within the bargaining zone: $\mathcal{B}=\{1,\ldots,N_B\}$8

The maximum negotiation horizon in the reported experiments is $\mathcal{B}=\{1,\ldots,N_B\}$9 rounds (Liu et al., 5 Feb 2026). This horizon makes strategic convergence, rather than mere proposal generation, a first-class capability. Many failures in AgenticPay are not semantic breakdowns but convergence failures under limited time, which is one reason the benchmark is informative about long-horizon strategic reasoning.

3. Task suite, scoring, and benchmark mechanics

AgenticPay scales along three principal complexity axes: the number of buyers, the number of sellers, and the size of the product set. The eight task categories are 1B-1P-1S, MB-1P-1S, 1B-MP-1S, 1B-1P-MS, MB-MP-1S, MB-1P-MS, 1B-MP-MS, and MB-MP-MS (Liu et al., 5 Feb 2026). Interaction can be sequential or parallel, allowing the framework to probe opportunity-cost reasoning and multi-threaded negotiation as well as isolated bilateral bargaining.

Evaluation is explicitly economic. For a feasible deal at price $\mathcal{S}=\{1,\ldots,N_S\}$0, buyer and seller surplus are normalized within the bargaining zone: $\mathcal{S}=\{1,\ldots,N_S\}$1 Balanced deal quality is then defined by

$\mathcal{S}=\{1,\ldots,N_S\}$2

which is maximal when surplus is split evenly (Liu et al., 5 Feb 2026). This choice is important: AgenticPay does not collapse welfare into simple deal completion or raw gains from trade. It treats balanced division of surplus as a benchmark notion of globally good bargaining.

The benchmark reports three primary scores. If a feasible deal is reached at round $\mathcal{S}=\{1,\ldots,N_S\}$3, with discount factor $\mathcal{S}=\{1,\ldots,N_S\}$4, deal-success reward $\mathcal{S}=\{1,\ldots,N_S\}$5, deal-quality reward $\mathcal{S}=\{1,\ldots,N_S\}$6, and round-efficiency reward $\mathcal{S}=\{1,\ldots,N_S\}$7, then

$\mathcal{S}=\{1,\ldots,N_S\}$8

$\mathcal{S}=\{1,\ldots,N_S\}$9

$i$0

$i$1

If a deal fails, scores are penalized by

$i$2

with $i$3 in the failure case (Liu et al., 5 Feb 2026). The reported experimental configuration sets $i$4, $i$5, $i$6, $i$7, $i$8, and $i$9. Because $b_i$0, deal quality is weighted more heavily than mere agreement, and because $b_i$1, late convergence is penalized.

Auxiliary metrics include Deal Rate, Timeout Rate, Overflow Rate, and Average Rounds (Liu et al., 5 Feb 2026). These reveal a second recurring misconception: a persuasive conversation is not necessarily a good transaction. AgenticPay therefore separates linguistic plausibility from economic admissibility, strategic success, and temporal efficiency.

4. Empirical findings and characteristic failure modes

Across all 111 tasks, the original benchmark reports GlobalScore values of 86.9 for Claude Opus 4.5, 82.2 for Gemini-3-Flash, 81.7 for GPT-5.2, 63.9 for Qwen3-14B, and 32.5 for Llama-3.1-8B (Liu et al., 5 Feb 2026). Deal rates are 100.0% for Claude Opus 4.5, Gemini-3-Flash, and GPT-5.2, but fall to 79.3% for Qwen3-14B and 51.4% for Llama-3.1-8B. Timeout rates are 0.0% for the three proprietary models, 20.7% for Qwen3-14B, and 48.6% for Llama-3.1-8B; overflow rates are 0.0%, 2.7%, 0.0%, 1.8%, and 10.8%, respectively (Liu et al., 5 Feb 2026).

These results support the paper’s central claim that negotiation competence depends on more than local language quality. Average rounds to termination are 3.7 for Claude Opus 4.5, 4.8 for Gemini-3-Flash, 3.8 for GPT-5.2, 7.8 for Qwen3-14B, and 15.0 for Llama-3.1-8B (Liu et al., 5 Feb 2026). The weaker models do not merely produce worse prices; they stall, drift, or fail to close.

One of the benchmark’s most stable empirical regularities is buyer disadvantage. SellerScore exceeds BuyerScore across models: for GPT-5.2, SellerScore is 81.1 versus BuyerScore 58.5; for Claude Opus 4.5, 76.1 versus 63.5; for Qwen3-14B, 58.9 versus 47.6 (Liu et al., 5 Feb 2026). Cross-play analysis sharpens the asymmetry. This does not prove a universal law of LLM bargaining, but it identifies a robust bias in the tested systems.

The benchmark also shows that greater market multiplicity can improve outcomes. GlobalScore often rises from 1B1S toward MBMS, which the authors attribute to greater market liquidity and a larger set of compatible matches (Liu et al., 5 Feb 2026). Conversely, Financial Assets are consistently among the hardest domains, and scenario-level breakdowns show particularly weak results on Business Acquisition and other valuation-intensive settings.

A particularly informative failure mode is the “last-mile” miss. Among failed episodes, 43.5% of Qwen3-14B failures are within 5 units and 52.2% within 10 units of agreement; the corresponding figures for Llama-3.1-8B are 46.3% and 55.6% (Liu et al., 5 Feb 2026). AgenticPay therefore reveals that many failures are not gross misunderstandings of the bargaining zone, but inability to execute the final convergence step under long-horizon strategic pressure.

5. From negotiation benchmark to payment execution architectures

The original AgenticPay paper stops at negotiated agreement. Separate work addresses the next layer: how an autonomous system could carry a negotiated or delegated economic action through payment execution, compliance enforcement, and request-level monetization. This suggests a broader AgenticPay stack in which negotiation is upstream of payment orchestration rather than identical to it.

One explicit extension is the Hierarchical Multi-Agent System for Payments (HMASP), which proposes an end-to-end payment architecture with four levels: a Conversational Payment Agent, Supervisor agents, Routing agents, and a Process Summary Agent (Chua et al., 27 Feb 2026). HMASP uses LangGraph, shared state variables, decoupled message states, and structured handoff protocols to coordinate workflows such as card registration, card retrieval, and payment processing. In simulation, GPT-4.1 achieves 99.6 on Task 1 and 99.6 on Task 2, while the strongest open-weight model, Qwen2.5:32b, reports task-success values of 96.4%, 98.8%, and 95.6% on the three main task families (Chua et al., 27 Feb 2026). The system is explicitly a feasibility study, not a production payment rail, but it clarifies how role-separated agent hierarchies can operationalize payment tasks beyond bargaining.

A second line of work embeds compliance directly into the execution path. The architecture in "Compliance-Aware Agentic Payments on Stablecoin Rails" combines x402-style, signature-based payment authorization with programmable compliance enforced by a PolicyWrapper and related guardrails, so settlement occurs only if applicable policies pass at execution time (See et al., 30 Apr 2026). The demonstrated outcomes are trivalent,

$b_i$2

with transaction-linked on-chain attestations and structured handling of incomplete requirements through tranching and escrow. The paper’s strongest engineering claim is that regulated agentic payments fail if compliance is treated as a separate off-chain workflow rather than as an execution-time gate (See et al., 30 Apr 2026).

A third strand focuses on request-level paid access rather than checkout. APEX adapts HTTP 402-style payment gating to UPI-like fiat workflows and implements a challenge-settle-consume lifecycle with HMAC-signed short-lived tokens, idempotent settlement handling, and policy-aware payment approval (Uddin et al., 2 Apr 2026). Its reported results show that policy enforcement reduces total spending by 27.3% while maintaining 52.8% success rate for legitimate requests, and that replay attacks and invalid tokens are blocked at 100% with low latency overhead reported as 19.6 ms average (Uddin et al., 2 Apr 2026). For AgenticPay, this is evidence that machine-native monetization can be expressed as protocol behavior rather than deferred billing.

6. Governance, security, and clearing lessons from adjacent work

As soon as AgenticPay-style interactions become executable, governance moves to the execution boundary. The Organizational Control Layer (OCL) paper studies this problem directly on adversarial buyer–seller negotiation environments adapted from AgenticPay (Shi et al., 3 Jun 2026). OCL intercepts generated actions before execution and assigns one of four control outcomes—Approve, Revise, Block, or Escalate—without modifying the underlying LLM. In the reported 50-episode adversarial benchmark, unsafe executions drop from 88% to 0%, while valid success rises from 12% to 96% (Shi et al., 3 Jun 2026). The same paper also documents a safety–utility tradeoff in tightly constrained markets, making governance a tuning problem rather than a one-sided win.

A second misconception addressed by the literature is that cryptographic payment authorization is sufficient security. Red-teaming of Google’s Agent Payments Protocol shows that upstream LLM manipulation can compromise transaction integrity before any mandate is signed: the Branded Whisper Attack manipulates product ranking via merchant-supplied descriptions, while the Vault Whisper Attack induces cross-user data leakage during credentials retrieval (Debi et al., 30 Jan 2026). The core lesson is that secure payment rails do not secure the decision rail.

Runtime misuse is a further layer. Zero-Trust Runtime Verification for AP2-style systems argues that signed mandates remain vulnerable to same-context replay and cross-context reuse unless execution-time verifiers enforce both context binding and consume-once semantics (Lan et al., 6 Feb 2026). The proposed verifier blocks all evaluated replay and context-redirect attacks, while maintaining verification latency of approximately 3.8 ms at throughput levels up to 10,000 transactions per second (Lan et al., 6 Feb 2026). For AgenticPay, this means mandate validity at issuance cannot be treated as equivalent to safe execution under retries, concurrency, and orchestration.

At the highest layer, RAILS introduces clearing as a separate primitive for agentic commerce (Valois-Franklin et al., 7 Jun 2026). Its central soundness property is

$b_i$3

meaning that no Settlement Instruction is emitted unless the supporting evidence basis meets the obligation’s admissibility floor. In other words, payment is not clearing, authorization is not clearing, and escrow is not clearing; each of those systems presumes a determination that RAILS tries to produce (Valois-Franklin et al., 7 Jun 2026). For AgenticPay, this is a strong proposal for how negotiated and authorized actions could be translated into financially material settlement only after admissible verification.

7. Limitations and open directions

AgenticPay remains a controlled benchmark rather than a complete commerce protocol. The original paper simplifies real exchange into scalar reservation thresholds, structured price tags, and a fixed MAKE_DEAL commitment marker; each task instance is executed once per model; and no strong non-LLM bargaining baseline is reported (Liu et al., 5 Feb 2026). These are deliberate design choices for benchmark tractability, but they limit direct inference to real payment environments with legal terms, delivery contingencies, taxes, reputation, and compliance obligations.

The broader systems literature remains similarly incomplete. The compliance-aware stablecoin architecture explicitly does not provide typed-data schemas, ABI definitions, or formal security proofs (See et al., 30 Apr 2026). HMASP is simulation-only and does not deliver live card-network or issuer integration (Chua et al., 27 Feb 2026). APEX is single-node, SQLite-backed, and uses a simulated UPI-like layer rather than full banking integration (Uddin et al., 2 Apr 2026). Even the stronger security proposals leave important questions open: RAILS does not solve verifier correctness or legal enforceability of chosen admissibility floors (Valois-Franklin et al., 7 Jun 2026).

Several research directions emerge from adjacent work. The SoK on blockchain A2A payments argues for stronger cross-stage consistency, behavior-aware control, and compositional payment workflows across discovery, authorization, execution, and accounting (Zhang et al., 4 Apr 2026). "Paying to Know" reframes agentic e-commerce as a micro-transaction market for verified information, suggesting that future AgenticPay systems may need to meter evidence acquisition rather than only terminal checkout (Ventirozos et al., 23 Jun 2026). "The Agentic Economy" argues that micro-transactions, unscripted agent interaction, and the choice between open interoperation and walled gardens will shape the surrounding market structure (Rothschild et al., 21 May 2025). PACT adds a different but related open problem: how to price agentic services as differentiated QoS contracts under information asymmetry and liability exposure (Yang et al., 27 May 2025).

Taken together, these works indicate that AgenticPay is best understood not as a single protocol, but as a research program. In its narrow sense, it is a benchmark for natural-language economic negotiation. In a broader and still evolving sense, it names the problem of building autonomous commerce systems in which negotiation, payment, compliance, verification, and final settlement remain aligned under private information, bounded delegation, and auditable evidence.