- The paper introduces MACS, a hybrid architecture that separates an LLM shopping agent from a deterministic merchant agent, achieving an 87.1% single-turn pass rate and 72% multi-turn Pass@5 compared with 72.1% and 56% for the strongest baseline.
- The system enforces catalog grounding, hard filters, brand exclusions, and cross-turn preferences through SQL predicates, typed session slots, knowledge-graph retrieval, and progressive relaxation rather than relying solely on prompts.
- Ablations show that persistent session state and SQL enforcement are essential: removing session tracking reduced multi-turn Pass@5 from 72% to 52%, while removing SQL constraints lowered brand compliance from 1.000 to 0.684.
MACS (Multi-Agent Commerce System) addresses a specific reliability problem in LLM-mediated conversational recommendation: deployments where recommendations must be drawn exclusively from a merchant's fixed catalog, without web search or unsupported product claims. The paper's central argument is that prompt-level constraint enforcement is structurally incapable of guaranteeing correctness, and that hard constraints must instead be enforced deterministically at the architectural level. The proposed system separates an LLM-based shopping agent from a deterministic merchant agent, and the evaluation supports this claim with both benchmark comparisons and controlled ablations (2608.14068).
Problem setting and architecture
The paper formalizes limited-catalog conversational recommendation as constrained multi-turn retrieval over a fixed catalog C, subject to three reliability requirements: catalog grounding (every returned product exists at a verified price), constraint correctness (hard requirements enforced deterministically rather than inferred), and cross-turn persistence (constraints remain active until explicitly revised). The system design maps these requirements onto two cooperating agents. The shopping agent handles query rewriting, intent routing, structured slot extraction, and response generation; the merchant agent executes SQL constraint filtering, brand exclusion, progressive relaxation, and knowledge-graph traversal over the relational store. Critically, the shopping agent cannot retrieve products directly — all access flows through the merchant agent's structured API via commerce protocols (MCP, UCP, or ACP), so catalog grounding is enforced architecturally rather than through prompting.
The session state is a typed slot dictionary supporting cross-turn accumulation, ordinal reference resolution ("add the second one"), preference pivots (budget overwrites, exclusion reversals such as "actually HP is fine"), and underspecification handling that triggers targeted clarifying questions when no hard constraint is extractable. Brand exclusions are enforced twice — at the SQL level and in a post-retrieval title filter — because some catalog entries store condition metadata in the brand field.
The data layer combines a relational store (21,000+ items; 1,490+ laptops with 12 structured specification fields), a knowledge graph (2,400 nodes, 8,500 edges) queried via Cypher for substitutes and compatibility in roughly 17 ms, and a TTL cache yielding approximately 12× latency reduction on hits (36 ms vs. 446 ms cold). Progressive relaxation drops least-critical optional specifications when fewer than three results satisfy all constraints, never relaxing price ceilings or brand exclusions, and always disclosing which constraints were loosened. A deterministic best-value scoring function weights price, rating, review volume, and use-case-specific specification tiers per detected use case.
Evaluation methodology
The evaluation deliberately separates deterministic constraint correctness from judged response quality. Hard-constraint compliance is verified by executing expected filters as SQL predicates against the product store, making violations objective and LLM-independent. Narrative quality uses G-Eval with GPT-4o-mini at temperature 0, restricted to quality assessment to avoid circularity with constraint checking. The single-turn composite score weights type match, brand compliance, filter compliance, stock availability, explainability sub-checks, and quality, with inapplicable component weights redistributed to quality. Multi-turn scoring combines deterministic constraint checks (0.55) with judge scores (0.45); notably, deterministic check results are injected into the judge prompt so the judge does not penalize responses for constraints enforced during retrieval but not restated verbatim. The pass threshold of 0.65 was raised specifically to eliminate floor effects under this weighting.
Baselines are catalog-bound prompt-only systems (GPT-4o-mini and Gemini 2.5 Flash Lite) receiving MACS's own top-K candidate products injected into the system prompt, with identical token budgets, no web search, and identical scoring. This is a deliberate scoping decision: the comparison isolates constraint-aware generation over a shared candidate pool, not end-to-end retrieval quality. The authors state this explicitly, and it bounds what the results can support.
Results
On the 140-query single-turn benchmark, MACS achieved an 87.1% pass rate versus 72.1% (GPT-based baseline) and 68.6% (Gemini-based baseline), with perfect brand compliance (1.000) and near-perfect filter compliance (0.970). Judge-assessed narrative quality was comparable across systems (0.394 vs. 0.427 / 0.392), indicating that MACS's advantage is reliability-oriented rather than fluency-oriented. On the expanded 225-query benchmark, MACS scored 85.3% overall, with the multi-constraint group (17 queries) reaching 94.1%, confirming deterministic enforcement of four simultaneous constraints. Weak groups were context-free comparison (0%), orchestrator routing (33.3%), and preference discovery (25%).
Multi-turn results are the strongest evidence for the architecture. Across 10 scripted scenarios at K=5 runs, MACS led macro Pass@5 at 72% versus 56% and 52%, with zero constraint drift. The advantage concentrates precisely where session-persistent state matters: exclusion reversal (100% vs. 20% / 0%) and constraint accumulation (100% vs. 60% / 40%). Mean judged quality was statistically indistinguishable across systems (0.751 vs. 0.736), so the Pass@5 gap reflects scenario-level consistency rather than mean-score differences. On disclosure of catalog-impossible queries (e.g., "RTX 4090 laptop, $150 budget"), MACS scored 0.925 versus 0.588 and 0.550 — the authors identify this +0.338/+0.375 gap as the clearest architectural differentiator, since baselines receive already-relaxed candidate lists with no mechanism to detect or disclose the constraint gap.
Two ablations isolate causal contributions. Removing the persistent slot dictionary (MACS-NoSession) dropped multi-turn macro Pass@5 from 72% to 52%, with constraint-accumulation and budget-overwrite scenarios collapsing from 100% to 0%, while within-turn-driven scenarios were unaffected. Removing SQL WHERE clauses (MACS-NoSQL) dropped brand compliance on brand-constrained queries from 1.000 to 0.684, confirming that SQL predicates, not prompt instructions, enforce brand compliance. A metric-sensitivity analysis shows macro Pass@5 ranging from 42% (judge-heavy weighting) to 96% (constraint-heavy), which the authors acknowledge transparently; they argue the headline findings are independently supported by deterministic metrics and ablations, since the S1/S8 advantages are driven entirely by the deterministic constraint term.
Limitations
The paper concedes several significant limitations. All evaluations use a single consumer-electronics domain (laptops and accessories); generalization to other catalogs is not claimed. The baselines are generation-over-shared-candidates systems, not full end-to-end retrieval agents, so the comparison excludes retrieval-quality differences. Benchmarks were authored by the research team, and the multi-turn evaluation comprises only 10 scripted scenarios (50 run-scenario data points), raising possible alignment between benchmark design and system structure — visible in the anomalous result where MACS's brand-exclusion Pass@5 (20%) fell below both baselines (40% each), attributed to stochastic variance at K=5. Response quality relies entirely on LLM-as-judge scoring with no human validation, and the judge introduces ~0.02–0.03 score variance between identical temperature-0 runs. Intent pivot remains unsolved across all systems (0% Pass@5), which the authors characterize as a structural failure mode rather than a system-specific weakness. Open questions include performance on standardized external benchmarks such as τ-Bench's retail subset, human validation of G-Eval scores, and comparison against tool-augmented agents with independent retrieval.
Conclusion
MACS demonstrates that architectural separation between language-facing LLM components and deterministic constraint enforcement yields measurable reliability gains in fixed-catalog conversational recommendation, without degrading judged response quality. The evidence is strongest where it matters most for the claimed contribution: ablations show the persistent slot dictionary and SQL predicates are each individually necessary for cross-turn constraint tracking and brand compliance respectively. The claims are appropriately scoped to a single product domain and a candidate-injection baseline design, leaving external-benchmark validation and cross-domain generalization as the principal unresolved questions.