SariBench: Retail AI Benchmark
- SariBench is a retail-specific benchmark that evaluates embodied AI agents in a virtual store using VR-collected human demonstrations and structured task suites.
- It organizes tasks into Easy, Average, and Difficult levels, testing skills like navigation, object manipulation, OCR-based interaction, and decision making.
- The simulation environment leverages Unity to provide a realistic retail setting, enabling direct human-agent performance comparisons and highlighting current AI limitations.
SariBench is the benchmark and dataset component of Sari Sandbox, a virtual retail-store environment for embodied AI. It is designed to measure retail shopping behaviors rather than generic indoor navigation, combining task suites for embodied agents with human demonstrations collected in virtual reality under matched conditions. In the supplied literature, SariBench is presented as a retail-focused benchmark in which agents must navigate, inspect, manipulate, and sometimes reason about grocery products in a photorealistic store simulation, with baseline comparisons against human performance (Gajo et al., 1 Aug 2025).
1. Definition and research scope
SariBench is introduced as a benchmark of retail tasks performed in Sari Sandbox plus human demonstrations of those tasks using VR. Its purpose is twofold: to provide a task suite for embodied agents in a retail context, and to offer human baseline demonstrations collected in the same environment. The benchmark is motivated by a gap in embodied-AI evaluation: many existing benchmarks focus on homes, generic indoor spaces, or locomotion, whereas retail tasks require product search, product recognition, manipulation, barcode scanning, expiration-date reading, checkout, and reasoning about product attributes (Gajo et al., 1 Aug 2025).
A central feature of SariBench is that it is not just a navigation benchmark. The tasks are intended to test the retail loop of perception, search, manipulation, memory, and task completion in a store-like setting. The paper explicitly frames the released evaluation as a “solvability proof-of-concept” rather than a fully solved benchmark, with the easier tasks evaluated first and harder tasks reserved for future work. This positioning is significant because it distinguishes SariBench from benchmarks that primarily assess locomotion or object-fetching in domestic scenes (Gajo et al., 1 Aug 2025).
The benchmark is therefore best understood as a retail-task evaluation layer grounded in a dedicated simulation environment. The environment supports both virtual reality (VR) for human interaction and a vision LLM (VLM)-powered embodied agent, allowing direct human-agent comparison under a common scene model and API (Gajo et al., 1 Aug 2025).
2. Task design and dataset construction
SariBench organizes tasks into three difficulty levels: Easy, Average, and Difficult. The paper defines these levels by the skills required. Easy tasks require perception, navigation, manipulation; an example is “Find and pick up a box of cereal.” Average tasks require perception, navigation, manipulation, memory, task execution; an example is “Pick up a bottle of soda and scan at checkout.” Difficult tasks require perception, navigation, manipulation, memory, task execution, decision making, comprehension; an example is “Which of these two products has lower sugar content: strawberry-flavored biscuit or chocolate-flavored biscuit? Scan the answer.” (Gajo et al., 1 Aug 2025)
The human-demonstration dataset currently includes 100 videos of participants completing tasks, collected from 20 human participants with 108 tasks assigned in total across difficulty levels and randomized store layouts. In addition to video, the dataset includes time-synchronized environment data at 10 frames per second, specifically global head position and rotation, global hand position and rotation, grip state, hovered item, and held item. Participants also performed think-aloud protocols, and those verbalizations were later transcribed. This makes the dataset a combination of visual record, embodied state trace, and verbal reasoning trace rather than a simple collection of demonstrations (Gajo et al., 1 Aug 2025).
The collection protocol is structured. Participants first received a briefing on the study and the data being collected. They then had 15 minutes of VR familiarization in a playground or training space. After this familiarization period, each participant was assigned 3 easy tasks, 2 average tasks, and 2 difficult tasks drawn from a curated pool stratified by difficulty. This protocol is consequential for interpretation: the benchmark is based on interactive demonstrations in a controlled virtual store rather than passive observation or post hoc annotation (Gajo et al., 1 Aug 2025).
A common misconception is to treat SariBench as only a video dataset. The supplied description instead characterizes it as a benchmark derived from interactive human demonstrations with metadata that supports analysis of behavior, timing, and decision-making (Gajo et al., 1 Aug 2025).
3. Simulation environment and retail-world representation
SariBench is grounded in the Sari Sandbox environment, a virtual retail store built in Unity using the Universal Render Pipeline (URP) and PhysX. The environment contains three distinct store layouts, created after surveying real small-scale retail stores. These layouts vary in size and arrangement and include shelves, a self-checkout counter, product groupings with shelf labels and price tags, and refrigeration-style cabinets with hinge and sliding doors. The layouts are described as being based on real local convenience-store configurations, which gives the benchmark a specifically retail-realistic rather than generic indoor character (Gajo et al., 1 Aug 2025).
A major component of the environment is its 250 interactive grocery products, modeled after real convenience-store packaged goods and grouped into 11 categories: Water: 12, Soda: 23, Juice: 16, Dairy: 20, Biscuit: 50, Can: 59, Chips: 40, Nuts: 15, Soup: 6, Noodles: 7, and Liquor: 2. The paper also classifies product meshes into simple, complex, and deformable meshes, although all products are simulated as rigid bodies for physics. Each product is annotated with ground-truth metadata including category, name, price, net weight, ingredients, nutritional facts, allergens, and manufacturing origin (Gajo et al., 1 Aug 2025).
The environment also includes retail-specific visual features such as barcodes, expiration-date decals, price tags, and product labels with randomized placements. The paper emphasizes that barcodes and expiration dates are generated or activated dynamically, and that the labels are readable enough for OCR-based interaction. This is an important design choice because difficult tasks depend on reading and comparing product information, not merely detecting object instances (Gajo et al., 1 Aug 2025).
Performance optimization is treated as part of the benchmark substrate. To improve runtime efficiency, the authors used box colliders only, JPG textures to reduce storage cost, Level of Detail (LOD), frustum culling, and occlusion culling. The reported runtime performance over 300 frames using Unity Profiler was 26.73 FPS for Layout 1, 23.23 FPS for Layout 2, and 35.14 FPS for Layout 3. Texture fidelity was evaluated with PaddleOCR, yielding Precision: 0.986, Recall: 0.943, and CER: 0.014 on product-label text. The paper notes that OCR still struggles with rotated text and stylized fonts (Gajo et al., 1 Aug 2025).
4. Interaction model, API, and human benchmarking protocol
Sari Sandbox is designed to be controlled through a Python API that communicates with a Unity/C# server over WebSocket. The API exposes functions for moving the agent or camera, moving the hands, gripping and poking, capturing screenshots, and resetting the environment. The listed core functions are TransformAgent(T, R), TransformHands(leftT, leftR, rightT, rightR), ToggleLeftGrip(), ToggleRightGrip(), ToggleLeftPoke(), ToggleRightPoke(), RequestScreenshot(), and Reset(). The API also provides state information such as avatar position, rotation, hand grip state, store layout, and randomization seed (Gajo et al., 1 Aug 2025).
For human data collection, the environment supports VR headsets, and the reported setup used a Meta Quest 2. The avatar consists of a camera, two hands, and character-controller-based movement. To improve comfort and reduce cybersickness, the environment includes teleportation, a tunneling vignette, haptic feedback on hover/interactables, and hand-based grabbing, touching, placing, and throwing. If no VR headset is detected, the camera is positioned at about 1.6 m eye height, with hands offset downward to mimic a standing human posture (Gajo et al., 1 Aug 2025).
The human evaluation protocol measures completion time and completion rate. The corresponding reported benchmark metrics are HAT for Human Average Time and HCR for Human Completion Rate. The paper states that lower time is better and higher completion rate is better. These metrics are mirrored on the agent side by AAT for Agent Average Time and ACR for Agent Completion Rate (Gajo et al., 1 Aug 2025).
This protocol is methodologically notable because it supports matched-condition evaluation. Humans and agents act in the same simulated store, with the same object inventory and the same operational task definitions. A plausible implication is that SariBench is intended to measure not only absolute task solvability but also the performance gap between contemporary embodied agents and human participants under retail-specific constraints.
5. Baseline embodied agent and benchmark results
The baseline embodied agent in the paper is a VLM-powered agent built with Gemini 2.5 Pro, using a ReAct-like reasoning-and-action loop plus memory. It is explicitly described as simple and not fine-tuned, serving primarily as a proof of concept for the API and environment rather than as a highly optimized policy. Its behavior is decomposed into navigation actions such as move_forward, pan_left, and pan_right; manipulation actions such as center_object_on_screen and retrieve_item; and tools including loc_object, ocr_object, and est_depth (Gajo et al., 1 Aug 2025).
Agent evaluation is limited to the easy tasks. The paper states that harder-task evaluation is future work. Agent success was assessed by manual visual inspection, with success defined as task completion within 45 minutes; failure included exceeding that limit or entering mode collapse by getting lost. Human and agent performance on the easy tasks is summarized below (Gajo et al., 1 Aug 2025).
| Task | Human (HAT / HCR) | Agent (AAT / ACR) |
|---|---|---|
| Easy-L1 | 47 s / 88.88% | 780 s / 68.63% |
| Easy-L2 | 73 s / 100.00% | 660 s / 45.10% |
| Easy-L3 | 61 s / 93.33% | 420 s / 33.33% |
For the more difficult tiers, the paper reports human-only results: Average-L1: 158 s, 87.50%; Average-L2: 106 s, 100.00%; Average-L3: 84 s, 100.00%; Difficult-L1: 76 s, 100.00%; Difficult-L2: 136 s, 100.00%; Difficult-L3: 113 s, 100.00% (Gajo et al., 1 Aug 2025).
The main empirical finding is that humans substantially outperform the embodied agent in both speed and reliability, even on easy tasks. The paper states that agent completion times were up to 16× longer than humans, and agent success rates were under 70%, while humans were often near 100%. The reported failure factors for the agent include slow VLM inference, heavy text-generation overhead, limited reasoning/planning quality, lack of robust navigation and manipulation planning, and susceptibility to mode collapse. Humans, by contrast, quickly reasoned about likely object locations, adapted to visual cues, and completed tasks more reliably (Gajo et al., 1 Aug 2025).
A second misconception addressed by the results is that a functioning retail simulation implies strong embodied-agent performance. The benchmark instead shows that a usable environment and a solvable task format do not by themselves close the gap between contemporary VLM-based control loops and human retail behavior.
6. Limitations, interpretation, and future development
The paper is explicit that both Sari Sandbox and SariBench remain incomplete. One limitation is runtime performance: the environment does not consistently reach 60 FPS on the tested desktop, and layout complexity affects runtime. A second limitation is evaluation coverage: the agent was tested only on easy tasks, so the benchmark does not yet provide agent baselines for the full average and difficult spectrum. A third limitation concerns OCR and perception, where rotated, vertical, and stylized text remain difficult. A fourth limitation is realism: products are interactive, but all are currently rigid bodies, and mesh colliders are not yet used extensively. A fifth limitation is human-study ergonomics, since some participants reported motion sickness (Gajo et al., 1 Aug 2025).
The paper also notes nuances in human performance. Humans were not perfect: some completion-rate dips were attributed to perseverance, occasional carelessness, unfamiliarity with the scanner, and motion sickness for some participants. Even so, human performance remained well above the agent baseline (Gajo et al., 1 Aug 2025).
Future directions are stated concretely. The authors highlight the need to improve runtime efficiency, automated annotation generation, dynamic dataset expansion, a user-friendly scene-creation tool, and updated store designs. They recommend optimizing to a stable 60 FPS, expanding SariBench with more tasks and dynamic additions, improving realism via deformable objects and mesh colliders, evaluating agents across all task difficulties, improving VLM context engineering for better reasoning and planning, automating dataset annotation, developing a scene-creation tool for rapid prototyping, and further mitigating VR discomfort (Gajo et al., 1 Aug 2025).
Taken together, these details place SariBench as a retail-specific embodied-AI benchmark whose current value lies in its domain specificity, its integration of human VR demonstrations with agent evaluation, and its exposure of a substantial gap between proof-of-solvability and robust embodied competence. The benchmark’s role is less that of a closed leaderboard and more that of an empirical substrate for future work on perception, manipulation, OCR-mediated interaction, memory, and decision making in retail environments (Gajo et al., 1 Aug 2025).