---
title: 'SariBench: Retail AI Benchmark'
url: https://www.emergentmind.com/topics/saribench
type: topic
---

# SariBench: Retail AI Benchmark

SariBench is the benchmark and dataset component of **Sari Sandbox**, a virtual retail-store environment for embodied AI. It is designed to measure retail shopping behaviors rather than generic indoor navigation, combining task suites for embodied agents with human demonstrations collected in virtual reality under matched conditions. In the supplied literature, SariBench is presented as a retail-focused benchmark in which agents must navigate, inspect, manipulate, and sometimes reason about grocery products in a photorealistic store simulation, with baseline comparisons against human performance [2508.00400].

## 1. Definition and research scope

SariBench is introduced as a **benchmark of retail tasks performed in Sari Sandbox plus human demonstrations of those tasks using VR**. Its purpose is twofold: to provide a task suite for embodied agents in a retail context, and to offer human baseline demonstrations collected in the same environment. The benchmark is motivated by a gap in embodied-AI evaluation: many existing benchmarks focus on homes, generic indoor spaces, or locomotion, whereas retail tasks require product search, product recognition, manipulation, barcode scanning, expiration-date reading, checkout, and reasoning about product attributes [2508.00400].

A central feature of SariBench is that it is **not just a navigation benchmark**. The tasks are intended to test the retail loop of perception, search, manipulation, memory, and task completion in a store-like setting. The paper explicitly frames the released evaluation as a **“solvability proof-of-concept”** rather than a fully solved benchmark, with the easier tasks evaluated first and harder tasks reserved for future work. This positioning is significant because it distinguishes SariBench from benchmarks that primarily assess locomotion or object-fetching in domestic scenes [2508.00400].

The benchmark is therefore best understood as a retail-task evaluation layer grounded in a dedicated simulation environment. The environment supports both **virtual reality (VR) for human interaction** and a **vision language model (VLM)-powered embodied agent**, allowing direct human-agent comparison under a common scene model and API [2508.00400].

## 2. Task design and dataset construction

SariBench organizes tasks into **three difficulty levels**: **Easy**, **Average**, and **Difficult**. The paper defines these levels by the skills required. **Easy** tasks require **perception, navigation, manipulation**; an example is *“Find and pick up a box of cereal.”* **Average** tasks require **perception, navigation, manipulation, memory, task execution**; an example is *“Pick up a bottle of soda and scan at checkout.”* **Difficult** tasks require **perception, navigation, manipulation, memory, task execution, decision making, comprehension**; an example is *“Which of these two products has lower sugar content: strawberry-flavored biscuit or chocolate-flavored biscuit? Scan the answer.”* [2508.00400]

The human-demonstration dataset currently includes **100 videos** of participants completing tasks, collected from **20 human participants** with **108 tasks assigned in total** across difficulty levels and randomized store layouts. In addition to video, the dataset includes time-synchronized environment data at **10 frames per second**, specifically **global head position and rotation**, **global hand position and rotation**, **grip state**, **hovered item**, and **held item**. Participants also performed **think-aloud protocols**, and those verbalizations were later transcribed. This makes the dataset a combination of visual record, embodied state trace, and verbal reasoning trace rather than a simple collection of demonstrations [2508.00400].

The collection protocol is structured. Participants first received a briefing on the study and the data being collected. They then had **15 minutes of VR familiarization** in a playground or training space. After this familiarization period, each participant was assigned **3 easy tasks**, **2 average tasks**, and **2 difficult tasks** drawn from a curated pool stratified by difficulty. This protocol is consequential for interpretation: the benchmark is based on interactive demonstrations in a controlled virtual store rather than passive observation or post hoc annotation [2508.00400].

A common misconception is to treat SariBench as only a video dataset. The supplied description instead characterizes it as a benchmark derived from **interactive human demonstrations** with metadata that supports analysis of behavior, timing, and decision-making [2508.00400].

## 3. Simulation environment and retail-world representation

SariBench is grounded in the **Sari Sandbox** environment, a virtual retail store built in **Unity** using the **Universal Render Pipeline (URP)** and **PhysX**. The environment contains **three distinct store layouts**, created after surveying real small-scale retail stores. These layouts vary in size and arrangement and include **shelves**, a **self-checkout counter**, **product groupings with shelf labels and price tags**, and **refrigeration-style cabinets with hinge and sliding doors**. The layouts are described as being based on **real local convenience-store configurations**, which gives the benchmark a specifically retail-realistic rather than generic indoor character [2508.00400].

A major component of the environment is its **250 interactive grocery products**, modeled after real convenience-store packaged goods and grouped into **11 categories**: **Water: 12**, **Soda: 23**, **Juice: 16**, **Dairy: 20**, **Biscuit: 50**, **Can: 59**, **Chips: 40**, **Nuts: 15**, **Soup: 6**, **Noodles: 7**, and **Liquor: 2**. The paper also classifies product meshes into **simple**, **complex**, and **deformable** meshes, although all products are simulated as rigid bodies for physics. Each product is annotated with ground-truth metadata including **category**, **name**, **price**, **net weight**, **ingredients**, **nutritional facts**, **allergens**, and **manufacturing origin** [2508.00400].

The environment also includes retail-specific visual features such as **barcodes**, **expiration-date decals**, **price tags**, and product labels with randomized placements. The paper emphasizes that barcodes and expiration dates are generated or activated dynamically, and that the labels are readable enough for OCR-based interaction. This is an important design choice because difficult tasks depend on reading and comparing product information, not merely detecting object instances [2508.00400].

Performance optimization is treated as part of the benchmark substrate. To improve runtime efficiency, the authors used **box colliders only**, **JPG textures to reduce storage cost**, **Level of Detail (LOD)**, **frustum culling**, and **occlusion culling**. The reported runtime performance over 300 frames using Unity Profiler was **26.73 FPS** for Layout 1, **23.23 FPS** for Layout 2, and **35.14 FPS** for Layout 3. Texture fidelity was evaluated with PaddleOCR, yielding **Precision: 0.986**, **Recall: 0.943**, and **CER: 0.014** on product-label text. The paper notes that OCR still struggles with rotated text and stylized fonts [2508.00400].

## 4. Interaction model, API, and human benchmarking protocol

Sari Sandbox is designed to be controlled through a **Python API** that communicates with a Unity/C# server over **WebSocket**. The API exposes functions for moving the agent or camera, moving the hands, gripping and poking, capturing screenshots, and resetting the environment. The listed core functions are `TransformAgent(T, R)`, `TransformHands(leftT, leftR, rightT, rightR)`, `ToggleLeftGrip()`, `ToggleRightGrip()`, `ToggleLeftPoke()`, `ToggleRightPoke()`, `RequestScreenshot()`, and `Reset()`. The API also provides state information such as avatar position, rotation, hand grip state, store layout, and randomization seed [2508.00400].

For human data collection, the environment supports **VR headsets**, and the reported setup used a **Meta Quest 2**. The avatar consists of **a camera**, **two hands**, and **character-controller-based movement**. To improve comfort and reduce cybersickness, the environment includes **teleportation**, **a tunneling vignette**, **haptic feedback on hover/interactables**, and hand-based **grabbing, touching, placing, and throwing**. If no VR headset is detected, the camera is positioned at about **1.6 m eye height**, with hands offset downward to mimic a standing human posture [2508.00400].

The human evaluation protocol measures **completion time** and **completion rate**. The corresponding reported benchmark metrics are **HAT** for **Human Average Time** and **HCR** for **Human Completion Rate**. The paper states that lower time is better and higher completion rate is better. These metrics are mirrored on the agent side by **AAT** for **Agent Average Time** and **ACR** for **Agent Completion Rate** [2508.00400].

This protocol is methodologically notable because it supports matched-condition evaluation. Humans and agents act in the same simulated store, with the same object inventory and the same operational task definitions. A plausible implication is that SariBench is intended to measure not only absolute task solvability but also the performance gap between contemporary embodied agents and human participants under retail-specific constraints.

## 5. Baseline embodied agent and benchmark results

The baseline embodied agent in the paper is a **VLM-powered agent built with Gemini 2.5 Pro**, using a **ReAct-like reasoning-and-action loop plus memory**. It is explicitly described as simple and **not fine-tuned**, serving primarily as a proof of concept for the API and environment rather than as a highly optimized policy. Its behavior is decomposed into navigation actions such as `move_forward`, `pan_left`, and `pan_right`; manipulation actions such as `center_object_on_screen` and `retrieve_item`; and tools including `loc_object`, `ocr_object`, and `est_depth` [2508.00400].

Agent evaluation is limited to the **easy tasks**. The paper states that harder-task evaluation is future work. Agent success was assessed by **manual visual inspection**, with success defined as task completion within **45 minutes**; failure included exceeding that limit or entering **mode collapse** by getting lost. Human and agent performance on the easy tasks is summarized below [2508.00400].

| Task | Human (HAT / HCR) | Agent (AAT / ACR) |
|---|---|---|
| Easy-L1 | 47 s / 88.88% | 780 s / 68.63% |
| Easy-L2 | 73 s / 100.00% | 660 s / 45.10% |
| Easy-L3 | 61 s / 93.33% | 420 s / 33.33% |

For the more difficult tiers, the paper reports human-only results: **Average-L1: 158 s, 87.50%**; **Average-L2: 106 s, 100.00%**; **Average-L3: 84 s, 100.00%**; **Difficult-L1: 76 s, 100.00%**; **Difficult-L2: 136 s, 100.00%**; **Difficult-L3: 113 s, 100.00%** [2508.00400].

The main empirical finding is that **humans substantially outperform the embodied agent** in both speed and reliability, even on easy tasks. The paper states that agent completion times were **up to 16× longer than humans**, and agent success rates were **under 70%**, while humans were often near **100%**. The reported failure factors for the agent include **slow VLM inference**, **heavy text-generation overhead**, **limited reasoning/planning quality**, **lack of robust navigation and manipulation planning**, and susceptibility to **mode collapse**. Humans, by contrast, quickly reasoned about likely object locations, adapted to visual cues, and completed tasks more reliably [2508.00400].

A second misconception addressed by the results is that a functioning retail simulation implies strong embodied-agent performance. The benchmark instead shows that a usable environment and a solvable task format do not by themselves close the gap between contemporary VLM-based control loops and human retail behavior.

## 6. Limitations, interpretation, and future development

The paper is explicit that both Sari Sandbox and SariBench remain incomplete. One limitation is runtime performance: the environment does **not consistently reach 60 FPS** on the tested desktop, and layout complexity affects runtime. A second limitation is evaluation coverage: the agent was tested only on **easy tasks**, so the benchmark does not yet provide agent baselines for the full average and difficult spectrum. A third limitation concerns OCR and perception, where **rotated, vertical, and stylized text** remain difficult. A fourth limitation is realism: products are interactive, but all are currently **rigid bodies**, and **mesh colliders** are not yet used extensively. A fifth limitation is human-study ergonomics, since some participants reported **motion sickness** [2508.00400].

The paper also notes nuances in human performance. Humans were not perfect: some completion-rate dips were attributed to **perseverance**, **occasional carelessness**, **unfamiliarity with the scanner**, and **motion sickness for some participants**. Even so, human performance remained well above the agent baseline [2508.00400].

Future directions are stated concretely. The authors highlight the need to improve **runtime efficiency**, **automated annotation generation**, **dynamic dataset expansion**, a **user-friendly scene-creation tool**, and updated store designs. They recommend optimizing to a stable **60 FPS**, expanding SariBench with more tasks and dynamic additions, improving realism via **deformable objects** and **mesh colliders**, evaluating agents across **all task difficulties**, improving **VLM context engineering** for better reasoning and planning, automating dataset annotation, developing a scene-creation tool for rapid prototyping, and further mitigating VR discomfort [2508.00400].

Taken together, these details place SariBench as a retail-specific embodied-AI benchmark whose current value lies in its domain specificity, its integration of human VR demonstrations with agent evaluation, and its exposure of a substantial gap between proof-of-solvability and robust embodied competence. The benchmark’s role is less that of a closed leaderboard and more that of an empirical substrate for future work on perception, manipulation, OCR-mediated interaction, memory, and decision making in retail environments [2508.00400].

Source: https://www.emergentmind.com/topics/saribench