Sari Sandbox for Retail Embodied AI

• Sari Sandbox is a high-fidelity, photorealistic 3D retail store simulation designed specifically for benchmarking embodied AI in grocery-shopping tasks.
• It models real convenience store environments with over 250 interactive grocery items, dynamic product labeling, and supports both VR and API-based control.
• The platform integrates SariBench, providing annotated human demonstrations and baseline agent evaluations to assess perception, memory, and decision making.

Sari Sandbox is a high-fidelity, photorealistic 3D retail store simulation introduced for embodied AI agents and human benchmarking in grocery-shopping tasks. It was motivated by a gap in retail-specific simulation environments for embodied agent training, since widely used embodied AI platforms have tended to emphasize household, general indoor navigation, industrial, or humanoid-control settings rather than retail-specific behaviors such as product search, shelf inspection, label reading, barcode scanning, item comparison, and checkout. The environment models small convenience stores, supports both virtual reality (VR) interaction and API-based agent control, includes over 250 interactive grocery items across three store configurations, and is paired with SariBench, a benchmark of annotated human demonstrations across varied task difficulties (Gajo et al., 1 Aug 2025).

1. Scope and design objectives

The environment was designed as a retail-specific benchmark rather than as a generic warehouse, domestic, or navigation-only simulator. Its stated target is shopping behavior in small convenience stores, including narrow aisles, grouped products, shelf browsing, and point-of-sale interaction. This focus distinguishes it from platforms such as Habitat, AI2-THOR, Matterport3D, and ThreeDWorld, which the paper treats as less well aligned with retail-specific embodied tasks (Gajo et al., 1 Aug 2025).

Implementation is in Unity using the Universal Render Pipeline (URP), with Unity’s default 3D engine, NVIDIA PhysX, handling physics. Four explicit requirements guided the design: performance, with a target of 60 FPS without compromising visual quality; diverse products, with textures detailed enough that humans and AI can read small text; API access, for control of avatar and environment and retrieval of state information; and VR capability, optimized to minimize fatigue and nausea. The resulting system is intended for both human studies and embodied-agent benchmarking rather than for a single training paradigm.

The store layouts were recreated from surveys of real-world small-scale retail stores. Each layout includes shelves, a self-checkout counter, semantically grouped products, overhead category labels, price tags, and, in some configurations, hinge-door cabinets and sliding-door cabinets to simulate refrigerated storage. The paper does not provide exact dimensions of the three layouts, but it states that they differ in size and object count.

2. Retail environment and product representation

A central feature of Sari Sandbox is the visual realism of the merchandise. The paper describes 250 3D grocery products across 11 food categories, modeled after real packaged goods commonly found in convenience stores. Packaging was based on scanned real goods, restricted to items whose packaging could be flattened for flatbed scanning; GIMP was used for image preprocessing and Blender for 3D modeling. Products were organized into three mesh types—simple, complex, and deformable-looking items—although all products are simulated as rigid bodies in the current system (Gajo et al., 1 Aug 2025).

The product inventory is summarized below.

Category	Count
Water	12
Soda	23
Juice	16
Dairy	20
Biscuit	50
Can	59
Chips	40
Nuts	15
Soup	6
Noodles	7
Liquor	2

The environment attaches rich metadata to products, including category, name, price, net weight, ingredients, nutritional facts, allergens, and manufacturing origin. This supports tasks that go beyond recognition, such as nutrition-based comparison and product selection under textual constraints. The paper repeatedly treats this as important for benchmarking perception, memory, and decision making jointly.

Retail realism is also reinforced through dynamic labels and inspection mechanisms. Barcodes are integrated into product prefabs, expiration dates are randomly generated at runtime and rendered as decals using URP, and price tags update dynamically according to randomized shelf placement. Similar product types remain semantically grouped even when exact shelf positions are randomized. To reduce rendering cost, barcodes and expiration dates activate only when a product is grabbed.

3. Interaction model and control API

Both humans and agents interact through an embodied avatar with a camera, two hands, and a character controller. When VR is not active, the camera is placed at approximately 1.6 meters eye height, with the hands offset 0.5 meters downward. Within this interface, users can navigate through the store, grab objects, place or throw objects, inspect labels, interact with refrigerated cabinets, scan barcodes, and use a self-checkout touchscreen (Gajo et al., 1 Aug 2025).

The self-checkout is modeled as a concrete interaction surface rather than as an abstract menu. It includes a touchscreen operated by the avatar’s index finger, buttons to add or remove products and start checkout, and a fixed barcode scanner. Barcode scanning is simulated by ray casting to a dedicated barcode plane placed in front of the actual barcode surface, with successful scanning depending on correct orientation. This makes checkout a manipulation and alignment problem rather than a trivial trigger.

Programmatic control is exposed through a direct Python API connected to Unity through a Python client and a C# server, communicating over a WebSocket server using JSON messages. The paper lists the following core functions: TransformAgent(T, R), TransformHands(leftT, leftR, rightT, rightR), ToggleLeftGrip(), ToggleRightGrip(), ToggleLeftPoke(), ToggleRightPoke(), RequestScreenshot(), and Reset(). The parameters T and R accept 3D vector inputs. The API can also return environment data such as avatar position, avatar rotation, hand grip state, store layout, and randomization seed.

The paper does not formalize Sari Sandbox as a full Markov decision process. It provides a practical action interface and benchmark task families instead of explicit state-space, reward-function, or transition equations. This suggests a systems benchmark orientation rather than a native reinforcement-learning formalization.

4. Human interaction studies and SariBench

SariBench is the benchmark and dataset introduced alongside Sari Sandbox. It is organized around three difficulty levels. Easy tasks involve perception, navigation, and manipulation, as in “Find and pick up a box of cereal.” Average tasks add memory and task execution, as in “Pick up a bottle of soda and scan at checkout.” Difficult tasks add decision making and comprehension, as in “Which of these two products has lower sugar content: strawberry-flavored biscuit or chocolate-flavored biscuit? Scan the answer” (Gajo et al., 1 Aug 2025).

Human interaction data were collected in VR using a Meta Quest 2 headset. The paper reports 20 human participants and states that SariBench currently comprises 100 videos of participants completing retail tasks. Each participant received a briefing on the project and recorded data, was given 15 minutes to familiarize themselves with VR controls, practiced in a separate playground environment, and then completed a task set consisting of 3 easy, 2 average, and 2 hard tasks. During execution, participants were encouraged to verbalize their thought process; these think-aloud protocols were recorded and later transcribed.

The dataset includes screen recordings and synchronized environment data logged at 10 frames per second, including global head position and rotation, global hand position and rotation, grip state, and hovered or held item. The paper describes these as annotated human demonstrations, but it does not provide a detailed annotation schema such as label taxonomy or inter-annotator agreement. It also reports 20 participants completing 7 tasks each while separately stating that the dataset currently contains 100 videos; the relationship between those counts is not further clarified.

A notable contribution of the human study is the extraction of generalized flowcharts of participant thought processes for easy, average, and difficult tasks. These flow patterns emphasize likely-location inference, iterative search and verification, memory-based comparison, barcode-orientation strategies at checkout, and conditional replanning. This suggests a benchmark role not only for action traces but also for comparative studies of reasoning structure.

5. Embodied agent baseline

The paper presents a VLM-powered embodied agent as a proof of solvability rather than as a state-of-the-art system. The architecture follows an adapted ReAct-style pattern, but instead of emitting a single action per cycle it emits an action sequence. The control loop is divided into three stages: an associative learning step, which combines current sandbox state and task prompt to produce a mode cue and context cue; a planning step, which uses current state, cues, and available tools to obtain a structured action sequence; and an execution step, where an external parser converts the action sequence into API calls and sends them to the environment (Gajo et al., 1 Aug 2025).

The baseline agent uses Gemini 2.5 Pro (gemini-2.5-pro-preview-05-06) with a reasoning budget of 2048 tokens. It also integrates PaddleOCR for text extraction and Depth Anything V2 Small for depth estimation. Memory is explicitly structured into procedural memory, working memory, semantic memory, and episodic memory. In implementation, semantic memory is a text file containing store layout and product-location information, while episodic memory is updated with a three-point reflection after each action: what occurred, what worked, and what to avoid in the future.

Perception is based on first-person screenshots returned by RequestScreenshot(), object localization by Gemini, OCR, and monocular depth estimation. The paper defines three tools: loc_object, returning a bounding box in [ymin, xmin, ymax, xmax] format; ocr_object; and est_depth. Action control is organized into two high-level modes: navigation and manipulation. Navigation includes move_forward, pan_left, and pan_right; manipulation includes routines such as center_object_on_screen and retrieve_item.

The current agent omits several capabilities that the environment itself supports. The paper states that complex behaviors such as multi-item handling or checkout were excluded from the current design, that A* was not used because the agent has no pre-built grid map, and that evaluation was limited to easy tasks only. This makes the baseline primarily a systems validation artifact rather than a comprehensive embodied-shopping agent.

6. Evaluation, human–agent gap, and limitations

The paper reports both environment-level and task-level evaluations. In Unity profiling over 300 frames, average frame rates were 26.73 FPS for Layout 1, 23.23 FPS for Layout 2, and 35.14 FPS for Layout 3. The environment therefore did not yet meet its own stated 60 FPS target on the reported hardware. To assess text readability, the authors benchmarked PaddleOCR on product label text and reported precision 0.986, recall 0.943, and CER 0.014, while noting failures for rotated text, mixed vertical and horizontal text, and stylized brand or logo typography (Gajo et al., 1 Aug 2025).

For human performance, the paper reports 108 assigned tasks across difficulty levels and layouts. Easy tasks were the fastest; average tasks often took longer than difficult tasks because participants were less familiar with the barcode scanner; and not all difficult tasks required checkout. For embodied-agent benchmarking, only easy tasks were evaluated. The human–agent gap on easy tasks is summarized below.

Layout	Human	Agent
Easy-L1	47 s, 88.88%	780 s, 68.63%
Easy-L2	73 s, 100.00%	660 s, 45.10%
Easy-L3	61 s, 93.33%	420 s, 33.33%

Humans also achieved 87.50–100.00% completion on average tasks and 100.00% completion on difficult tasks, with reported times of 84–158 s for average tasks and 76–136 s for difficult tasks. By contrast, the embodied agent’s easy-task completion rates remained below 70% in all layouts, and its completion times were up to 16 times longer than human performance. The paper attributes this gap to VLM computational overhead, long inference and text-generation time, suboptimal reasoning and decision making, and the baseline status of the current agent.

Several limitations are explicit. Deformable-looking products are still simulated as rigid bodies, only box colliders are used, barcode scanning is implemented through raycast and orientation logic rather than a more physically grounded scanner model, and the current agent has not been evaluated on average or difficult tasks. The paper also reports no formal ablation study within the environment paper itself.

7. Terminological ambiguity and research role

In embodied AI, “Sari Sandbox” denotes the virtual retail store environment and benchmark described above. The phrase, however, is not entirely unique. A different paper uses “SARI Sandbox” to describe a controlled simulation environment for bias(stress)-testing fairness algorithms, where specific biases can be injected into an ML pipeline and fairness interventions can be compared against an unbiased benchmark (Akpinar et al., 2022). This suggests that the term is ambiguous across research areas, even though the retail embodied-AI usage is the exact title associated with the shopping-task simulator (Gajo et al., 1 Aug 2025).

Within embodied AI, the significance of Sari Sandbox lies in extending benchmark design into a retail domain that had been underrepresented relative to household and navigation settings. Its combination of photorealistic packaged goods, runtime-generated retail labels, self-checkout interaction, VR human studies, synchronized motion logs, think-aloud transcripts, and API-based agent control makes it a benchmark for jointly studying perception, OCR, manipulation, memory, and decision making in shopping tasks. A plausible implication is that it can serve as a controlled experimental substrate for future work in imitation learning, human–agent comparison, and retail-specific embodied planning, although the paper itself presents the platform primarily as a benchmark and proof-of-solvability environment rather than a finished training stack.