GemBench: Robotic Generalization Benchmark

• GemBench is a simulation benchmark that tests robotic manipulation policies on tasks requiring spatial, object, and compositional generalization.
• It organizes tasks into four levels targeting novel placements, rigid objects, articulated objects, and long-horizon compositions using seven action primitives.
• The benchmark employs standardized evaluation with multi-view RGB-D inputs and binary success rates to diagnose perception, grounding, planning, and execution challenges.

GemBench is a simulation benchmark for evaluating generalization in vision-language robotic manipulation. Introduced by Garcia, Chen, and Schmid as a benchmark built on RLBench, it combines language instructions, multi-view RGB-D observations, realistic physics, and a task suite organized around seven general action primitives and four progressively harder levels of generalization. Its central purpose is to test whether a manipulation policy trained on a bounded set of tasks can transfer to novel placements, unseen rigid objects, unseen articulated objects, and multi-step long-horizon compositions rather than merely reproducing performance on seen tasks (Garcia et al., 2024).

1. Origin and motivation

GemBench was introduced in response to a specific gap in language-conditioned robot learning: many existing simulation benchmarks evaluate policies on the same tasks used for training, or emphasize perturbations that do not amount to true task-level generalization. The benchmark paper contrasts this with several existing environments. RLBench-18Task and RLBench-74Task provide realistic physics and varied tasks, but train and test on the same tasks. CALVIN includes long-horizon language-conditioned tasks but has limited generalization categories and uses PyBullet rather than RLBench. ALFRED evaluates generalization but operates in AI2-THOR without realistic physics. VIMA-Bench evaluates generalization but is primarily pick-and-place with a suction gripper. Ravens and Arnold cover some generalization to attributes or instances but are less comprehensive for multi-skill and articulated-object manipulation. Colosseum focuses more on environment perturbations such as lighting and camera changes than on novel tasks (Garcia et al., 2024).

Against that background, GemBench was designed as a benchmark for “generalizable vision-language robotic manipulation.” It is not simply a larger task collection; it is a structured test of transfer under controlled novelty. The benchmark stresses generalization to changed spatial layouts, new object-color compositions, new object shapes, new articulated mechanisms, and new task compositions assembled from previously learned skills. This design makes it particularly suited to analyzing where failure originates: object grounding, geometric transfer, articulation understanding, or multi-step planning (Garcia et al., 2024).

2. Benchmark structure and task space

GemBench is built on the RLBench simulator and runs in the RLBench/CoppeliaSim environment with language instructions. The benchmark defines seven general action primitives: press, pick, push, screw, close, open, and stack/put. These primitives are instantiated by manipulation tasks such as Push button and Lamp on for press, Pick and lift for pick, Slide block and Reach and drag for push, Screw bulb in and Close jar for screw, articulated-object tasks such as Open drawer and Close microwave for open and close, and placement or assembly tasks such as Stack blocks, Stack cups, Put groceries in cupboard, Put money in safe, and Put cube in safe for stack/put (Garcia et al., 2024).

The benchmark is organized into a training split and four test levels.

Level	Composition	Generalization target
L1	16 tasks, 31 variations	Novel placements and distractors
L2	15 unseen tasks, 28 variations	Novel rigid objects
L3	18 unseen tasks, 21 variations	Novel articulated objects
L4	6 tasks, 12 variations	Novel long-horizon task compositions

The training set contains 16 tasks with 31 variations and, in the benchmark paper, 100 demonstrations per variation, giving 3,100 demonstrations in total. The test set contains 44 tasks with 92 variations, including 23 tasks from the RLBench 100 tasks and 21 newly scripted tasks. The training set spans 20 objects, 20 colors, and 3 articulated parts: top, middle, and bottom (Garcia et al., 2024).

The four levels are deliberately heterogeneous. L1 preserves the same tasks as training but changes placements and adds distractors. L2 introduces novel rigid objects, including both novel object-color compositions and novel object shapes. L3 introduces new articulated objects through novel action-part compositions, novel instances, and novel categories. L4 contains six long-horizon tasks, including Push Buttons4, Take shoes out of box, Put items in drawer, Tower4, Stack cups, and Put all groceries in cupboard, each requiring sequencing of multiple sub-actions rather than single-step execution (Garcia et al., 2024).

GemBench also fixes a concrete sensor and control regime. The simulator uses four cameras—front, left shoulder, right shoulder, and wrist—at $256 \times 256$ resolution. RGB-D images are fused into a world-coordinate point cloud, with points downsampled to one point per $1 \text{ cm}^3$ voxel and robot-arm points removed using a CAD model and joint poses. The action space is waypoint-based: at time $t$, the action is $a_t = (a_t^p, a_t^r, a_t^o)$, where $a_t^p \in \mathbb{R}^3$ is position, $a_t^r \in \mathbb{R}^3$ is Euler-angle rotation, and $a_t^o \in \{0,1\}$ is the gripper open state (Garcia et al., 2024).

3. Evaluation protocol

GemBench uses binary task success as its primary online metric. For each episode, success is recorded as $1$ if the task-specific success condition is met and $0$ otherwise, and success rate is reported as

$$SR = \frac{1}{N_{\text{episodes}}} \sum_{i=1}^{N_{\text{episodes}}} \mathbf{1}[\text{success}_i].$$

Results are averaged per variation and then aggregated within each level, with mean and standard deviation reported across seeds (Garcia et al., 2024).

The standard evaluation protocol uses 5 seeds and 20 episodes per task variation per seed, for a total of $1 \text{ cm}^3$0 evaluation episodes across the full benchmark. Episodes are capped at 25 steps. Random seeds differ from those used in training so that initial scenes are diversified. Success conditions are inherited from RLBench task logic and may depend on placements, contacts, or articulated-joint configurations (Garcia et al., 2024).

This protocol makes GemBench unusually strict relative to benchmarks that average over fewer task families or do not separate types of novelty. A plausible implication is that mean performance on GemBench reflects several distinct competencies simultaneously: language grounding, 3D spatial reasoning, articulation handling, and temporal composition. Later papers reinforce this interpretation by reporting sharply different behavior across L1-L4 even when overall averages appear similar (Li et al., 9 Jun 2025).

4. Baselines and the initial performance profile

The benchmark paper evaluates several families of manipulation policies: Hiveformer, PolarNet, 3D Diffuser Actor, RVT-2, 3D-LOTUS, and 3D-LOTUS++ (Garcia et al., 2024). These methods differ in representation and modularity. Hiveformer is a history-aware multi-modal transformer policy. PolarNet is a 3D point-cloud policy. RVT-2 uses multi-view orthographic projections and coarse-to-fine refinement. 3D Diffuser Actor is diffusion-based. 3D-LOTUS is a language-conditioned 3D point-cloud transformer. 3D-LOTUS++ augments 3D-LOTUS with LLM-based task planning and VLM-based object grounding (Garcia et al., 2024).

The original performance pattern is strongly level-dependent. On L1, 3D-LOTUS is best at $1 \text{ cm}^3$1, indicating that precise 3D control and efficient training are sufficient when the benchmark shift is limited to new placements. On L2-L4, however, the modular 3D-LOTUS++ becomes state of the art, reaching $1 \text{ cm}^3$2 on L2, $1 \text{ cm}^3$3 on L3, and $1 \text{ cm}^3$4 on L4, while most end-to-end baselines remain near zero on long-horizon tasks (Garcia et al., 2024).

This split is one of GemBench’s most important empirical contributions. It shows that success on RLBench-like seen-task evaluation or even on GemBench L1 does not guarantee transfer to novel object identities, new articulated mechanisms, or compositional tasks. The benchmark therefore separates control competence from generalization competence more cleanly than evaluations that average both together (Garcia et al., 2024).

5. Use in subsequent research

GemBench rapidly became a stress test for methods claiming generalization in robot manipulation. BridgeVLA uses GemBench specifically to probe generalization beyond in-distribution RLBench training. In that work, GemBench training uses only keyframes from each trajectory and no demo augmentation. BridgeVLA projects reconstructed 3D scenes into three orthographic 2D views—top, front, and right—and predicts one 2D heatmap per view for translation, with back-projection into a 3D grid. On GemBench it reports a new average of $1 \text{ cm}^3$5, with $1 \text{ cm}^3$6 on L1, $1 \text{ cm}^3$7 on L2, $1 \text{ cm}^3$8 on L3, and $1 \text{ cm}^3$9 on L4, surpassing the baselines reported in that paper on average and specifically on novel rigid and articulated objects (Li et al., 9 Jun 2025).

Gondola evaluates GemBench in two ways: as an offline grounded planning problem and as an online closed-loop manipulation benchmark. Its planner consumes multi-view images and history plans and predicts interleaved text plans and segmentation masks. In the online setting, integrated with the same 3D low-level policy as 3D-LOTUS++, Gondola reaches $t$0 on L1, $t$1 on L2, $t$2 on L3, and $t$3 on L4. Its offline results also show that mask grounding, multi-view input, and pseudo long-horizon training materially improve planning correctness and mask IoU, especially on L4 (Chen et al., 12 Jun 2025).

CLAP, a later hierarchical coarse-to-fine method, treats GemBench as a benchmark for generalization under both distribution shift and restricted data. It trains with only 20 trajectories per variation, described as one-fifth of the data used by other baselines, and still reports $t$4 average success, with $t$5 on L1, $t$6 on L2, $t$7 on L3, and $t$8 on L4. Its coarse branch fine-tunes Qwen2.5-VL-3B with LoRA to decompose a task into step-wise sub-instructions and predict language-aligned 3D keypoints, while the fine branch uses SigLIP, DINOv2, 3D position embeddings, and a Multi-View Transformer (Hu et al., 28 Sep 2025).

Taken together, these results indicate that GemBench is sensitive to design choices in three distinct places: representation alignment between 3D geometry and VLM backbones, grounded planning over articulated and long-horizon tasks, and hierarchical decomposition into reusable step-conditioned skills. This suggests that GemBench is not merely a harder version of RLBench, but a diagnostic benchmark for how a system distributes competence across perception, grounding, planning, and execution.

6. Limitations, open problems, and name ambiguity

A persistent finding across GemBench studies is that long-horizon generalization remains difficult. In the original benchmark paper, most baselines score essentially zero on L4, and even 3D-LOTUS++ reaches only $t$9 (Garcia et al., 2024). BridgeVLA likewise records $a_t = (a_t^p, a_t^r, a_t^o)$0 on L4 despite strong L2 and L3 performance (Li et al., 9 Jun 2025). Gondola improves L4 to $a_t = (a_t^p, a_t^r, a_t^o)$1, but still reports failure modes tied to history-plan distribution shift, partial observability, and contact-rich motion execution (Chen et al., 12 Jun 2025). CLAP raises L4 further to $a_t = (a_t^p, a_t^r, a_t^o)$2, yet still records $a_t = (a_t^p, a_t^r, a_t^o)$3 success on some long-horizon tasks such as “Put Items in Drawer” and “Put All Groceries in Cupboard,” and explicitly notes that the policy lacks self-correction, so an incorrect step can cascade to failure (Hu et al., 28 Sep 2025).

These results have produced a consistent interpretation within the GemBench line of work: grounding and perception can substantially improve L2 and L3, but L4 requires stronger task decomposition, memory, and recovery from execution error. The original benchmark paper explicitly suggests future integration with LLMs for task decomposition, and later systems such as 3D-LOTUS++, Gondola, and CLAP can be read as progressively more explicit realizations of that direction (Garcia et al., 2024).

The name “GemBench” also has important ambiguities outside robotics. In another paper, the official benchmark name is GRE-bench, a peer-review generation benchmark built on the GEM and GEM-S mutual-information metrics rather than a robot benchmark (Xu et al., 2024). “GEM-Benchmark” has also been used for graph embedding evaluation on 100 real-world networks (Goyal et al., 2019), while GEMv2 is a multilingual natural language generation benchmark that is sometimes informally miscalled “GemBench” even though the paper uses “GEM” and “GEMv2” (Gehrmann et al., 2022). Within robotics and manipulation research, however, GemBench most commonly denotes the RLBench-based hierarchical generalization benchmark introduced for vision-language robotic manipulation (Garcia et al., 2024).