BEHAVIOR-1K: Human-Centered Embodied AI Benchmark

• BEHAVIOR-1K is a large-scale embodied AI benchmark defining 1,000 human-centered activities across diverse, high-fidelity simulated scenes using formal logical task specifications.
• It integrates survey-driven activity selection from over 2,000 candidate tasks with OmniGibson, a simulation environment supporting rigid, deformable, and fluid dynamics for realistic interactions.
• The benchmark exposes key challenges in long-horizon, contact-rich robot learning, highlighting sim-to-real transfer issues, the need for robust grasping, and memory integration in robotic systems.

Searching arXiv for the BEHAVIOR-1K paper and closely related benchmark context. BEHAVIOR-1K is a large-scale embodied AI benchmark for human-centered robotics that couples a survey-grounded task distribution with a high-fidelity simulation stack. It defines 1,000 everyday activities, grounded in 50 scenes and more than 9,000 objects annotated with rich physical and semantic properties, and pairs that task layer with OmniGibson, a simulation environment built to support rigid bodies, deformable bodies, and liquids under realistic physics and rendering (Li et al., 2024). The benchmark is explicitly framed around a human-centered question—what people actually want robots to do for them—rather than a researcher-selected catalog of canonical manipulation problems. The result is a benchmark that combines formal task specification, ecologically structured scenes and assets, nontrivial object-state dynamics, and initial sim-to-real calibration, with the stated aim of exposing the long-horizon and contact-rich challenges that remain difficult for current robot learning systems.

1. Human-centered rationale and survey grounding

A defining feature of BEHAVIOR-1K is that its activity distribution is survey-driven rather than purely researcher-designed. The benchmark is motivated by the claim that embodied AI evaluation should be grounded in human needs, goals, and values, and the paper operationalizes that claim by conducting a large survey on “what do you want robots to do for you?” (Li et al., 2024).

The candidate activity pool is assembled from approximately 2,090 activities drawn from three time-use survey resources—the American Time Use Survey (ATUS), Harmonised European Time Use Surveys (HETUS), and Multinational Time Use Study (MTUS)—together with WikiHow. The activity list is then filtered using simulation feasibility constraints. Excluded categories include tasks involving unsupported physics or chemistry, media consumption or creation, tasks taking more than a day, non-visual modalities, too-fine geometric constraints, branded items, and humans or live animals.

The preference study uses 1,461 Amazon Mechanical Turk participants. Each respondent answers 50 Likert-scale questions per batch, scoring how much they want a robot to perform each activity on a 1–10 scale. The study also varies wording—“robot” versus “assistant” versus “automation”—and compares Likert against best-worst scaling. The reported outcome is that wording changes do not produce a statistically significant difference and that Likert rankings strongly agree with best-worst scaling, so the benchmark adopts the 10-point Likert formulation.

The survey results show substantial dispersion in automation preferences, quantified by a Gini index of 0.158. Tasks characterized as tedious chores and unpleasant work rank highly, whereas pleasurable activities rank low. The final benchmark composition reflects those rankings: 909 highest-ranked activities from the survey are combined with 91 activities from BEHAVIOR-100 to produce 1,000 activities total. A common misconception is that BEHAVIOR-1K is merely a scale-up of earlier benchmark suites; the paper instead presents it as a reprioritization of task selection around human preference data.

2. Benchmark composition, formal task language, and asset inventory

BEHAVIOR-1K consists of two tightly coupled components: the BEHAVIOR-1K Dataset and OmniGibson. The dataset layer defines activities in BDDL, described as a BEHAVIOR extension of PDDL, while the simulator provides the physical substrate needed to instantiate those definitions (Li et al., 2024).

The activity definitions use logical predicates for initial and goal conditions and encode object types, object states, object-object relations, and transition rules for complex processes. The paper emphasizes that these activities are substantially more diverse and longer-horizon than prior suites: each activity requires between 2 and 11 distinct state changes and involves between 3 and 47 objects.

The environment layer comprises 50 fully interactive scenes across 8 scene types: houses, houses with gardens, hotels, offices, grocery stores, halls, restaurants, and schools. The scenes contain 373 rooms in total. This expands beyond the 15 house scenes of BEHAVIOR-100 to a broader collection of residential, commercial, and institutional environments.

The object layer is similarly broad. The paper summarizes the asset base as 9,318 object models across 1,949 categories, and elsewhere describes it as spanning 1,900+ categories. Objects are annotated with semantic properties, physical properties, and functional properties. Semantic annotations include category and synset labels with WordNet hierarchy information. Physical annotations include friction, mass, articulation, center of mass, and moment of inertia. Functional annotations include cookable, fillable, openable, sliceable, foldable, and toggleable.

Component	Quantitative scope	Formal content
Activities	1,000	BDDL initial and goal conditions
Scenes	50 scenes, 373 rooms, 8 scene types	Fully interactive environments
Objects	9,318 models, 1,949 categories	Physical, semantic, and functional properties

The object taxonomy is built from WikiHow-based activity analyses and mapped to WordNet synsets, yielding 2,964 leaf synsets in total: 1,538 WordNet leaf synsets and 1,426 custom synsets. For non-leaf synsets, properties are defined as the intersection of descendant properties to avoid “unsolveability” caused by overly generic categories. Annotation quality is checked by humans; the reported approval and accuracy figures are typically above 96.8%, with F1 scores above 0.91 and low false positive and false discovery rates. This suggests that the benchmark’s scale is paired with a nontrivial attempt at semantic and physical consistency rather than simple asset accumulation.

3. OmniGibson as the execution substrate

OmniGibson is the simulation environment that makes the benchmark executable. It is built on Nvidia Omniverse and PhysX 5 and is designed to support realistic physics, rendering, sensor modalities, and object-state tracking (Li et al., 2024). The paper treats this simulator as a constitutive part of the benchmark rather than an interchangeable backend.

A central technical claim is that OmniGibson simulates more than rigid-body kinematics. It supports rigid bodies, deformable bodies, cloth, liquids, and flexible materials. This matters because many benchmark activities require physically nontrivial interactions such as folding towels, pouring liquids, cleaning with wet cloths, mixing ingredients, or handling dough or cloth. The paper explicitly states that without these capabilities, over half of the 1,000 activities would not be simulatable.

OmniGibson also tracks extended non-kinematic states. Examples include Temperature, MaxTemperature, SoakedLevel, CoveredLevel, ToggledState, SlicedState, and BrokenState. These state variables are updated every step by heuristic rules tied to nearby sources or contacts. The paper gives representative thresholds, including OnFire(o) when Temperature > T_{\mathit{onfire}} with default $T_{\mathit{onfire}} = 300^\circ C$, Frozen(o) when $T_o \le T_{\mathit{frozen}} = 0^\circ C$, Heated(o) when $T_o \ge 75^\circ C$, Boiled(l) when $T_o \ge 100^\circ C$, Soaked(o,l) when absorbed liquid exceeds a threshold with default 50 particles, and Filled(o,l) when volume occupancy exceeds a threshold with default 0.5.

The benchmark formalizes tasks through logical predicates that are both checkable from physical state and sampleable into valid initial conditions. The paper lists predicates such as InsideOf, OnTopOf, NextTo, InContactWith, ConnectedWith, Open, Cooked, Burnt, Frozen, Heated, Boiled, Filled, Covered, ToggledOn, Sliced, Broken, Folded, Assembled, Blended, InFoVOfAgent, InHandOfAgent, and InReachOfAgent. This dual role—predicate evaluation and predicate-based sampling—is important because the benchmark is intended to support infinitely many valid task instantiations rather than a small set of manually curated episodes.

The paper also evaluates visual realism through a human study with 60 participants comparing OmniGibson against AI2-THOR, ThreeDWorld, Habitat 2.0, and iGibson 2.0. OmniGibson receives the highest reported realism score, $3.20 \pm 1.23$, compared with 1.74 ± 1.33 for Habitat 2.0, 1.73 ± 1.37 for AI2-THOR, 1.69 ± 1.24 for iGibson 2.0, and 1.65 ± 1.23 for ThreeDWorld. The paper uses this result to argue that realism in BEHAVIOR-1K is not limited to task semantics or physics coverage but extends to visual appearance as well.

4. Task difficulty, long-horizon structure, and baseline performance

The paper’s empirical analysis focuses on three representative tasks: StoreDecoration, CollectTrash, and CleanTable (Li et al., 2024). These are chosen to span distinct physical demands: rigid-body manipulation, articulated-object manipulation, and cloth-and-fluid interaction.

Three baseline solution families are evaluated. RL-VMC uses end-to-end visuomotor control with SAC. RL-Prim. uses PPO over action primitives. RL-Prim. + history augments the primitive-based policy with a short observation history. The action primitive set includes navigate, pick, place, push, dip, and wipe. These primitives are implemented with sampling-based motion planning, and the pick primitive uses an assistive grasping abstraction during training to reduce the difficulty of physical grasping.

Task	RL-Prim.	RL-Prim. + history
StoreDecoration	0.48	0.55
CollectTrash	0.42	0.63
CleanTable	0.77	0.88

The paper reports that RL-VMC fails completely on all three tasks, with 0% success. Primitive-based methods perform substantially better, and adding history improves all three tasks. CleanTable can be solved in about 6 primitive steps, whereas CollectTrash requires at least 16 primitive steps. The interpretation given is that BEHAVIOR-1K tasks are difficult because they are long-horizon, sparse-reward, multi-step, and multi-modal in physical interaction. This difficulty profile is not incidental; it follows from the benchmark’s effort to model everyday activities rather than isolated rearrangement subproblems.

Efficiency is also evaluated using distance traveled, simulated time, and kinematic object disarrangement. The paper reports that history generally improves efficiency by reducing redundant action loops, especially in tasks that require remembering what has already been cleaned or collected. This suggests that memory and partial progress tracking are operationally important even for apparently simple household procedures.

An important ablation concerns realism. Simplifying grasping during training substantially distorts downstream performance. When fully physics-based grasping is enabled at evaluation time, success drops sharply, whereas enabling full motion execution causes much less degradation. The stated implication is that planning and navigation simplifications may be more tolerable than grasping simplifications, and that robust contact-rich manipulation remains a major bottleneck for benchmark performance.

5. Sim-to-real calibration with a mobile manipulator

BEHAVIOR-1K includes an initial sim-to-real calibration study built around CollectTrash in a digitally reconstructed apartment (Li et al., 2024). The paper presents this not as a solved transfer problem, but as an empirical probe of where the simulation stack aligns with reality and where it does not.

The setup uses a digital twin of a real mock apartment. The apartment is scanned, walls, floors, and ceilings are replaced with modeled textures, and 3D objects from the dataset are inserted. The physical robot is a Tiago++ mobile manipulator with an omnidirectional base, two 7-DoF arms, parallel-jaw grippers, a prismatic torso, front and rear LiDAR, and an ASUS Xtion RGB-D camera. Simulation and real deployment both use the same 2D map for localization.

The real-world action pipeline implements navigate, pick, and place. For object localization, the system runs YOLOv3 on RGB images, uses the depth map over a 21×21 window around the detection center, estimates a 3D centroid, and feeds that estimate to the same sampling-based motion planner used in simulation. The evaluation compares an optimal policy using human input with a learned policy trained in OmniGibson, and applies image-based data augmentation during training to improve transfer.

The experimental protocol randomizes robot starting location, object positions, and object orientations, and also varies lighting conditions for vision-based policies. Reported outcomes are approximately 40% success in simulation over 50 runs, approximately 22% success for the real-world optimal policy over 27 runs, and 0% success for the real-world trained policy over 26 runs. These numbers make the sim-to-real gap explicit rather than implicit.

The paper’s failure analysis identifies several major discrepancy sources. Grasping failures account for around 40% of real-world failures. For the learned policy, 44% of errors are attributed to wrong action selection due to perception mismatch. Additional failure sources include differences in camera dynamic range and texture or reflection modeling, navigation errors that place the robot poorly for subsequent manipulation, and object detection or depth failures. The paper stresses that real-world failure modes differ from simulation failure modes: in simulation, policy-selection or planning errors dominate, while in reality actuation and perception noise are much more consequential.

The study is explicitly limited. The paper notes simplified grasping in simulation, perfect localization assumptions in parts of the simulator, missing sensor noise models, and restriction to a single apartment and robot setup. A plausible implication is that the benchmark is intended less as a turnkey transfer suite than as a controlled environment for diagnosing which forms of realism matter most for downstream embodied performance.

6. Research significance and benchmark role

BEHAVIOR-1K occupies a distinctive position among embodied AI benchmarks because it combines four axes that are often only partially present elsewhere: human-centered task selection, large-scale activity diversity, rich physical semantics, and executable simulation realism (Li et al., 2024). The paper argues that the benchmark matters not simply because it contains more tasks, but because it changes which tasks are emphasized and what kinds of capabilities they require.

From the task-distribution perspective, the benchmark is anchored in what survey respondents prefer to automate. From the representational perspective, activities are specified in a formal logical language with initial and goal conditions and a large knowledge base of scenes, objects, properties, and transitions. From the simulator perspective, OmniGibson supports rigid bodies, deformable bodies, cloth, fluids, thermal processes, mixing, cleaning, composition, and decomposition. From the evaluation perspective, the benchmark includes baseline results and a real-world calibration study rather than remaining purely notional.

The benchmark therefore exposes several limitations of current methods. End-to-end visuomotor control with SAC fails on the representative tasks considered. Primitive-based methods are more viable, but still depend on memory and are sensitive to realism in grasp execution. Sim-to-real transfer remains weak, with the trained policy achieving 0% success in the real apartment study. The paper uses these observations to motivate future work in hierarchical policies, memory, task and motion planning, better grasping, sim-to-real transfer, and richer world models.

A common misunderstanding is to reduce BEHAVIOR-1K to a dataset of activity names or to treat OmniGibson as merely an implementation convenience. The benchmark is better understood as a coupled task-and-simulator ecosystem. The dataset defines what counts as a valid activity in logical and semantic terms; OmniGibson defines the physical mechanisms by which those activities can be instantiated, checked, and learned. That coupling is central to the benchmark’s identity and to its intended use as a testbed for human-centered embodied AI.