Do researcher-designed embodied AI benchmarks address actual human needs?

Ascertain whether the tasks and activities included in existing embodied AI simulation benchmarks that are designed by researchers in fact address the real needs and preferences of humans, rather than reflecting only researcher assumptions about useful tasks.

Background

The paper motivates BEHAVIOR-1K by noting that many existing embodied AI benchmarks are created by researchers without direct grounding in layperson needs. The authors conducted a large-scale survey to prioritize activities people most want robots to perform, explicitly to address this gap.

This uncertainty matters for human-centered AI: if benchmarks are misaligned with actual human needs, advances measured on those benchmarks may not translate into meaningful real-world utility. Establishing this alignment requires systematic evaluation of benchmark task sets against human preference data.