RoboView-Bias Benchmark

• RoboView-Bias is a benchmark that systematically quantifies visual bias in robotic manipulation by isolating visual attributes while keeping task variables constant.
• It employs a structured variant-generation framework and a two-stage perceptual-fairness validation protocol to ensure clear, unambiguous task instances.
• Empirical results show that camera viewpoint is the dominant factor, and a semantic grounding layer can reduce bias by approximately 54.5% in evaluated agents.

RoboView-Bias is a benchmark for systematically measuring visual bias in embodied agents for robotic manipulation. It defines “visual bias” as any systematic fluctuation in an agent’s task success that can be traced directly to changes in purely visual attributes—color, camera angle, and distance—while all other task-relevant variables remain fixed. The benchmark is presented as the first benchmark specifically designed to systematically quantify visual bias in robotic manipulation, and it adopts a principle of factor isolation to separate the effects of individual visual factors and their interactions. Using a structured variant-generation framework and a perceptual-fairness validation protocol, it constructs 2,127 task instances and evaluates three representative embodied agents across two prevailing paradigms, reporting that all agents exhibit significant visual biases, that camera viewpoint is the most critical factor, and that a semantic grounding layer can substantially reduce bias on MOKA by approximately $54.5\%$ (Liu et al., 26 Sep 2025).

1. Conceptualization of visual bias

In RoboView-Bias, visual bias is not treated as a generic robustness failure. Instead, it is defined narrowly as performance variation caused by purely visual attributes under controlled task conditions. The benchmark focuses on robotic grasping and asks whether an embodied agent’s success rate changes systematically when the scene differs only in factors such as color, camera pose, or camera-to-scene distance, with non-visual variables held fixed (Liu et al., 26 Sep 2025).

This framing distinguishes visual bias from broader forms of unfairness and distribution shift discussed in robot-learning literature. The survey “Fairness and Bias in Robot Learning” identifies a wider taxonomy spanning data collection, model training, deployment, and socio-structural context, including representation bias, measurement bias, evaluation bias, and deployment domain shift (Londoño et al., 2022). RoboView-Bias isolates one specific part of that landscape: perception-linked instability in embodied manipulation caused by controlled visual perturbations rather than by protected attributes or broader social context (Liu et al., 26 Sep 2025).

A common misconception is to equate robustness benchmarks with bias benchmarks. RoboView-Bias explicitly addresses a gap left by existing benchmarks that mainly emphasize generalization and robustness under perturbations, while systematic quantification of visual bias remains scarce (Liu et al., 26 Sep 2025). This suggests that a model can be robust in the conventional sense yet still exhibit structured visual preferences that are consequential for downstream planning and control.

2. Factor-isolated benchmark design

The benchmark is built around a structured variant-generation framework (SVGF) that partitions variability into two disjoint sets: Visual Perturbation Dimensions, $V=\{V_1,\dots,V_n\}$, and Task Context Dimensions, $D=\{D_1,\dots,D_m\}$ (Liu et al., 26 Sep 2025). The visual dimensions are the attributes under test, while the task-context dimensions comprise non-visual factors such as object position, shape, and language phrasing (Liu et al., 26 Sep 2025).

RoboView-Bias uses a single canonical manipulation task—grasping—because even one task with a combinatorial explosion of visual variants is already challenging (Liu et al., 26 Sep 2025). Within $V$, it includes color, camera pose, and distance scale. The color axis consists of 141 named HTML colors, spanning achromatic to highly saturated hues. The camera-pose axis consists of 21 full-orbit top-down viewpoints plus 9 small Euler-angle perturbations, specified as yaw/pitch offsets of $\pm 6^\circ$. The distance-scale axis consists of 9 discrete camera-to-scene distances via line-of-sight translations. Within $D$, it varies 4 initial positions, 4 object shapes, and 3 semantically identical language instructions (Liu et al., 26 Sep 2025).

The benchmark states that the total number of instances is

$$141\times(21+9)\times9\;\times\;4\times4\times3 \;=\;2{,}127$$

scenes, each guaranteed to be visually clear and solvable under a fixed set of baseline values $B$ for all other visual dimensions (Liu et al., 26 Sep 2025).

To formalize factor isolation, RoboView-Bias defines, for dimension $V_i$, a Generalization Context Space

$C_{\text{Gen}(V_i)\;=\;\{\,d\cup B_{-i}\mid d\in D_{\text{context}\}\,,$

where $V=\{V_1,\dots,V_n\}$0 is the structured union of single-factor context variations (Liu et al., 26 Sep 2025). The corresponding experimental Task Subspace is

$V=\{V_1,\dots,V_n\}$1

The intended effect is to isolate the impact of one visual factor from all others while still testing under controlled contextual variation (Liu et al., 26 Sep 2025).

The factor-isolation principle has methodological significance beyond this single benchmark. Earlier work on visual news bias surfaced bias through cluster inspection rather than numerical metrics, with human users comparing differing image clusters side by side (Narwal et al., 2017). RoboView-Bias instead turns controlled visual variation into an explicit benchmark substrate, making bias measurable at the level of embodied task outcomes rather than only through human inspection (Liu et al., 26 Sep 2025).

3. Perceptual-fairness validation protocol

Before robot evaluation, RoboView-Bias applies a two-stage perceptual-fairness validation pipeline to ensure that each generated image is unambiguous (Liu et al., 26 Sep 2025). The purpose of this stage is to ensure that any observed performance variation stems from the intended visual perturbation rather than from occlusion, ambiguity, or missing context (Liu et al., 26 Sep 2025).

The first stage is automated GPT-4o screening. A visual evaluator prompt rejects instances in which the object or its relation to a yellow box is unclear, and parameter adjustments are iterated until at least $V=\{V_1,\dots,V_n\}$2 of variants pass (Liu et al., 26 Sep 2025). The second stage is human adjudication: all passed instances are then reviewed by experts, again with a $V=\{V_1,\dots,V_n\}$3 pass requirement (Liu et al., 26 Sep 2025).

This protocol functions as a fairness filter in the perceptual sense rather than in the group-fairness sense used in much of the robot-learning fairness literature. In the broader survey literature, fairness criteria are often phrased as demographic parity, equalized odds, equal opportunity, or individual fairness (Londoño et al., 2022). RoboView-Bias instead validates that the benchmark inputs are perceptually fair with respect to the task definition: visually clear, solvable, and not confounded by unintended ambiguities (Liu et al., 26 Sep 2025).

A plausible implication is that the benchmark attempts to disentangle two distinct failure modes that are often conflated: failure due to poor perception under clear conditions, and failure due to ill-posed or ambiguous visual scenes. Only the former is meant to count as visual bias in the benchmark’s sense (Liu et al., 26 Sep 2025).

4. Evaluated agents and measurement framework

RoboView-Bias evaluates three representative embodied agents across two paradigms (Liu et al., 26 Sep 2025). The VLM-driven agents are SimpleAgent and MOKA. SimpleAgent is described as a minimal system coupling a single Vision–LLM, such as Qwen-VL or GPT-4o, directly to a heuristic action policy. MOKA is a modular pipeline that uses a VLM for planning plus Grounding DINO and SAM for 2D-to-3D affordance generation. The Vision–Language Action model is $V=\{V_1,\dots,V_n\}$4, described as a flow-matching (diffusion) policy conditioned on a large VLM backbone and trained on 10 k hours of cross-platform imitation data (Liu et al., 26 Sep 2025).

For each subspace $V=\{V_1,\dots,V_n\}$5, the benchmark runs 5 trials per instance and reports two main quantities (Liu et al., 26 Sep 2025). The first is the Average Success Rate,

$V=\{V_1,\dots,V_n\}$6

which aggregates task success over the subspace. The second is the Bias Coefficient (CV),

$V=\{V_1,\dots,V_n\}$7

which quantifies variability in success rates across the values of a visual factor under controlled contexts (Liu et al., 26 Sep 2025).

For color–viewpoint interactions, RoboView-Bias runs 10 trials per instance in a representative context $V=\{V_1,\dots,V_n\}$8 and computes the Interaction Effect Coefficient (IEC),

$V=\{V_1,\dots,V_n\}$9

This metric is designed to measure how the bias associated with one factor changes as another factor varies (Liu et al., 26 Sep 2025).

The introduction of explicit bias coefficients and interaction coefficients distinguishes RoboView-Bias from systems in which bias is identified only qualitatively. The earlier UnbiasedCrowd system, for example, did not define any numerical bias score or statistical metric; “bias” was surfaced via human inspection of differing image clusters (Narwal et al., 2017). RoboView-Bias instead operationalizes visual bias as measurable task-performance fluctuation under factor-isolated perturbations (Liu et al., 26 Sep 2025).

5. Empirical findings

The central empirical result is that all evaluated agents show large bias coefficients (Liu et al., 26 Sep 2025). Under Qwen-VL, SimpleAgent peaks at $D=\{D_1,\dots,D_m\}$0 overall CV, MOKA at $D=\{D_1,\dots,D_m\}$1, and $D=\{D_1,\dots,D_m\}$2 at $D=\{D_1,\dots,D_m\}$3 (Liu et al., 26 Sep 2025). These values support the benchmark’s claim that visual bias is substantial even in representative contemporary embodied-agent pipelines.

Camera viewpoint is reported as the single most critical factor (Liu et al., 26 Sep 2025). When only the full-orbit poses vary, $D=\{D_1,\dots,D_m\}$4 for all VLM-driven agents, and success rates bounce from near $D=\{D_1,\dots,D_m\}$5 to $D=\{D_1,\dots,D_m\}$6 across adjacent viewpoints (Liu et al., 26 Sep 2025). This finding places viewpoint variation at the center of the benchmark’s analysis and aligns with the broader concern, visible in view-robust manipulation work, that fixed-camera training creates substantial cross-view fragility (Liu et al., 2024, Gu et al., 23 Apr 2026).

Color bias is also pronounced (Liu et al., 26 Sep 2025). The benchmark reports that success on high-saturation reds and blues reaches approximately $D=\{D_1,\dots,D_m\}$7, while achromatic grays and whites hover near $D=\{D_1,\dots,D_m\}$8, indicating inherited color preferences from upstream VLMs (Liu et al., 26 Sep 2025). The benchmark interprets this as evidence that embodied-agent failures may reflect perceptual priors inherited from the underlying Vision–LLMs rather than only deficiencies in downstream control (Liu et al., 26 Sep 2025).

RoboView-Bias also reports strong asymmetric coupling between visual factors (Liu et al., 26 Sep 2025). On average,

$D=\{D_1,\dots,D_m\}$9

while

$V$0

The benchmark’s interpretation is that changing viewpoint amplifies color-induced variation far more than vice versa (Liu et al., 26 Sep 2025). This is not a claim of symmetric interaction; rather, it explicitly characterizes the coupling as strong and asymmetric (Liu et al., 26 Sep 2025).

These findings can be situated relative to recent view-robust manipulation research. RoboUniView argues that decoupling visual encoding from action learning via a unified 3D representation directly addresses view-bias by making downstream tasks “see” the same 3D grid, and reports stability under unseen camera parameters on CALVIN (Liu et al., 2024). VistaBot similarly targets limited robustness to camera viewpoint changes and introduces the View Generalization Score (VGS), reporting gains over ACT and $V$1 under novel views (Gu et al., 23 Apr 2026). RoboView-Bias does not evaluate those methods, but its results make the benchmark relevant as an instrument for testing claims of view robustness (Liu et al., 26 Sep 2025).

6. Mitigation, relation to de-biasing research, and implications

RoboView-Bias includes an initial mitigation study based on a Semantic Grounding Layer (SGL) (Liu et al., 26 Sep 2025). The module is inserted into MOKA as a pre-execution “semantic grounding” component with three steps: scene parsing and decomposition, in which a VLM extracts objects and action from the instruction $V$2 and the image; ambiguity detection, in which simple heuristics flag multiple “geometry” mentions; and instruction refinement, in which a refined instruction $V$3 is synthesized by prefixing the discriminative attribute, such as “small red cube” (Liu et al., 26 Sep 2025).

When MOKA is re-evaluated with SGL on the same benchmark, its bias coefficient is reduced by $V$4, computed as

$V$5

(Liu et al., 26 Sep 2025). Smaller improvements are reported for SimpleAgent and $V$6, and the benchmark suggests that the payoff may be greater in modular multi-stage pipelines (Liu et al., 26 Sep 2025).

This mitigation result belongs to a broader family of robot de-biasing strategies, but its intervention point is specific. In causal de-biasing for motion estimation, Xu et al. use inverse propensity weighting and doubly robust estimation to correct biased observational data in state-transition learning (Xu et al., 2022). In robotic assistance, FSPGRL reduces bias through a self-reflective reward-penalty mechanism triggered by an offline bias detector, reducing TotalIssueScore for both REINFORCE and PPO in a restaurant-service scenario (Zhu et al., 2023). In social robot navigation, a learning-to-relearning framework detects biased clusters and retrains with a bias penalty to reduce unjustified subgroup disparities (Hurtado et al., 2021). RoboView-Bias differs from these works in that it is primarily a benchmark for systematic quantification of visual bias in manipulation, not a general-purpose training algorithm (Liu et al., 26 Sep 2025).

The benchmark argues for routine factor-isolated bias benchmarking alongside standard generalization tests, for designing agents with explicit semantic-to-perceptual alignment as in SGL, and for extending benchmarks to lighting, material, and sim-to-real gaps (Liu et al., 26 Sep 2025). This suggests a research program in which measurement and mitigation are treated as complementary: systematic quantification identifies failure structure, while architectural or training interventions attempt to reduce the measured coefficients (Liu et al., 26 Sep 2025).

Within the wider literature on fairness and bias in robot learning, RoboView-Bias occupies the niche of perception-conditioned manipulation evaluation. The general survey literature emphasizes that robot-learning systems can inherit bias from data, models, deployment settings, and social context (Londoño et al., 2022). RoboView-Bias narrows that broad concern to a controlled, measurable question: how much task success changes when only visual factors change (Liu et al., 26 Sep 2025). Its main contribution is therefore methodological as much as empirical: it supplies a benchmark framework for exposing hidden failure modes tied to mundane visual variations before those variations are amplified across planning, grounding, and low-level control (Liu et al., 26 Sep 2025).