Adversarial Stress Testing of SPARK Humanoid Safety Filters

Abstract: Humanoid robots are difficult to deploy safely because they have high-dimensional bodies, many collision constraints, and must operate near people and obstacles. Safety filters help by modifying a nominal control action when it may violate collision-avoidance constraints. Still, nominal benchmark scores do not fully show how these filters behave in harder environments. In this work, we study the robustness of SPARK humanoid safety filters through replication and stress testing. We replicate the SPARK benchmark case G1SportMode_D1_WG_SO_v1 in MuJoCo and evaluate RSSA, RSSS, SSA, CBF, PFM, and SMA under controlled random seeds. We also built a post-processing pipeline that converts raw SPARK logs into goal-tracking, minimum-distance, and collision-step metrics. Our results show that some methods track the goal more closely, while others reduce collision steps more effectively. The stress tests further indicate that safety behavior can change under obstacle crowding, noisy distance estimates, and delayed obstacle information. These findings suggest that humanoid autonomy should be evaluated beyond nominal performance, using metrics that expose failure modes before deployment.

• The paper demonstrates that SPARK humanoid safety filters exhibit a trade-off between goal-tracking precision and collision avoidance.
• The study employs replicated benchmarks and adversarial scenarios including obstacle crowding, perception noise, and sensor latency attacks to extract safety and efficiency metrics.
• Results reveal that no single filter uniformly dominates, highlighting the need for multi-metric evaluations in dynamic, non-nominal conditions.

Adversarial Robustness of Humanoid Safety Filters in SPARK

Overview

This paper presents a systematic replication and adversarial stress-testing of the SPARK humanoid safety filter benchmark, with an emphasis on evaluating the robustness of several canonical safety filter algorithms under challenging sensing and environmental stressors. The authors instantiate the SPARK benchmark case D1 with the Unitree G1 humanoid in MuJoCo, comparing six safety filters (RSSA, RSSS, SSA, CBF, PFM, SMA) using a common experimental pipeline. The analysis comprises both nominal and adversarial conditions, including obstacle crowding, Gaussian perception noise injection, and sensor-latency attacks. To facilitate reproducibility, the authors developed a comprehensive post-processing workflow converting raw trajectory logs into meaningful safety and efficiency metrics.

Figure 1: MuJoCo execution view of the replicated SPARK G1 humanoid benchmark, visualizing the robot, obstacles, and goal structure.

Methodology

The evaluation focuses on six safety filter instantiations—RSSA, RSSS (variants of the Safe Set Algorithm), SSA (Safe Set Algorithm), CBF (Control Barrier Functions), PFM (Potential Field Method), and SMA (Sliding-Mode Algorithm)—applied uniformly within the SPARK G1 SportMode static-obstacle scenario. The experimental procedure is divided into three core phases:

1. Replication: Each method is run under fixed seeds and simulation settings, with reproducibility ensured by holding the task, simulation horizon, and random seeds constant.
2. Metric Extraction: Raw MuJoCo/SPARK npz logs are algorithmically parsed into easily comparable metrics: goal-arm distance, minimum robot–environment distance, and per-step collision indicators.
3. Adversarial Stress Testing: The environment is systematically altered along three adversarial axes: increased obstacle density, additive Gaussian noise in perception, and quantized sensor latency leading to stale observations.

Figure 2: Aggregate multi-seed comparison of safety filters, showing collision steps, goal-arm distance, and safety–performance trade-off.

Results and Claims

Baseline Filter Performance

Across nominal settings, a clear trade-off emerges between safety and goal-tracking accuracy:

• PFM offers the tightest goal-tracking but suffers by far the most collision steps.
• SMA yields the lowest average collision count but at a cost to precise goal attainment.
• SSA, RSSA, and RSSS provide a more moderate balance between performance and safety.
• CBF achieves low collision counts in moderate-crowding regimes, but does not consistently minimize both metrics.

There is no single filter that uniformly dominates; the results demonstrate an explicit need to measure both safety and efficacy, as optimizing one metric can significantly degrade the other.

Obstacle Crowding

Stress-testing with increased obstacle count (5, 15, 30) reveals dynamic shifts in filter behavior:

• With few obstacles (5), all algorithms can achieve zero collision, emphasizing the relative ease of the setting.
• At moderate (15) and high (30) densities, only specific filters maintain robustness; for instance, CBF minimizes collisions at 15 obstacles, while SMA surpasses all others at 30 obstacles.
• PFM exhibits acute degradation under dense environments, with rapidly escalating collision events.

  Figure 3: Time-series trace for the 15-obstacle stress test, highlighting when filters breach the collision boundary.

  

  Figure 4: Trade-off plot summarizing safety (collision steps) vs performance (mean arm-goal distance) at 15-obstacle density.

Perception Noise and Latency

Adversarial attacks on the observation pipeline further differentiate filter robustness:

• Perception noise attacks: PFM rapidly accumulates collisions and goal error, failing even at low noise intensity. SSA maintains relative safety at moderate intensities; CBF and SSS are stable.
• Latency attacks: All filters receive similar collision counts under latency, but PFM is unexpectedly most robust in this condition, whereas SSA degrades severely as latency increases.
• Diagnostic traces indicate infeasible filter responses manifest as repeated “No Solution” warnings, especially under high-latency attack.

  Figure 5: Degradation comparison under perception-noise attacks, emphasizing that higher noise leads to sharp safety-performance decay for specific filters.

  

  Figure 6: Degradation comparison under sensor-latency attacks, illustrating how filters diverge in their ability to maintain safety and task performance.

  

  Figure 7: Diagnostic metric evolution for SSA under high-latency, visualizing minimum robot–environment distances and goal tracking failures.

Implications and Future Directions

The study provides strong evidence that nominal benchmark performance is an insufficient indicator of overall safety and robustness for humanoid robot control policies. The observed contradictory strengths—e.g., PFM’s superior goal tracking against SMA’s collision avoidance—corroborate the need for multi-metric evaluation regimes. Importantly, filter robustness is non-monotonic under stress; the safest filter in a benign scenario may not retain that property if perception conditions worsen.

Stress-testing approaches of this type should become obligatory in safety evaluation pipelines since practical deployments will inevitably encounter degraded sensing, delayed information, and environmental crowding not seen in carefully controlled benchmarks.

Integrating adversarial scenario generation into the SPARK pipeline marks a substantial methodological step toward more comprehensive benchmarking for safe robot autonomy. The versatility of the post-processing and attack pipelines described in this work will likely facilitate further systematic exploration of safety filter behavior, especially if expanded to additional SPARK scenarios, more non-stationary threats, and larger random seed sets. Future theoretical work may formalize the observed trade-offs and inform both the design of hybrid filters and runtime safety monitors capable of dynamic adaptation to adversarial or non-nominal contexts.

Conclusion

Through careful replication, metric-driven analysis, and adversarial stress testing, the paper demonstrates that current safety filter algorithms for high-DOF humanoid robots exhibit distinct and sometimes contradictory patterns of robustness. Strong nominal scores do not guarantee resilience to perception corruption or complex multi-obstacle environments. Practitioners must therefore look beyond one-dimensional benchmarks and incorporate adversarial evaluation and stress testing to reliably assess the trustworthiness of autonomous humanoid control systems.