Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

Degraded behaviour in failed trajectories under perturbation

Determine whether input perturbations result in failed robot trajectories with meaningfully degraded behaviour in Vision-Language-Action model evaluations.

Background

The evaluation framework focuses on successful trajectories because it requires sufficient successful samples to compare behavioural metrics under baseline and perturbed conditions. Consequently, the effects of perturbations on failed trajectories were not analysed. The authors explicitly identify as a future-work question whether failures are associated with meaningful behavioural degradation, extending the framework beyond its current success-conditioned analysis.

References

Interesting questions for future work, include whether perturbations result in failed trajectories with meaningfully degraded behaviour.

— Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks  (2610.01351 - Higham et al., 1 Oct 2026) in Section LIMITATIONS AND FUTURE WORK

Since all perturbed predictions are evaluated against the original demonstration, our results show that language influences the action, so closed-loop rollouts or counterfactual targets for meaning-changing perturbations would test whether the observed sensitivity reflects appropriate use of the instruction.