PerAct: Transformer for Robotics

• PerAct is a language-conditioned multi-task robotic manipulation agent that uses a Perceiver Transformer to predict discretized 3D voxel actions.
• It employs a structured 3D voxelized observation and action space to address challenges posed by limited and expensive manipulation data.
• Experiments demonstrate its effectiveness by outperforming unstructured image-to-action agents across 18 RLBench and 7 real-world tasks.

PerAct, short for Perceiver-Actor, is a language-conditioned, multi-task behavior-cloning agent for 6-DoF robotic manipulation that encodes RGB-D voxel observations and language goals with a Perceiver Transformer, then predicts discretized actions by “detecting the next best voxel action” (Shridhar et al., 2022). It was introduced to test whether transformer-based formulations can remain effective in robotic manipulation despite the limited and expensive nature of manipulation data, and it does so by imposing a structured 3D voxelized observation and action space rather than learning a direct unstructured image-to-action mapping (Shridhar et al., 2022). In the reported experiments, a single model is trained for 18 RLBench tasks with 249 variations and 7 real-world tasks with 18 variations from only a few demonstrations per task, and it is reported to significantly outperform both unstructured image-to-action agents and 3D ConvNet baselines across a broad set of tabletop tasks (Shridhar et al., 2022).

1. Origin and problem formulation

PerAct was introduced in “Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation” (Shridhar et al., 2022) as a response to a central tension in robot learning: transformers have scaled effectively in vision and natural language processing, but robotic manipulation typically lacks the large, cheap datasets that support such scaling elsewhere. The method therefore does not simply transplant a vision transformer into manipulation