---
title: 'CATCH: Robotic Catching System'
url: https://www.emergentmind.com/topics/catch
type: topic
---

# CATCH: Robotic Catching System

Searching arXiv for the main target paper and closely related robotics work on catching.
{"query":"2409.10319 Catch It! Learning to Catch in Flight with Mobile Dexterous Hands", "max_results": 5}
{"query":"arXiv robot catching thrown objects dexterous hand reinforcement learning mobile manipulator", "max_results": 10}
CATCH denotes a robotic catching system for objects in flight developed in “Catch It! Learning to Catch in Flight with Mobile Dexterous Hands,” where a high-DoF mobile manipulator learns, entirely in simulation, to intercept and grasp thrown objects with zero-shot transfer to hardware [2409.10319]. The system couples an omni-directional mobile base, a 6-DoF arm, and a 12-DoF dexterous hand, and frames catching as a whole-body control problem under partial observability, short interaction horizons, uncertain trajectories, and substantial object diversity. Its central methodological contribution is a two-stage reinforcement learning framework that separates long-range interception from final grasp stabilization while preserving a unified deployment policy.

## 1. Task definition and system scope

The target task is catching thrown objects in mid-air from random initial positions and random velocities. The robot must move its mobile base to extend workspace, coordinate a 6-DoF arm to place the hand near the object trajectory, and control a dexterous hand to grasp at the correct time and orientation. The paper identifies four coupled sources of difficulty: the airborne window is short, at approximately \(2\) s; the action space is high-dimensional; whole-body motion must be agile and accurate; and object throws and shapes vary substantially, making trajectories uncertain [2409.10319].

The manipulation setting is explicitly beyond arm-only reach. The mobile platform is used to cover an approximately \(1.5\) m catch region, whereas the arm alone reaches approximately \(0.8\) m. This makes locomotion part of the catching policy rather than a separate positioning precondition. A further implication is that grasp synthesis cannot be isolated from interception: hand pose, palm orientation, base motion, and arm motion are all coupled during the terminal phase of the catch.

A common misunderstanding is to treat the problem as ballistic prediction followed by grasp execution. The formulation instead emphasizes continuous closed-loop whole-body control from partial observations. Another frequent misconception is that the “two-stage” design implies two controllers at run time. In fact, the paper states that the deployed policy is unified and outputs base, arm, and hand commands simultaneously; the two-stage distinction applies to training rather than execution.

## 2. Platform architecture, sensing, and state estimation

The physical system consists of a Ranger Mini V2 omni-directional mobile base, a 6-DoF UFACTORY XArm, and a 12-DoF LEAP Hand. The LEAP hardware has 16 mechanical DoFs, but 4 joints are fixed in the policy for efficiency. Perception and control run on a Thunderobot MIX mini-PC with an i7-13620H CPU and RTX 4060 GPU, powered through the base’s 48 V interface [2409.10319].

Object tracking is performed by an overhead Intel RealSense D455 RGB-D camera. The pipeline detects the object in RGB image space, retrieves depth to recover 3D coordinates in the camera frame, and applies eye-on-base calibration to transform the estimate into the arm-base frame. Proprioceptive inputs include the base’s 2D planar velocity, arm joint states used internally for forward kinematics, and hand joint positions for the catching-stage policy.

State estimation combines object position \(\mathbf{p}_t\), approximate object velocity from finite differences,
\[
\mathbf{v}_t \approx \mathbf{p}_t - \mathbf{p}_{t-1},
\]
end-effector position \(\mathbf{e}_t\), the palm \(z\)-axis \(\hat{\mathbf{u}}_t\), base velocities, and hand joint angles into a compact RL observation. The use of consecutive observations is deliberate: the policy input contains “two consecutive proprioception \(O^{t,t-1}\),” so motion information is encoded implicitly through differencing.

This sensing stack is visually driven and lacks tactile feedback. The paper therefore treats impact handling and object retention as learned consequences of coordinated hand closure, palm orientation, and small post-contact body adjustments rather than contact-rich feedback control.

## 3. Control formulation as a partially observable decision process

Both tracking and catching are formulated as a Partially Observable Markov Decision Process,
\[
\mathcal{M} = (S, O, A, R, P),
\]
where \(S\) includes the full object and robot state, \(O\) contains partial sensor observations, \(A\) is a continuous whole-body action, \(R\) is the reward, and \(P\) is the environment dynamics. The objective is to learn a parameterized policy \(\pi_\theta : O \to A\) maximizing
\[
J(\theta) = \mathbb{E}_{\pi_\theta}\left[\sum_{t=0}^{T} r\bigl(s_t, \pi_\theta(o_t)\bigr)\right].
\]
The POMDP abstraction is motivated by camera-based tracking: the object’s full state, including precise velocity and contact state, is only partially observed [2409.10319].

For both training stages, the observation includes two consecutive 3D object positions \(\mathbf{p}_{t-1}, \mathbf{p}_t\), two consecutive 3D end-effector positions \(\mathbf{e}_{t-1}, \mathbf{e}_t\), and base planar velocity \((v_x,v_y)\). Hand joint positions are added only in the catching stage. Actions are continuous and comprise base velocity commands \((v_x,v_y)\), arm end-effector position increments \(\Delta \mathbf{e}_t \in \mathbb{R}^3\), arm roll increment \(\Delta \phi_t^{\text{roll}} \in \mathbb{R}\), and, in the catching stage, 12-dimensional hand joint deltas \(\Delta \mathbf{q}^{\text{hand}}_t\).

The arm action space deliberately excludes yaw and pitch updates. The paper states that these destabilized training and often produced poor hand orientations. Roll is retained because palm orientation relative to the incoming trajectory is critical. This design choice materially constrains the policy class, but it also regularizes learning in a system with already high whole-body dimensionality.

## 4. Two-stage reinforcement learning and reward shaping

The learning framework decomposes catching into a tracking task and a catching task. In the first stage, the hand is fixed in a neutral configuration and the policy learns to move the base and arm so that the palm tracks the flying object. In the second stage, the base and arm policy is initialized from the first stage, hand control is added, and the policy is fine-tuned to grab and stabilize the object [2409.10319].

Both stages are trained with PPO in 64 parallel MuJoCo environments using 128 CPU cores and one RTX 3090 Ti. The simulation runs at 500 Hz and the policy at 25 Hz, matching real deployment timing with one action every 20 simulation steps. The paper argues that single-stage training from scratch is inefficient and unstable because the full action space is large and hand actions are irrelevant until the object is close; early exploration of hand motion therefore wastes samples and injects noise.

Reward shaping is built from several terms. Let \(d_t = \|\mathbf{e}_t - \mathbf{p}_t\|_2\). Then the position reward is
\[
r_t^{\text{pos}} = \|d_{t-1}\|_2 - \|\mathbf{e}_t - \mathbf{p}_t\|_2,
\]
the precision reward is
\[
r_t^{\text{pre}} = \exp\bigl(-50 \cdot \|d_t\|_2^2\bigr),
\]
and the orientation reward is
\[
r_t^{\text{orient}} = \operatorname{clamp}\bigl(\mathbf{v}_t \cdot \hat{\mathbf{u}}_t, -1, 1\bigr).
\]
The tracking stage also uses a binary touch reward, while the catching stage adds an object stability reward
\[
r_t^{\text{stab}} = \Delta t_{\text{grasp}},
\]
which measures stable holding duration. Smoothness and feasibility are enforced by
\[
r_t^{\text{ctrl}} = -\|\mathbf{a}_t\|_2^2
\]
and a joint-limit penalty
\[
r_t^{\text{cstr}} =
\begin{cases}
-1, & \text{if any joint exceeds its limit},\\
0, & \text{otherwise}.
\end{cases}
\]
The total reward is a weighted sum \(r_t^{\text{track/catch}} = \sum_k \lambda_k r_t^k\), with tuned but unpublished weights.

Two points are especially significant. First, the orientation term operationalizes the idea that successful catching requires the palm to face into the object’s motion, which is why roll is retained as a controlled DoF. Second, the staged curriculum aligns with the task structure: coarse long-range interception is learned before fine grasp closure and post-impact stabilization.

## 5. Simulation environment, domain randomization, and sim-to-real transfer

Training is performed in MuJoCo using robot models exported from URDF/MJCF via `sw2urdf`, with PID controllers tuned to approximate real base, arm, and hand dynamics. Arm motion is executed through inverse kinematics, with both Levenberg–Marquardt and QP-based IK implemented; the latter includes null-space motion to respect joint limits. The policy itself does not solve dynamics explicitly, but MuJoCo simulates them implicitly [2409.10319].

Domain randomization is extensive. Training uses five primitive shapes—box, sphere, ellipsoid, cylinder, and capsule—with randomized shape, size, mass, damping, and other physical properties. Throw initial positions, velocities, landing points, and timing are also randomized. The landing region extends to approximately \(1.5\) m from the robot start position, which enforces use of the mobile base. Evaluation in simulation additionally uses five unseen object categories: bowls, bottles, wine cups, cups, and bread-like shapes.

Sim-to-real transfer is zero-shot. The paper identifies four mechanisms for narrowing the gap. First, system identification tunes PID parameters to approximate real response. Second, domain randomization includes PID gains, slight gravity variation, throw timing, observation noise, and action noise. Third, a first-order low-pass filter is applied to base commands,
\[
v_t^{\text{filtered}} = \alpha v_{t-1}^{\text{filtered}} + (1-\alpha) v_t^{\text{raw}},
\]
with \(\alpha = 0.9\), because the real base cannot change steering instantaneously while moving forward. Fourth, real-world observations are expressed in the same coordinate conventions as those used in training.

This transfer strategy is notable for what it does not use. There is no on-robot fine-tuning, no explicit ballistic interception module, and no tactile sensing. The policy must therefore generalize directly from randomized simulation and from a perception stack that only approximates object state through visual tracking and finite-difference velocity estimates.

## 6. Empirical performance, failure modes, and research significance

In simulation, the paper compares one-stage training, two-stage training without arm roll, and the full two-stage method. On the tracking task, both two-stage variants converge quickly to high success, above approximately \(90\%\), whereas one-stage training is slower. On the catching task, the full two-stage method outperforms one-stage training in both convergence speed and final success, and adding arm roll improves catching relative to the no-roll variant [2409.10319].

On five unseen object classes over \(200 \times 64\) trials, the full two-stage method attains tracking success rates of \(92\pm3\) for bowls, \(90\pm4\) for bottles, \(88\pm3\) for wine cups, \(94\pm5\) for cups, and \(95\pm4\) for breads. Catching success rates are \(84\pm5\), \(78\pm6\), \(65\pm3\), \(80\pm4\), and \(80\pm3\), respectively. These results support the paper’s summary claim that simulation catching success is around \(80\%\) for many object types. By contrast, one-stage catching remains much lower, at \(22\pm2\) for bowls, \(10\pm3\) for bottles, \(6\pm2\) for wine cups, \(13\pm2\) for cups, and \(15\pm3\) for breads.

Real-world deployment uses four sandbag types—cube, sphere, cylinder, and irregular composite—with 40 trials per shape, split evenly between low-pass-filtered and unfiltered control. Without LPF, tracking success is 10, 10, 5, and 15 percent, and catching success is 0, 0, 0, and 5 percent. With LPF, tracking rises to 70, 65, 70, and 75 percent, while catching reaches 25, 25, 15, and 20 percent. The paper therefore distinguishes sharply between simulation and hardware performance: the real robot achieves robust tracking but only non-trivial, rather than high, catching success.

The reported failure modes are specific. Elastic sandbags can bounce off the palm at high relative speeds, and the RGB-D tracker is vulnerable to fast motion and occlusion by the hand near the decisive contact interval. The absence of tactile sensing further limits the system’s ability to reason about impact and slip. The experiments also do not test extreme throw speeds or very long distances, so operating limits in more aggressive regimes remain uncharacterized.

These results suggest a particular research significance. The paper provides evidence that whole-body catching with a mobile manipulator and dexterous hand is feasible when the task is structured as staged RL, reward shaping explicitly encodes interception and stabilization objectives, and domain randomization is paired with hardware-aware command smoothing. A plausible implication is that future progress on dynamic object interception will depend less on monolithic policy scaling than on better decomposition of subproblems, richer contact sensing, and tighter integration of perception, locomotion, and dexterous grasping.

Source: https://www.emergentmind.com/topics/catch