GEM: Generalizable Entropy-based Manipulation

• The paper demonstrates that GEM reduces entropy in both observations and actions using a probabilistic Gaussian mixture model to enable robust imitation learning with minimal demonstrations.
• GEM is a dynamic manipulation system featuring object-centric geometry perception, visual servo tracking, and recurrent visual memory to manage fast-moving, contact-rich tasks.
• Empirical results highlight GEM's effectiveness with high in-domain and cross-domain success rates, including a real-world canteen deployment without in-scene demonstrations.

Searching arXiv for the specified GEM paper and closely related acronym usages to ground the article. arXiv search query: (Li et al., 19 Aug 2025) OR "Generalizable Entropy-based Manipulation" OR GEM manipulation Generalizable Entropy-based Manipulation (GEM) is a dynamic object manipulation system that studies whether strong generalization in imitation learning can be achieved with only a few demonstrations by reducing the entropy of the learning problem itself. In this formulation, robotic manipulation is analyzed through the joint entropy of successful observations and actions, and GEM is designed to reduce both observation complexity and action ambiguity while preserving successful behavior. The system targets moving-object manipulation rather than static tabletop interaction, and combines object-centric geometry perception, visual servo tracking, recurrent visual memory, and probabilistic action prediction. It is evaluated on four dynamic manipulation primitives—Pick, Put, Rotate, and Insert—and is reported to generalize across diverse environment backgrounds, robot embodiments, motion dynamics, object geometries, sim-to-real transfer, and a real canteen deployment without in-scene demonstration (Li et al., 19 Aug 2025).

1. Problem formulation and scope

GEM is introduced for dynamic object manipulation, where a robot must manipulate objects that move, typically on a conveyor belt. The paper emphasizes four manipulation primitives: Pick, Put/Place, Rotate, and Insert. The central difficulty is the simultaneous requirement of time efficiency and precision: the robot must rapidly synchronize with object motion while still executing contact-rich interactions accurately, especially for insertion and reorientation (Li et al., 19 Aug 2025).

The method is motivated by the claim that conventional imitation learning treats policy learning as approximation of a high-entropy observation-action distribution. In deployment, this distribution shifts along several axes: environment and background appearance, robot embodiment, object geometry, motion trajectory, and sim-to-real transfer. The paper explicitly targets generalization across environment/background changes, robot embodiments, object geometries, motion dynamics and trajectories, and real-world environments, including an unseen canteen scene (Li et al., 19 Aug 2025).

Robotic manipulation is modeled as an MDP

$$\mathcal{E} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{T}, s_0),$$

and for a task $\tau \in \mathcal{T}$ the successful trajectory set $\mathcal{C}$ consists of trajectories

$$\{s_0, a_0, \cdots, s_{T-1}, a_{T-1}, s_T\}, \quad s_T=\iota.$$

The learning target is framed as minimizing

$$\min\limits_{\pi} D_{KL}(p_{\mathcal{C}(s_{0:T}, a_{0:T-1})} \ || \ p_{\pi}(s_{0:T}, a_{0:T-1})),$$

with policy-induced trajectory distribution

$$p_{\pi}(s_{0:T}, a_{0:T-1}) = p(s_0) \prod\limits_{t=0}^{T-1} \pi(a_t | s_t) P(s_{t+1} | s_t, a_t).$$

The paper then moves to the imitation-learning setting, where the policy observes $o_t=f(s_t)$ rather than the full state, and optimizes maximum likelihood over demonstrations (Li et al., 19 Aug 2025).

2. Entropy-based formulation

The paper’s central conceptual contribution is an entropy decomposition over successful observation-action data:

$$\mathcal{H}(\mathcal{O}_\mathcal{C}, \mathcal{A}_\mathcal{C}) = \mathcal{H}(\mathcal{O}_\mathcal{C}) + \mathcal{H}(\mathcal{A}_\mathcal{C} | \mathcal{O}_\mathcal{C}). \tag{5}$$

Here, $\mathcal{H}(\mathcal{O}_\mathcal{C})$ is observation complexity, and $$\mathcal{H}(\mathcal{A}_\mathcal{C} | \mathcal{O}_\mathcal{C})$$ is action prediction ambiguity. The stated thesis is that higher entropy implies greater demonstration demand, so data efficiency can be improved by lowering either term without discarding task-relevant information or reducing the probability of successful action selection (Li et al., 19 Aug 2025).

This leads to two design principles. First, the system should remove redundant observation information irrelevant to inferring accurate actions. Second, it should reduce action ambiguity through better temporal context and better action distribution modeling. The observation-complexity analysis groups nuisance factors into environment information, robot proprioception, sensor configuration, and task instruction. This motivates object-only geometric observations, embodiment-agnostic end-effector control, and offloading motion tracking to classical control rather than learning it jointly with contact behavior (Li et al., 19 Aug 2025).

For action ambiguity, the paper argues that current observation plus joint state may be insufficient because identical instantaneous inputs can correspond to different phases of a manipulation sequence. It explicitly criticizes memory designs based on historical action trajectories because they can induce shortcut learning, and instead adopts recurrent memory over observation features. It also argues that direct regression with L1-like losses implicitly estimates a unimodal action distribution and can average incompatible action modes. GEM therefore represents the policy as a Gaussian mixture:

$\tau \in \mathcal{T}$0

with $\tau \in \mathcal{T}$1 and $\tau \in \mathcal{T}$2 (Li et al., 19 Aug 2025).

The paper does not present formal sample-complexity theorems. Instead, the entropy framework is validated empirically with a toy navigation task showing that increasing irrelevant observation entropy or increasing action ambiguity worsens approximation under fixed data, whereas enough demonstrations can eventually recover the expert distribution even under high-entropy conditions. This suggests that GEM’s contribution is primarily architectural and systems-oriented rather than theorem-driven (Li et al., 19 Aug 2025).

3. System architecture

GEM comprises two main blocks: object-centric geometry perception and hybrid action control. The perception stack segments task-relevant objects, removes background points, converts RGB-D inputs to 3D point clouds, and assigns distinct instance IDs to points from different objects. The downstream policy thus consumes uncolored 3D points with instance IDs rather than RGB images, suppressing appearance variation and background clutter (Li et al., 19 Aug 2025).

The control stack separates dynamic tracking from contact interaction. Using the external-view point cloud

$\tau \in \mathcal{T}$3

the target object top centroid is estimated as

$\tau \in \mathcal{T}$4

Velocity $\tau \in \mathcal{T}$5 is estimated via Gaussian process regression from centroid history, and predicted from history under occlusion. Visual servo control then maintains a preset offset $\tau \in \mathcal{T}$6 and orientation $\tau \in \mathcal{T}$7 relative to the moving object, so the end-effector tracks the object before manipulation (Li et al., 19 Aug 2025).

The learned policy operates on the wrist-view point cloud

$\tau \in \mathcal{T}$8

and predicts end-effector manipulation offsets $\tau \in \mathcal{T}$9, orientation offsets $\mathcal{C}$0, and a finish flag. The final end-effector command is the sum of tracking and manipulation terms:

$\mathcal{C}$1

for position, and

$\mathcal{C}$2

for orientation, with velocity $\mathcal{C}$3. Robot-specific joint motion is then obtained through inverse kinematics, which is the basis for embodiment transfer (Li et al., 19 Aug 2025).

The recurrent policy network uses Sonata as point feature extractor, with instance IDs used as surrogate color information. Point features are encoded as

$\mathcal{C}$4

The previous memory state is

$\mathcal{C}$5

and is updated through a Transformer-decoder-based memory encoder:

$\mathcal{C}$6

A learnable skill token

$\mathcal{C}$7

conditions the policy on Pick, Place, Rotate, or Insert. Three Transformer decoders and an ACT-style chunking mechanism then predict the parameters of the Gaussian mixture policy and the finish signal. At test time, the action with maximum probability is selected (Li et al., 19 Aug 2025).

4. Data, training, and implementation

The training corpus consists of 6,600 demonstrations: 600 Pick demonstrations collected by human teleoperation in simulation, with 30 object assets and 20 demonstrations per asset, plus 2,000 Put, 2,000 Rotate, and 2,000 Insert demonstrations generated automatically in simulation. The simulation environment is built in Isaac Gym with 30 representative object geometries from the Google Scanned Objects dataset (Li et al., 19 Aug 2025).

For real-world lab transfer, the system is fine-tuned with 20 demonstrations per skill, for 80 demonstrations total. For the canteen deployment, the policy is fine-tuned with 100 Pick demonstrations collected on the lab platform rather than in the canteen itself. This point is important because the paper’s claim of deployment “without in-scene demonstration” refers to absence of target-environment collection, not absence of real-world finetuning altogether (Li et al., 19 Aug 2025).

Training uses AdamW, learning rate $\mathcal{C}$8, batch size 32, and 10K iterations on a single RTX 4090, with total training time of about 14 hours. Additional implementation settings include image resolution $\mathcal{C}$9, action chunk size $$\{s_0, a_0, \cdots, s_{T-1}, a_{T-1}, s_T\}, \quad s_T=\iota.$$0, memory token number $$\{s_0, a_0, \cdots, s_{T-1}, a_{T-1}, s_T\}, \quad s_T=\iota.$$1, feature dimension $$\{s_0, a_0, \cdots, s_{T-1}, a_{T-1}, s_T\}, \quad s_T=\iota.$$2, and robot control frequency 20 Hz. The policy is trained by likelihood maximization under the GMM output distribution (Li et al., 19 Aug 2025).

The paper presents this training setup as a data-efficiency strategy rather than a pure scaling strategy. This suggests that the main reduction in demonstration demand comes from narrowing what the learned policy must model: local contact geometry and skill-specific offsets, while delegating tracking and embodiment-specific kinematics to other components (Li et al., 19 Aug 2025).

5. Empirical results and generalization behavior

In simulation, GEM reports in-domain success rates of 0.88 for Pick, 0.98 for Put, 0.83 for Rotate, and 0.80 for Insert, with Avg_in 0.87. Out-of-domain, it reports 0.77, 0.91, 0.70, and 0.72, with Avg_out 0.78 and overall Avg 0.83, at 23.81 Hz. In real-world lab evaluation after finetuning with 80 real demonstrations, the reported success rates are 0.92 for Pick, 0.96 for Put, 0.79 for Rotate, and 0.74 for Insert, with Avg 0.85 (Li et al., 19 Aug 2025).

The paper also reports nontrivial sim-to-real transfer without real tuning: 0.76 for Pick, 0.83 for Put, 0.55 for Rotate, and 0.48 for Insert. Background generalization is reported as essentially unchanged between seen and unseen conveyor backgrounds. Motion-pattern generalization shows only moderate degradation between linear and changed trajectories. Real-world seen versus unseen objects yield 0.85 average versus 0.74 average, with unseen-object task breakdown of 0.80 Pick, 0.90 Put, 0.67 Rotate, and 0.59 Insert (Li et al., 19 Aug 2025).

Embodiment transfer is one of the method’s strongest claims. In simulation, training on Franka and validating on UR10e gives 0.91 / 0.97 / 0.85 / 0.79 across the four tasks, close to same-embodiment Franka validation. In real-world experiments, training on Piper and validating on ARX5 gives 0.90 / 0.95 / 0.77 / 0.69, again close to same-embodiment performance. The paper attributes this to end-effector-level action prediction followed by inverse kinematics (Li et al., 19 Aug 2025).

The largest practical result is the canteen deployment: 97.2% success rate, over seven consecutive days, across more than 10,000 spoon-collection operations, with no demonstrations collected in the canteen itself. The paper notes difficult cases such as food residue, tightly spaced spoons, stacked spoons, and high conveyor speed. At 0.25 m/s the system still works with reduced success, while at 0.50 m/s it fails because the robot cannot keep up (Li et al., 19 Aug 2025).

Ablations align closely with the entropy rationale. Replacing point inputs with images degrades generalization markedly, especially real-world out-of-domain performance. Removing visual servo control causes success to drop to 0.35 / 0.28 / 0.20 / 0.16 across the four simulation-real summary columns reported in Table 12, while the full system yields 0.87 / 0.78 / 0.85 / 0.74. Removing memory lowers success to 0.72 / 0.60 / 0.69 / 0.55. Replacing probabilistic output with direct regression lowers success to 0.81 / 0.73 / 0.79 / 0.67, and in a special two-target ambiguity experiment direct prediction hovers at the midpoint with nearly 0% success, whereas probabilistic prediction reaches 87% success. These results are presented as evidence that entropy reduction in both observation and action spaces is the operative mechanism (Li et al., 19 Aug 2025).

6. Interpretation, limitations, and acronym ambiguity

GEM is specifically Generalizable Entropy-based Manipulation in the dynamic-manipulation sense of train-time entropy reduction in imitation learning. The paper’s limitations are mostly architectural and operational. It addresses only four manipulation primitives; it depends on accurate instance segmentation; its effectiveness is bounded by tracking and hardware speed limits; and its entropy framework remains conceptual rather than a formal quantitative theory of sample complexity. The method also still relies on some human teleoperation for grasp demonstrations and on representative simulation assets (Li et al., 19 Aug 2025).

Within the broader literature, the acronym “GEM” is ambiguous. In deep reinforcement learning, GEM also denotes Generalizable Episodic Memory, a memory-augmented value-learning method for continuous control (Hu et al., 2021). In exploration for reinforcement learning, GEM denotes Geometric Entropy Maximisation, a geometry-aware state-visitation entropy method (Guo et al., 2021). In one-shot rearrangement, GEM refers to Graph-based Equivalence Mappings, a reward-learning framework for generalizable spatial goals (Netanyahu et al., 2022). These are distinct from the dynamic-manipulation system discussed here.

Adjacent work on generalizable manipulation is also conceptually related but technically separate. For example, Gondola addresses grounded vision-language planning for generalizable robotic manipulation and improves results on GemBench, but it does not develop an entropy-based control or imitation-learning framework (Chen et al., 12 Jun 2025). This contrast is useful because it places GEM on the side of low-level dynamic manipulation and execution, rather than high-level language-conditioned planning.

The broader implication is that GEM’s contribution lies in recasting dynamic imitation learning as a systems problem of entropy reduction: narrowing observations to task-relevant geometry, narrowing action learning to multimodal local interaction, and isolating motion tracking in classical control. This suggests a design principle for generalizable manipulation systems: stronger transfer can emerge not only from larger datasets or models, but from structurally lowering the entropy of the learned problem itself (Li et al., 19 Aug 2025).