Ag2x2: Robust Agent-Agnostic Visual Representations

• Ag2x2 is a two-stage framework for zero-shot bimanual manipulation that learns from human videos by preserving essential hand-position data for coordination.
• It modifies traditional visual representations by integrating inpainted images with explicit 2D hand coordinates using a vision transformer and MLP encoders.
• Experimental results on 13 tasks show a 73.5% success rate, outperforming prior baselines and highlighting the benefits of retaining hand-aware cues.

Searching arXiv for the Ag2x2 paper and closely related baseline/benchmark papers to ground the article. arxiv_search(query="Ag2x2 Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation", max_results=5) arxiv_search(query="Ag2Manip agent-agnostic visual representations manipulation arXiv", max_results=10) arxiv_search(query="R3M visual representations robot manipulation arXiv VIP curious representation learning reward robot arXiv Eureka reward design arXiv Bi-DexHands PerAct2 arXiv", max_results=10) Ag2x2 is a two-stage framework for zero-shot bimanual robot manipulation that learns from human videos rather than expert robot demonstrations or hand-designed rewards. Its defining idea is a coordination-aware, agent-agnostic visual representation that jointly encodes the state of manipulated objects and the motion pattern of two hands, while preserving only the hand-position information needed for coordination and removing human appearance and morphology. In the literature represented here, “Ag2x2” refers to the robotics framework introduced in “Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation” (Xiong et al., 26 Jul 2025). By contrast, related silver-based chemistry papers discuss ${\rm Ag_2}M_2{\rm TeO_6}$ and Cu[Ag(SO$_4$)$_2$], but do not define a compound literally named “Ag2x2” (Masese et al., 2021, Domanski et al., 2023).

1. Terminological scope and problem setting

Ag2x2 addresses bimanual manipulation, i.e., tasks requiring two hands working together in a coordinated way. The task set discussed for the framework includes opening and closing doors, placing objects into containers, pushing a box with two hands, pressing buttons with coordinated motions, and straightening a rope (Xiong et al., 26 Jul 2025).

The “zero-shot” setting is defined narrowly. The robot is not trained from task-specific robot demonstrations or manually engineered task rewards. Instead, it relies on a representation pre-trained on human videos and uses that representation to solve new robot tasks directly. The stated motivation is that bimanual tasks require more than object-state recognition: they also depend on coordination-sensitive signals such as where each hand is, how the hands move relative to each other, when one hand stabilizes while the other manipulates, and how object-state changes depend on both hands (Xiong et al., 26 Jul 2025).

A central terminological point is the paper’s use of “agent-agnostic.” In Ag2x2, agent-agnosticism does not mean discarding all information about the acting body. Prior object-centric masking strategies can remove exactly the cues that matter for two-handed coordination. Ag2x2 therefore preserves 2D hand positions while removing the human body image, retaining coordination structure without tying the representation to human-specific appearance or kinematics. This distinguishes the framework from single-arm or purely object-centric transfer methods (Xiong et al., 26 Jul 2025).

2. Design rationale and relation to earlier representation-learning approaches

The framework is motivated by three limitations attributed to prior approaches. First, imitation learning requires large numbers of demonstrations, while RL systems for bimanual manipulation, including benchmarked systems such as Bi-DexHands, often depend on carefully engineered rewards. Second, single-arm representation methods do not encode the spatial relationship between two hands, and body-masking approaches can eliminate the hand cues required for coordination. Third, transfer from human motion to robot execution is obstructed by the embodiment gap: limb proportions, appearance, and kinematics differ substantially across humans and robots (Xiong et al., 26 Jul 2025).

Ag2x2 is presented as a modification of the agent-agnostic idea introduced in Ag2Manip. In the baseline formulation, human regions are masked out and a time-contrastive embedding is learned from videos. The training data are written as

$$\mathcal{D}=\left\{v^{c}:=\left(o_{1}^{c}, o_{2}^{c}, \ldots, o_{n_c}^{c}\right)\right\}_{c=1}^{N},$$

and the visual encoder is

$$\mathcal{F}_{\phi}: \mathbb{R}^{H \times W \times 3} \rightarrow \mathbb{R}^{K}.$$

The original objective combines a time-contrastive loss $\mathcal{L}_{\mathrm{tcn}}$ with a regularizer $\mathcal{L}_{\mathrm{reg}}$, where

$$\mathcal{L}_{\mathrm{reg}} = \lVert \mathcal{F}_{\phi}(o)\rVert_1 + \lVert \mathcal{F}_{\phi}(o)\rVert_2.$$

The limitation, as stated explicitly, is that removing all human information prevents the representation from capturing hand-hand coordination (Xiong et al., 26 Jul 2025).

This framing also clarifies the paper’s relation to other baselines. R3M and VIP are described as strong visual representations, but not designed specifically for bimanual coordination. Ag2Manip is described as effective for single-arm manipulation because it learns object-centric agent-agnostic features, but its full removal of human presence is detrimental when hand placement itself is task-relevant. Eureka is included as a reward-design baseline, and expert reward shaping is treated as a strong but non-scalable reference point (Xiong et al., 26 Jul 2025).

3. Representation learning in Ag2x2

Ag2x2 modifies the baseline representation by augmenting the inpainted image with the 2D coordinates of both hands. The encoder becomes

$$\mathcal{F}_{\phi}: \mathbb{R}^{H\times W\times 3 + 4}\rightarrow \mathbb{R}^{K},$$

where the additional four inputs are $(x_L, y_L)$ for the left hand and $_4$0 for the right hand (Xiong et al., 26 Jul 2025).

This is the framework’s core representational decision. The human body is removed from the image, but the hands remain encoded as positions. The intended effect is to preserve information about where the hands are, how they approach objects, and whether their motions are coordinated, while abstracting away human appearance and morphology. The paper characterizes this as maintaining agent-agnosticism while adding coordination awareness (Xiong et al., 26 Jul 2025).

The hand coordinates are encoded using a two-layer MLP with hidden sizes $_4$1 and ReLU activations. These hand tokens are concatenated with image tokens in a Vision Transformer. The visual backbone is ViT-Large pre-trained on ImageNet-21k and adapted using LoRA on the query and value matrices (Xiong et al., 26 Jul 2025).

The hand-aware dataset $_4$2 is derived from EPIC-KITCHENS human videos. The stated preprocessing pipeline is:

1. detect hands using HaMeR with a YOLO-v5 backbone,
2. estimate 3D keypoints and project them into 2D,
3. compute each hand’s position as the mean of its 21 keypoints,
4. segment humans using ODISE,
5. inpaint the human regions using E$_4$3FGVI,
6. combine the inpainted image with hand coordinates.

The result is a training corpus in which the scene is visually agent-agnostic but still hand-aware. A plausible implication is that the method treats coordination as a low-dimensional geometric signal rather than an appearance-based one, which is consistent with its intended human-to-robot transfer regime (Xiong et al., 26 Jul 2025).

4. Policy learning, goal specification, and reward construction

After representation pretraining, Ag2x2 uses the learned embedding as a reward signal for autonomous policy learning. The policy-learning stage is formulated around two proxy agents representing the two end-effectors. These proxies are abstract action primitives that are later converted into actual robot joint commands using inverse kinematics (Xiong et al., 26 Jul 2025).

The policy $_4$4 maps proxy states $_4$5 and environment states $_4$6 to actions

$_4$7

where $_4$8 are target positions for both proxies and $_4$9 are intended interaction forces. A PD controller converts these targets into movements (Xiong et al., 26 Jul 2025).

Goal specification is likewise coordination-aware. The goal is defined as

$_2$0

so it includes both a goal image and desired 2D end-effector positions. This means that the target is not only a visual object configuration but also a desired hand arrangement (Xiong et al., 26 Jul 2025).

The reward is defined through similarity between the current embedding and the goal embedding: $_2$1 where $_2$2 and $_2$3 are the embeddings of the current observation and goal, $_2$4 is their similarity, $_2$5 is the initial similarity at the start of the episode, and $_2$6 is a reward scaling factor. The paper uses

$_2$7

The stated intuition is that the reward remains modest near the initial state and becomes steeper once the policy exceeds the starting similarity baseline, thereby encouraging movement beyond small local improvements (Xiong et al., 26 Jul 2025).

5. Experimental regime and empirical results

Ag2x2 is evaluated on 13 bimanual tasks drawn from two benchmarks: 6 tasks from Bi-DexHands and 7 tasks from PerAct$_2$8. The Bi-DexHands tasks are close door outward, close door inward, open pen cap, lift pot, swing cup, and close scissors. The PerAct$_2$9 tasks are push box, put cube into drawer, put cube into microwave, lift tray, press buttons, sweep dirt, and straighten rope (Xiong et al., 26 Jul 2025).

The experimental platform uses two 9-DoF Franka Emika arms with grippers in NVIDIA IsaacGym. Training is performed with 68 parallel environments and 200 episodes per task. Evaluation uses 3 random seeds and 3 camera viewpoints, for a total of 9 evaluation runs per task. A task is counted as successful if the object reaches the goal configuration within tolerance (Xiong et al., 26 Jul 2025).

The headline result is an overall success rate of $$\mathcal{D}=\left\{v^{c}:=\left(o_{1}^{c}, o_{2}^{c}, \ldots, o_{n_c}^{c}\right)\right\}_{c=1}^{N},$$0 across the 13 tasks. The reported comparison is as follows (Xiong et al., 26 Jul 2025):

Method	Overall success
Eureka	15.4%
R3M	15.4%
VIP	26.5%
Ag2Manip	56.4%
Expert reward	63.2%
Ag2x2-H (no hand features)	57.3%
Ag2x2	73.5%

These numbers support the paper’s main empirical claim: preserving hand-position information in pretraining materially improves bimanual transfer. The ablation Ag2x2-H, which omits hand features, performs near Ag2Manip and well below the full method, reinforcing the claim that hand-position awareness, rather than generic extra input capacity, drives the gain (Xiong et al., 26 Jul 2025).

The paper also emphasizes performance on the deformable-object task straighten rope. This is highlighted because deformable manipulation is difficult to capture through simple object-pose rewards, and the task requires continuous coordination of both hands over a changing geometry. The reported success on this task is used as evidence that the representation is not restricted to rigid-body settings (Xiong et al., 26 Jul 2025).

6. Auxiliary analyses, downstream imitation learning, and stated limitations

Beyond aggregate success rates, Ag2x2 is evaluated through trajectory smoothness and progress consistency. Trajectory smoothness is measured by cumulative end-effector acceleration,

$$\mathcal{D}=\left\{v^{c}:=\left(o_{1}^{c}, o_{2}^{c}, \ldots, o_{n_c}^{c}\right)\right\}_{c=1}^{N},$$1

where $$\mathcal{D}=\left\{v^{c}:=\left(o_{1}^{c}, o_{2}^{c}, \ldots, o_{n_c}^{c}\right)\right\}_{c=1}^{N},$$2 and $$\mathcal{D}=\left\{v^{c}:=\left(o_{1}^{c}, o_{2}^{c}, \ldots, o_{n_c}^{c}\right)\right\}_{c=1}^{N},$$3 are left and right end-effector accelerations. Lower values indicate smoother motion. The paper reports that Ag2x2 is comparable to expert rewards and better than most baselines. Progress consistency is measured by Spearman rank correlation between time and similarity to the goal state, and Ag2x2 performs best or near-best overall (Xiong et al., 26 Jul 2025).

The representation is also used for imitation learning. The reported protocol collects 12 expert trajectories for rope straightening using Ag2x2, then trains an imitation policy that generalizes to unseen rope configurations. This demonstrates a secondary use of the framework: not only direct RL-based skill acquisition, but also autonomous generation of demonstrations for downstream imitation learning (Xiong et al., 26 Jul 2025).

The paper states several limitations. Goals are represented as a single final image plus end-effector positions, so tasks requiring intermediate states are difficult. Tasks involving unobservable internal dynamics remain hard. Only end-effector positions are modeled, not fully dexterous hands with articulated fingers. Multi-stage tasks are excluded from the benchmark selection because the framework does not yet model intermediate goal sequences. Failure modes include object interference in container tasks, imperfect hand coordination, and reward ambiguity in visually dominant scenes (Xiong et al., 26 Jul 2025).

These limitations also delimit the meaning of “agent-agnostic” in the framework. Ag2x2 does not attempt full embodiment invariance across arbitrary manipulators or task structures. Instead, it isolates a specific transferable substrate—object progression plus coarse bimanual geometry—and shows that this substrate is sufficient to outperform both prior visual-representation baselines and policies trained with expert-engineered rewards on the reported 13-task benchmark suite (Xiong et al., 26 Jul 2025).