EgoAVFlow: Active Vision Robot Learning

• EgoAVFlow is a robot learning framework that leverages egocentric RGB-D videos and a shared 3D flow representation to simultaneously control manipulation and camera viewpoints.
• It couples three diffusion models to predict future robot actions, scene motions, and camera poses, using reward-maximizing denoising for visibility-aware planning.
• Empirical results show that EgoAVFlow achieves higher success rates and improved visibility maintenance compared to methods that mimic human viewpoint trajectories.

Searching arXiv for the primary paper and closely related work on EgoAVFlow. EgoAVFlow is a robot learning framework that takes only egocentric human RGB-D videos as input and learns a policy that both manipulates objects and actively moves a robot-mounted camera to keep task-relevant entities visible. Its defining mechanism is a shared 3D flow representation that serves as an embodiment-agnostic interface between human demonstrations, robot actions, future scene motion, and camera viewpoint planning. The framework couples three diffusion models—a robot policy, a future 3D flow predictor, and a view policy—and refines camera trajectories at test time with reward-maximizing denoising under a visibility-aware reward computed from predicted motion and scene geometry, all without robot demonstrations (Cho et al., 25 Feb 2026).

1. Problem formulation and scope

EgoAVFlow addresses the problem of learning robot manipulation from egocentric human videos while avoiding direct imitation of human viewpoint trajectories. The central premise is that human viewpoint imitation is not ideal for robots: humans exploit eye movements, vestibulo-ocular reflex, and head/body motion, whereas a robot-mounted camera operates under different kinematic and embodiment constraints. As a result, blindly imitating human egocentric camera motion often yields occlusions or awkward poses rather than task-critical visibility (Cho et al., 25 Feb 2026).

The framework therefore treats active vision as a first-class control problem. It seeks to learn a robot manipulation policy from egocentric human videos such that the robot not only executes the task but also actively controls the camera viewpoint to maintain visibility of objects and hands. This emphasis on visibility maintenance distinguishes EgoAVFlow from methods that learn only action generation or that assume fixed or passively observed viewpoints. A plausible implication is that the system is designed for manipulation regimes in which observability is itself a bottleneck, not merely an auxiliary concern.

The method uses 3D flow as the shared latent interface across human demonstrations, robot end-effector actions, future scene dynamics, and camera control. This design is explicitly intended to decouple the learned policy from exact human camera paths at test time. Human egocentric motion is retained as a prior through diffusion modeling, but the deployed system is allowed to deviate from that prior whenever visibility-aware planning indicates that another camera trajectory is better suited to the robot embodiment (Cho et al., 25 Feb 2026).

2. Shared 3D flow representation

In EgoAVFlow, “3D flow” is a per-point 3D motion field derived from tracked pixels plus depth, expressed in a consistent 3D coordinate frame. The processing pipeline begins with 2D point tracking using CoTracker3 over $N$ query pixels across frames, producing trajectories $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$, where the first two channels are image-plane coordinates and the third is a binary tracking indicator. These tracked pixels are then unprojected using depth maps and camera intrinsics to obtain 3D positions $\mathbf{X}_{t,i} \in \mathbb{R}^3$, assembled into

$\mathbf{F}_t \in \mathbb{R}^{N \times 4},$

whose first three channels are 3D point coordinates and whose fourth channel is the tracking indicator (Cho et al., 25 Feb 2026).

A key technical step is global reference frame alignment. Egocentric videos have a moving camera and arbitrary SLAM frames, so the framework runs DROID-SLAM to estimate camera poses $\mathbf{v}_t \in \mathrm{SE}(3)$ and introduces a marker frame based on a ChArUco board. The 3D tracks $\mathbf{F}_t$, camera poses $\mathbf{v}_t$, and human-hand-derived “robot” proprioception $p_t$ are all expressed in this marker frame. Across time, the implicit 3D flow is the displacement

$\mathbf{F}_{t+1} - \mathbf{F}_t.$

This provides a geometry-centric representation that is less sensitive to appearance and better aligned with cross-embodiment transfer than raw RGB or 2D image features.

The same 3D flow representation supports both manipulation learning and visibility reasoning. For manipulation, human hand pose is mapped to a gripper-equivalent pose and open/close command, while tracked 3D scene points describe object and hand motion in an embodiment-agnostic form. For active vision, predicted future 3D query point positions $\mathbf{q}_{t,i} \in \mathbb{R}^3$ can be evaluated against candidate camera poses using geometric visibility tests. This suggests that 3D flow is not merely a predictive target; it is the central state representation through which action, scene dynamics, and sensing geometry are coupled (Cho et al., 25 Feb 2026).

3. Coupled diffusion models and learning structure

EgoAVFlow consists of three diffusion models with a shared design based on a diffusion transformer backbone with conditioning via cross-attention. At time $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$0, the system consumes histories of 3D flows $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$1, human-derived proprioception $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$2, and camera poses $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$3, and predicts future robot actions, future 3D flows, and future camera poses over a horizon $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$4, with $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$5. At execution time it operates in chunks of size $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$6: it samples a 24-step window, executes the first 12 steps, and then replans (Cho et al., 25 Feb 2026).

Component	Conditioning	Output
Robot policy $$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$7	$$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$8	$$\{\mathbf{u}_t\}_{t=1}^{T'} \in \mathbb{R}^{N \times 3}$$9
Future flow model $\mathbf{X}_{t,i} \in \mathbb{R}^3$0	$\mathbf{X}_{t,i} \in \mathbb{R}^3$1	$\mathbf{X}_{t,i} \in \mathbb{R}^3$2
View policy $\mathbf{X}_{t,i} \in \mathbb{R}^3$3	$\mathbf{X}_{t,i} \in \mathbb{R}^3$4	$\mathbf{X}_{t,i} \in \mathbb{R}^3$5

Each component is trained as a DDPM/DDIM-style sequence model. For a target sequence $\mathbf{X}_{t,i} \in \mathbb{R}^3$6, the forward noising process is

$\mathbf{X}_{t,i} \in \mathbb{R}^3$7

and the reverse process is parameterized by

$\mathbf{X}_{t,i} \in \mathbb{R}^3$8

where $\mathbf{X}_{t,i} \in \mathbb{R}^3$9 denotes the conditioning context. Training uses a standard DDPM objective that predicts either noise or the denoised sample and minimizes MSE under the relevant context. Sampling uses DDIM-style deterministic or low-step procedures (Cho et al., 25 Feb 2026).

The robot policy $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$0 predicts 10D actions corresponding to next-step target proprioception $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$1, where the proprioception vector contains 3D position, a 6D rotation representation, and binary gripper state. The future flow model $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$2 predicts how tracked 3D points move under the future robot actions and future camera motion. The view policy $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$3 predicts future camera poses but is trained only by imitation; its distinctive behavior arises at test time, where sampling is altered by visibility-aware reward guidance rather than by an additional training-time reinforcement signal.

The paper presents the training objectives conceptually as three DDPM losses, $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$4, $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$5, and $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$6, combined as

$\mathbf{F}_t \in \mathbb{R}^{N \times 4},$7

Critically, no robot demonstrations are used. Egocentric human hand trajectories are treated as if they were gripper trajectories, and 3D flow of tracked points supplies the embodiment-agnostic motion description needed for transfer (Cho et al., 25 Feb 2026).

4. Visibility-aware planning and reward-maximizing denoising

The active-vision component of EgoAVFlow is organized around a visibility-aware reward defined over predicted future camera poses, predicted future 3D query points, and a reconstructed scene mesh. The environment mesh $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$8 is built with Nvblox, and a robot mesh $\mathbf{F}_t \in \mathbb{R}^{N \times 4},$9 is obtained from planned actions via inverse kinematics; their union is $\mathbf{v}_t \in \mathrm{SE}(3)$0. For camera position $\mathbf{v}_t \in \mathrm{SE}(3)$1 and query point $\mathbf{v}_t \in \mathrm{SE}(3)$2, line-of-sight is defined through the segment

$\mathbf{v}_t \in \mathrm{SE}(3)$3

with unobstructed visibility indicator

$\mathbf{v}_t \in \mathrm{SE}(3)$4

A field-of-view indicator $\mathbf{v}_t \in \mathrm{SE}(3)$5 is computed from the camera projection $\mathbf{v}_t \in \mathrm{SE}(3)$6, and per-point visibility is

$\mathbf{v}_t \in \mathrm{SE}(3)$7

The aggregate visibility reward is then

$\mathbf{v}_t \in \mathrm{SE}(3)$8

This formalizes visibility maintenance as a geometric property of future motion and scene layout, rather than as an image-space heuristic (Cho et al., 25 Feb 2026).

Because $\mathbf{v}_t \in \mathrm{SE}(3)$9 is binary and sparse, EgoAVFlow adds smooth auxiliary terms. The proximity term

$\mathbf{F}_t$0

encourages cameras to remain near the query points. A robustness term

$\mathbf{F}_t$1

is computed from perturbed camera poses to favor viewpoints whose visibility is stable under small perturbations. A safety term

$\mathbf{F}_t$2

discourages camera poses that are too close to the robot end effector. These are combined into

$\mathbf{F}_t$3

Test-time camera planning is performed by reward-maximizing denoising, termed RewMaxDiff in the exposition. The pre-trained view policy defines a prior $\mathbf{F}_t$4 over camera trajectories $\mathbf{F}_t$5. EgoAVFlow considers the entropy-regularized objective

$\mathbf{F}_t$6

whose solution is the reward-tilted distribution

$\mathbf{F}_t$7

Sampling from the induced reverse kernel is approximated by per-step importance resampling over candidate denoising trajectories, with weights proportional to $\mathbf{F}_t$8, where $\mathbf{F}_t$9 is the visibility-aware reward of the denoised trajectory implied by the candidate state. This procedure biases the sampled camera trajectory toward visibility-preserving solutions while remaining close to the human-view prior learned during imitation (Cho et al., 25 Feb 2026).

5. Training data, robot deployment, and empirical results

The training data consist of 150 egocentric demonstrations per task across four tasks, recorded by a human wearing a head-mounted Intel RealSense D435 RGB-D camera. Human hand pose is estimated with HaMeR, then converted into a gripper-equivalent 6D pose plus a binary gripper command. The resulting proprioception vector is 10D, and the action is defined as $\mathbf{v}_t$0. CoTracker3 provides 2D point tracking, depth maps support unprojection to 3D, and DROID-SLAM together with a ChArUco board places all signals into a shared marker frame. For execution, the environment mesh $\mathbf{v}_t$1 is reconstructed from robot camera observations using Nvblox (Cho et al., 25 Feb 2026).

Deployment uses two synchronized robots. The manipulation platform is a Trossen WidowX arm, whose action space is end-effector 6-DoF pose plus binary gripper command. The viewpoint platform is a Unitree Z1 arm carrying a RealSense D435; its action space is future 6-DoF camera end-effector poses in the marker frame. Every $\mathbf{v}_t$2 steps, the system collects histories, updates the environment mesh, samples robot actions from $\mathbf{v}_t$3, samples future 3D flows from $\mathbf{v}_t$4, and generates future camera poses with RewMaxDiff using $\mathbf{v}_t$5. The first 12 steps of both action trajectories are then executed via inverse kinematics at 4 Hz. This yields a receding-horizon loop in which manipulation and active vision are jointly replanned (Cho et al., 25 Feb 2026).

The empirical evaluation emphasizes tasks that require viewpoint adjustment because fixed viewpoints do not maintain visibility. In a spray task, four fixed cameras were tested, and no fixed view kept all annotated query points visible for the full rollout; the best view index changed over time. This motivates the active-vision requirement rather than a static-camera baseline. Against Human Viewpoint Imitation, defined as using the same manipulation policy $\mathbf{v}_t$6 but sampling $\mathbf{v}_t$7 without reward-maximizing denoising, EgoAVFlow achieved success rates of $\mathbf{v}_t$8 on Spray, Doll, Toilet, and Towel, respectively, whereas HVI achieved $\mathbf{v}_t$9. The visibility reward $p_t$0 was also consistently higher for EgoAVFlow.

To isolate the role of the manipulation representation under active viewpoints, the paper compares EgoAVFlow with AMPLIFY, EgoZero, and Phantom while giving all baselines the same active-view module, namely $p_t$1 with visibility-maximizing denoising. Success rates were $p_t$2 for AMPLIFY, $p_t$3 for EgoZero, $p_t$4 for Phantom, and $p_t$5 for EgoAVFlow across the four tasks. The reported interpretation is that AMPLIFY and Phantom lack inherent 3D geometry and therefore suffer distribution shift under novel viewpoints, while EgoZero relies on static-object assumptions for triangulation and degrades when the object moves with the gripper. EgoAVFlow’s 3D flow conditioned on robot actions is presented as the reason for robustness under actively changing views (Cho et al., 25 Feb 2026).

6. Assumptions, failure modes, and related nomenclature

EgoAVFlow assumes that the task-relevant query points are known and visible at the start of execution. It does not address discovering which points are important or searching for objects that begin out of view. Its 3D flow and marker-frame alignment depend on accurate depth from the egocentric RGB-D camera, reliable DROID-SLAM tracking, and ChArUco-board calibration. Its visibility estimation depends on an environment mesh built with Nvblox, so reconstruction errors or moving obstacles can affect planning quality. Reward-maximizing denoising is also computationally intensive because it requires multiple diffusion steps, multiple candidate samples per step, and repeated visibility evaluation via raycasting (Cho et al., 25 Feb 2026).

The reported failure analysis groups errors into grasping failure, out-of-viewpoint failure, and object pose failure. AMPLIFY, Phantom, and EgoZero account for most manipulation failures, including 91.6% of grasping failures and 81.3% of object pose failures, whereas Human Viewpoint Imitation accounts for more than 50% of out-of-viewpoint failures. EgoAVFlow has the fewest manipulation-related failures, though it still exhibits some out-of-viewpoint failures. This suggests that the framework improves both perception and control robustness, but does not remove the dependence on accurate prediction, tracking, and geometric reconstruction.

A common source of confusion is the similarity of names across recent arXiv papers. EgoAVFlow is distinct from “E-MoFlow,” which jointly estimates egomotion and optical flow from event data via implicit regularization (Li et al., 14 Oct 2025); from “EFlow,” which is an evidence-first framework for long-video reasoning with separate temporal grounding and reasoning trajectories (Zhang et al., 1 Jul 2026); and from “EgoFlow,” which is a gradient-guided flow-matching model for egocentric 6DoF object motion generation conditioned on multimodal observations (Saroha et al., 1 Apr 2026). The shared vocabulary of “ego,” “flow,” and “AV” reflects overlapping interests in egocentric perception, motion representation, and multimodal reasoning, but the specific technical object of EgoAVFlow is robot policy learning with active vision from human egocentric videos via 3D flow (Cho et al., 25 Feb 2026).