Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos

Abstract: Understanding physical transformation processes is crucial for both human cognition and artificial intelligence systems, particularly from an egocentric perspective, which serves as a key bridge between humans and machines in action modeling. We define this modeling process as Egocentric Instructed Visual State Transition (EIVST), which involves generating intermediate frames that depict object transformations between initial and target states under a brief action instruction. EIVST poses two challenges for current generative models: (1) understanding the visual scenes of the initial and target states and reasoning about transformation steps from an egocentric view, and (2) generating a consistent intermediate transition that follows the given instruction while preserving object appearance across the two visual states. To address these challenges, we propose the EgoIn framework. It first infers the multi-step transition process between two given states using TransitionVLM, fine-tuned on our curated dataset to better adapt to this task and reduce hallucinated information. It then generates a sequence of frames based on transition conditions produced by the proposed Transition Conditioning module. Additionally, we introduce Object-aware Auxiliary Supervision to preserve consistent object appearance throughout the transition. Extensive experiments on human-object and robot-object interaction datasets demonstrate EgoIn's superior performance in generating semantically meaningful and visually coherent transformation sequences.

• The paper presents the EgoIn framework to model transitions in egocentric video, using TransitionVLM for object state transformation.
• Object-aware Auxiliary Supervision ensures appearance consistency, leveraging advanced detection methods for effective transition preservation.
• Experimental evaluations show EgoIn's superior performance in video generation quality on datasets like EpicKitchens-100 and Ego4D FHO.

Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos

Introduction

The paper "Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos" (2604.17749) addresses the challenge of modeling object state transformations from an egocentric perspective, a crucial aspect for artificial intelligence systems aimed at bridging the gap between human cognition and machine understanding of physical processes. The research introduces the Egocentric Instructed Visual State Transition (EIVST) task, requiring generative models to produce intermediate frames that depict object transitions from initial states to target states under a specified action instruction.

The EIVST task poses two significant challenges: the need to understand and reason about visual scenes from an egocentric view, and to generate consistent intermediate transitions that preserve object appearance across states. To address these challenges, the proposed EgoIn framework leverages TransitionVLM for transition process modeling and integrates Object-aware Auxiliary Supervision to ensure consistent appearance throughout the transformation, effectively overcoming the limitations of current generative models in handling egocentric video transformations.

Figure 1: Examples of diverse generated object state transition sequences under different textual and visual conditions.

Methodology

EgoIn Framework

The EgoIn framework employs a divide-and-conquer strategy to tackle EIVST. It consists of two key stages:

1. Transition Process Modeling: Utilizing a custom-tuned Vision-LLM (TransitionVLM) to interpret and generate transition steps between initial and target states, the framework significantly reduces hallucinated information and boosts the reasoning about transitional transformations.
2. Intermediate Frame Generation: Based on transition conditions, EgoIn synthesizes intermediate frames with a Transition Conditioning (TC) module. This module refines transition-aware features, facilitating the generation of semantically coherent sequences.
3. Object-aware Auxiliary Supervision: This component ensures the consistency and smoothness of object appearance during transitions, implemented through localization masks obtained via advanced object detection methods like Qwen2.5-VL and SAM2.

   Figure 2: Illustration of the proposed EgoIn framework with emphasis on transition process modeling and object-awareness.

TransitionVLM and TC Module

TransitionVLM is finely tuned using curated data to address transition modeling's inherent limitations, preventing the generation of fantasy transitions ungrounded in actual visual input. The TC module further enhances video synthesis by explicitly incorporating frame-wise conditions through a frame-wise cross-attention mechanism. This mechanism ensures semantic integrity and temporal coherence across generated video frames.

Figure 3: Illustration of the tuning process for TransitionVLM, emphasizing data curation and adaptation.

Results and Evaluation

Experimental Results

The proposed EgoIn framework underwent extensive evaluation against state-of-the-art methods on multiple datasets, including EpicKitchens-100, Ego4D FHO, DualArm, and Bridge. EgoIn consistently demonstrated superior performance across numerous metrics, such as FVD, VTQ, VTC, and VIC, showcasing its ability to generate visually coherent transitions aligned with given instructions.

The user study further validated these findings, with participants expressing higher preference for videos generated by EgoIn, citing better reasonability in transitions, alignment with instructions, and overall motion quality.

Figure 4: Illustration of the Transition Conditioning module's components and workflow.

Ablation Studies

Ablation studies confirmed the effectiveness of each EgoIn component. TransitionVLM and TC module jointly contributed to significant improvements in video generation quality, while Object-aware Auxiliary Supervision reinforced appearance consistency, demonstrating the importance of each layer in refining transitional video generation.

Figure 5: Qualitative comparison on Epic100 and Bridge datasets showcasing intermediate frames from generated sequences.

Conclusion

The research presented in "Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos" offers a robust framework for understanding and generating complex object state transitions from an egocentric perspective. The EgoIn framework achieves semantic richness and appearance consistency through its innovative use of TransitionVLM and Object-aware Auxiliary Supervision. Although generating long-horizon transitions remains challenging, EgoIn sets a solid foundation for future advancements in embodied AI and its applications in robotic manipulation, computer-aided design, and educational tools. Future work can expand upon these findings to tackle long-range dependencies and dynamic environments.