Action Genome: Actions as Composition of Spatio-temporal Scene Graphs (1912.06992v1)

Abstract: Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However in Computer Vision, few explorations on representations encoding event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.

• The paper presents a novel dataset of 10,000 videos annotated with 400,000 objects and 1.7M relationships for detailed action recognition.
• The authors use hierarchical decomposition of actions into spatio-temporal scene graphs to capture dynamic object interactions.
• They achieve improved action recognition, including robust few-shot learning with 42.7% mAP, setting a benchmark for scene graph prediction.

Action Genome: Actions as Composition of Spatio-temporal Scene Graphs

The paper "Action Genome: Actions as Composition of Spatio-temporal Scene Graphs" by Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles introduces a novel approach to action recognition in videos. The researchers propose a new representation, Action Genome, which decomposes actions into spatio-temporal scene graphs. This representation aims to provide a structured decomposition of actions by capturing the changes in objects and their pairwise relationships as an action unfolds.

Key Contributions

1. Action Genome Dataset: The researchers construct a significant dataset as part of the Action Genome framework. It includes 10,000 videos, with annotations of 400,000 objects and 1.7 million visual relationships. This dataset is built upon the Charades dataset and is tailored to enhance the granularity of action recognition tasks by integrating spatio-temporal scene graph labels.
2. Hierarchical Event Decomposition: The authors emphasize the efficacy of breaking down actions into smaller, manageable units by looking into the temporal changes of visual interactions. Inspired by Cognitive Science theories suggesting that people naturally encode activities in hierarchical part structures, the Action Genome reflects this by focusing on the segmentations of interactions between actors and objects.
3. Improved Action Recognition and Few-shot Learning: By applying the spatio-temporal scene graphs for action recognition, the model shows an improvement in performance. Notably, when using only a limited number of samples for training (as few as 10 examples), the method achieves a mean Average Precision (mAP) of 42.7% for few-shot action recognition, indicating robust generalization capabilities.
4. Scene Graph Feature Banks: The paper introduces the concept of Scene Graph Feature Banks, which are integral to forecasting actions based on object relationships. These feature banks extend existing models by incorporating scene graph predictions for enhanced action recognition, achieving state-of-the-art results on the Charades dataset.
5. Benchmark for Spatio-temporal Scene Graph Prediction: The paper also establishes benchmarks for a new task—spatio-temporal scene graph prediction. This task assesses the ability to predict how objects and relationships evolve over time within a video sequence.

Implications and Future Directions

The implications of this work are manifold. From a theoretical standpoint, the paper advocates for a paradigm shift in understanding and modeling actions in videos as dynamic compositions rather than static, isolated events. The application of scene graphs in video analysis may lead to more nuanced and precise models that can discern complex interactions in dynamic environments.

Practically, the work can inform the development of more sophisticated action recognition systems that are needed in various fields ranging from automated video surveillance to human-computer interaction. Moreover, the dataset and methodologies introduced can be catalysts for further research in video understanding, allowing for innovations in domains that require temporal insight into human activities.

Potential future research prompted by the Action Genome framework could delve into real-time spatio-temporal prediction and recognition, scene graph prediction advancements by leveraging temporal context more effectively, and transfer learning capabilities for applications in different video domains without extensive retraining. Additionally, exploration into applications of spatio-temporal graphs in multi-agent interactions and complex scenarios could broaden the utility of this approach.

Conclusively, Action Genome sets forth a significant stride toward refining action recognition tasks, offering a scalable and interpretable framework that enhances both practical applications and theoretical explorations in video-based AI.