Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling

Abstract: We present an approach for weakly supervised learning of human actions. Given a set of videos and an ordered list of the occurring actions, the goal is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. To address this task, we propose a combination of a discriminative representation of subactions, modeled by a recurrent neural network, and a coarse probabilistic model to allow for a temporal alignment and inference over long sequences. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes. To this end, we adapt the number of subaction classes by iterating realignment and reestimation during training. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.

• The paper presents a hybrid fine-to-coarse model that integrates GRU-based RNNs and HMMs to align video actions with minimal supervision.
• It employs RNNs for detailed subaction modeling and HMMs for enforcing sequential order, achieving notable improvements in temporal segmentation accuracy.
• Iterative refinement of subaction estimates delivers robust performance on benchmarks like the Breakfast and Hollywood Extended datasets.

Weakly Supervised Action Learning with RNN Based Fine-to-Coarse Modeling

The paper "Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling" by Richard, Kuehne, and Gall introduces a method for learning human actions within videos in a weakly supervised context. Rather than relying on extensively annotated datasets that include exact frame boundaries for each action, the authors propose employing an ordered list of actions present in the entire video, thereby alleviating the need for detailed annotations.

Methodology

The proposed approach leverages a hybrid architecture, combining a fine-grained discriminative model rooted in recurrent neural networks (RNNs) with a coarse probabilistic model. This architecture allows for accurate temporal alignment by capturing both local and global sequential dependencies of action subcomponents within long video sequences.

1. Fine-grained Modeling with RNNs: The RNN, particularly GRUs, models subactions for capturing short-term temporal dependencies. The challenges of processing long video sequences are addressed by introducing manageable chunks of input frames, facilitating the training process through parallelization while maintaining accuracy.
2. Coarse Model with HMMs: Subactions are treated as latent variables organized by a hidden Markov model (HMM). This structure ensures correct subaction sequencing, enabling robust alignment of actions to video frames.
3. Iterative Refinement: The model undergoes iterative refinements, optimizing the alignment between video frames and subaction classes. Noteworthy is the process of dynamically reassessing and adjusting the number of subactions per main action, which is crucial for capturing the heterogeneity in action sequences.

Results

The model demonstrates competitive performance on two benchmark datasets, the Breakfast dataset and the Hollywood extended dataset, outperforming previously established methods. For the task of temporal action segmentation, where the goal is both the classification and localization of actions within untrimmed videos, the method shows improved accuracy over state-of-the-art methods. Similarly, for the alignment task—given an ordered list of actions, determining their precise temporal locations—the model also achieves superior results.

On the Breakfast dataset, the approach with subaction reestimation achieves an accuracy of 33.3% (Mof), while on the Hollywood Extended dataset, it attains a Jaccard index of 46.3% (IoD) for action alignment. This indicates significant improvements over existing methods like OCDC and HTK by leveraging the dynamic reassessment process.

Implications and Future Directions

This research presents significant implications for the field of video action recognition, particularly in resource-constrained settings where fully annotated datasets are impractical. The methodological innovation lies in circumventing expensive annotation processes by focusing on weakly labeled data, which is more feasible for real-world applications.

Further developments may explore enhanced architectures beyond basic RNNs, such as more advanced transformer models, to capture even finer temporal dependencies in untrimmed videos. Additionally, further refinement in estimating the optimal number of subactions dynamically could potentiate even greater accuracy improvements.

In conclusion, the paper lays a robust foundation for weakly supervised action recognition, providing valuable insights into integrating probabilistic sequencing and discriminative features through RNNs. This approach not only improves current benchmarks but also sets a direction for future research aimed at bridging the gap between fully and weakly supervised learning paradigms in video analytics.