AdaFrame: Adaptive Frame Selection for Fast Video Recognition (1811.12432v2)

Abstract: We present AdaFrame, a framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame contains a Long Short-Term Memory network augmented with a global memory that provides context information for searching which frames to use over time. Trained with policy gradient methods, AdaFrame generates a prediction, determines which frame to observe next, and computes the utility, i.e., expected future rewards, of seeing more frames at each time step. At testing time, AdaFrame exploits predicted utilities to achieve adaptive lookahead inference such that the overall computational costs are reduced without incurring a decrease in accuracy. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet. AdaFrame matches the performance of using all frames with only 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We further qualitatively demonstrate learned frame usage can indicate the difficulty of making classification decisions; easier samples need fewer frames while harder ones require more, both at instance-level within the same class and at class-level among different categories.

AdaFrame: Adaptive Frame Selection for Fast Video Recognition

AdaFrame proposes an innovative solution to optimize computational efficiency in video recognition tasks, leveraging the adaptive selection of video frames. This paper addresses the challenges associated with efficiently processing the substantial quantity of video data generated daily, such as the 300 hours uploaded to platforms like YouTube every minute. The key contribution lies in reducing redundant computations while maintaining high accuracy in video classification.

AdaFrame introduces a novel framework that relies on a Long Short-Term Memory (LSTM) network augmented with a global memory component. This system selectively identifies relevant frames from videos on a per-input basis, making it particularly suited for datasets where each video is typically annotated with a single, broad label. The framework utilizes policy gradient methods to drive this adaptive selection, allowing AdaFrame to generate predictions and assess the utility of observing further frames throughout the processing cycle.

The model's effectiveness and efficiency are validated through extensive experiments on two major video benchmarks, FCVID and ActivityNet. Remarkably, AdaFrame achieves comparable performance to exhaustive frame usage methods, but with significantly reduced amounts of frame engagements—only 8.21 frames for FCVID and 8.65 for ActivityNet on average. This reduction reflects a substantial saving in computational cost without sacrificing classification accuracy.

AdaFrame's experimental setups explore varying configurations of the global memory module and different reward functions, confirming the robustness of its design choices. For instance, utilizing representations from 16 frames in the global memory proved optimal, balancing computational overhead with classifier performance gains. Additionally, the frame usage is shown to correlate with the difficulty level required for making classification decisions, validating AdaFrame's ability to adaptively dedicate computational resources according to the complexity of input instance.

The implications of AdaFrame's approach are manifold. Practically, its capacity to halve or even reduce the computational requirement by up to 90.6% opens up avenues for more scalable video processing applications. This is crucial for deployment in environments with limited computational resources or in real-time settings where processing speed is critical. Theoretically, the concept of integrating global context information into LSTM networks might inspire future developments in sequence-based AI models, particularly in fields like autonomous video surveillance systems or complex event detection.

In summary, AdaFrame presents a comprehensive strategy for efficient video recognition via adaptive frame selection. Its methodologies promise notable efficiencies in computational processing, holding significant potential for application in both industry and academia. Future research may focus on extending AdaFrame's principles to multi-modal scenarios or exploring alternative neural architectures that can further optimize frame selection dynamics.