ElasticTok: Adaptive Tokenization for Image and Video (2410.08368v1)

Abstract: Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents.

ElasticTok: Adaptive Tokenization for Image and Video

The paper introduces ElasticTok, an innovative approach aimed at addressing inefficiencies in video tokenization necessary for training robust vision models capable of handling extensive video sequences. Current tokenization strategies often adopt a fixed token allocation, resulting in inefficiencies and computational overhead due to either excessive or insufficient tokenization. ElasticTok proposes a dynamic token allocation mechanism, adjusting token quantity according to the complexity of the visual data, thereby optimizing resource use.

Methodology

ElasticTok employs an adaptive mechanism that utilizes a masking strategy to apply dynamic tokenization of frames dependent on the complexity of prior frames’ content. This method integrates with traditional autoencoders, introducing an adaptive mask that determines the number of tokens used based on a sampled latent variable. This flexibility enables the model to increase tokens for complex scenes while reducing them for simpler content, promoting efficiency.

Two primary use cases for inference are proposed: specifying a target encoding length or achieving a target reconstruction quality. Various search algorithms are employed to optimize inference, including exhaustive search (Full Search), reduced search space (Binned Search), and learning-based Regression.

Experimental Results

The empirical analysis confirms that ElasticTok significantly reduces token usage without compromising reconstruction quality. For instance, ElasticTok achieves efficient representation with 2-5x fewer tokens. Evaluations across image and video datasets reveal that ElasticTok meets targeted reconstruction thresholds while undershooting the token usage of fixed token baselines. Notably, performance gains are more pronounced with less stringent reconstruction thresholds.

The adaptive nature of ElasticTok is further validated through its application in downstream visual-language tasks. The model demonstrates the capability to either match or exceed the efficiency and accuracy of fixed-token baselines in tasks like VQA, offering flexibility in allocating computational resources based on budget constraints.

Implications and Future Directions

ElasticTok offers substantive contributions to the computational efficiency of vision models by tailoring token usage according to data complexity. Its successful application suggests potential in broader temporal modalities, including audio and trajectory analysis in reinforcement learning contexts.

Despite its benefits, ElasticTok does present some limitations, such as reduced performance at extreme encoding lengths. Future explorations could involve refined masking schemes and investigating learnable masking patterns, potentially enhancing performance further.

In summary, ElasticTok sets a precedent in adaptive video tokenization, efficiently bridging the gap between computational overhead and model effectiveness, offering a pathway toward more scalable and adaptable models in multimodal data processing.