When Do We Not Need Larger Vision Models? (2403.13043v2)

Abstract: Scaling up the size of vision models has been the de facto standard to obtain more powerful visual representations. In this work, we discuss the point beyond which larger vision models are not necessary. First, we demonstrate the power of Scaling on Scales (S$^2$), whereby a pre-trained and frozen smaller vision model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation. Notably, S$^2$ achieves state-of-the-art performance in detailed understanding of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the conditions under which S$^2$ is a preferred scaling approach compared to scaling on model size. While larger models have the advantage of better generalization on hard examples, we show that features of larger vision models can be well approximated by those of multi-scale smaller models. This suggests most, if not all, of the representations learned by current large pre-trained models can also be obtained from multi-scale smaller models. Our results show that a multi-scale smaller model has comparable learning capacity to a larger model, and pre-training smaller models with S$^2$ can match or even exceed the advantage of larger models. We release a Python package that can apply S$^2$ on any vision model with one line of code: https://github.com/bfshi/scaling_on_scales.

When Do We Not Need Larger Vision Models?

Introduction

The pursuit of increasingly larger vision models has been a dominant trend in the arena of artificial intelligence research, driven by the belief that scaling up model size directly correlates with improved performance across a spectrum of visual understanding tasks. This paper, through an extensive analysis, introduces an alternative scaling strategy—Scaling on Scales (S$^2$)—challenging the conventional wisdom that "bigger is always better." It demonstrates that a strategic scaling of image inputs, without proportionally increasing model parameters, can not only compete with but in certain instances, surpass the performance of heftier counterparts.

The Concept of S$^2$

S$^2$ diverges from traditional model scaling approaches by focusing on manipulating the input scale rather than the complexity of the model itself. By employing a pre-trained vision model across multiple image scales, S$^2$ yields a multi-scale representation that inherently captures a broad spectrum of visual details—ranging from granular to global perspectives. The intriguing part is that these enriched representations are achieved without any alterations to the model architecture or an increase in parameters. This process involves interpolating images to varying scales and subsequently, pooling and concatenating the generated features to forge a comprehensive multi-scaled representation.

Empirical Validation

Extensive experiments across several benchmarks—including classification, segmentation, depth estimation, multimodal LLMs, and robotic manipulation—reveal the efficacy of S$^2$. Remarkably, models enhanced with S$^2$ consistently demonstrated competitive or superior performance relative to their larger counterparts, showcasing the viability of S$^2$ as a scalable and efficient alternative to blindly scaling model size. This is illustrated through the state-of-the-art results achieved on the V$^\ast$ benchmark for detailed visual understanding in multimodal LLMs, where S$^2$ scaled models outperformed notable entities such as GPT-4V and commercial models.

Analyzing Model Performance and Capacity

A deeper investigation into why larger models outperform in some instances pointed towards their better generalization on rare or ambiguous examples. However, when analyzing the representational overlap between smaller models with S$^2$ and larger models, it was found that the former can approximate the features of the latter quite effectively. This revelation, denoting a similar capacity for learning between smaller S$^2$ models and larger models, suggests that with appropriate training strategies, smaller models could achieve or exceed the generalization capabilities and performance efficiencies of their larger counterparts.

Practical Implications and Future Outlook

The findings invigorate a discourse on the necessity of model scaling strategies in advancing visual understanding. By offering an alternative that circumvents the computational and resource-intensive demands of larger models, S$^2$ unlocks new potentials for efficient and scalable AI development. It posits a future where focusing on the input dimensions, like image scales, could be as impactful, if not more, as scaling model sizes. This invites further exploration into scale-selective processing and parallel processing of single images, promising directions that could redefine efficiency and performance benchmarks in visual computing tasks.

Conclusion

Scaling on Scales (S$^2$) emerges as a compelling paradigm, challenging the enduring convention of associating model performance with size. Through rigorous analysis and empirical evidence, this work elucidates the potential of S$^2$ to redefine the metrics of efficiency and performance in visual understanding tasks, heralding a shift towards more pragmatic and resource-conscious approaches in the development of AI models.