BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation

Abstract: Continual Test Time Adaptation (CTTA) is required to adapt efficiently to continuous unseen domains while retaining previously learned knowledge. However, despite the progress of CTTA, it is still challenging to deploy the model with improved forgetting-adaptation trade-offs and efficiency. In addition, current CTTA scenarios assume only the disjoint situation, even though real-world domains are seamlessly changed. To address these challenges, this paper proposes BECoTTA, an input-dependent and efficient modular framework for CTTA. We propose Mixture-of Domain Low-rank Experts (MoDE) that contains two core components: (i) Domain-Adaptive Routing, which helps to selectively capture the domain adaptive knowledge with multiple domain routers, and (ii) Domain-Expert Synergy Loss to maximize the dependency between each domain and expert. We validate that our method outperforms multiple CTTA scenarios, including disjoint and gradual domain shits, while only requiring ~98% fewer trainable parameters. We also provide analyses of our method, including the construction of experts, the effect of domain-adaptive experts, and visualizations.

• The paper presents a novel CTTA framework that leverages a Mixture-of-Domain Low-rank Experts module to dynamically adapt to shifting domains.
• It employs domain-adaptive routing and a domain-expert synergy loss to selectively update parameters, reducing forgetting while enhancing adaptation speed.
• Experimental results on benchmarks including CGS show significant IoU improvements and up to 98% fewer parameters compared to state-of-the-art baselines.

Review of "BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation"

The paper "BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation" introduces a novel framework for addressing the challenges encountered in Continual Test-Time Adaptation (CTTA) scenarios. CTTA is a crucial area in machine learning that deals with adapting models to continually changing domains during the test phase while efficiently preserving previously acquired knowledge. Despite the recognition of CTTA's significance, the current body of research has not fully explored the trade-offs between forgetting and adaptation or the model's efficiency. This paper aims to fill these gaps with its innovative approach.

Contributions and Methodology

The authors propose BECoTTA, a framework that provides input-dependent and efficient CTTA by leveraging a Mixture-of-Domain Low-rank Experts (MoDE) module. The MoDE module integrates two pivotal components:

1. Domain-Adaptive Routing: This component selectively captures domain-adaptive knowledge utilizing multiple domain routers. This process allows models to adapt to current data distributions dynamically and efficiently, minimizing interference and enhancing adaptation speed.
2. Domain-Expert Synergy Loss: This component is designed to maximize the dependency between specific domains and their corresponding experts, facilitating more effective collaboration and specialization among domain experts.

By focusing on these components, the BECoTTA framework enables a selective update of domain experts, catering to the domain-specific knowledge required for different test scenarios. This modular design enhances both parameter and memory efficiency by implementing sparse parameter updates only when necessary, thereby preserving past knowledge and ensuring rapid adaptability to new domains.

Additionally, the authors introduce the Continual Gradual Shifts (CGS) benchmark, which offers a more realistic CTTA scenario by considering gradual shifts in domains over time. This benchmark presents the opportunity to test models in environments that closer reflect real-world conditions, such as the seamless transition between weather conditions in autonomous driving scenarios.

Experimental Results

The paper presents an extensive evaluation of BECoTTA's performance across several CTTA scenarios, including CDS-Hard, CDS-Easy, and the newly proposed CGS benchmark. In direct comparisons with strong baseline models such as TENT, SAR, and EcoTTA, BECoTTA consistently outperformed across all tested scenarios. Notably, it achieved a significant increase in performance metrics such as mean Intersection over Union (IoU) while utilizing substantially fewer parameters (~98% fewer in some instances compared to state-of-the-art models), which speaks to its efficiency and practical applicability.

Furthermore, the paper explores the potential of BECoTTA in zero-shot domain generalization tasks, demonstrating its ability to generalize well to unseen domains, reinforced by its performance on varied datasets such as BDD-100k, Mapillary, GTAV, and Synthia.

Implications and Future Work

The authors have laid a robust foundation for exploring CTTA with a more application-oriented approach that capitalizes on low-rank experts for efficient adaptation. The promising results underscore the potential for BECoTTA to be deployed in real-world systems where domain shifts are inevitable, particularly in settings constrained by computational resources and memory, like edge devices.

Potential future work could explore the extension of this framework to other machine learning tasks and domains, evaluating the universality of this approach. Furthermore, there is room to investigate more complex scenarios that feature even finer-grained domain transitions or multi-modal domain environments, accentuating BECoTTA's flexibility and robustness in diverse conditions.

Overall, BECoTTA marks a significant stride in CTTA research, providing both a compelling framework and a set of rigorous benchmarks to drive future exploration in this domain.