Reward-Guided Semantic Evolution for Test-time Adaptive Object Detection

Abstract: Open-vocabulary object detection with vision-LLMs (VLMs) such as Grounding DINO suffers from performance degradation under test-time distribution shifts, primarily due to semantic misalignment between text embeddings and shifted visual embeddings of region proposals. While recent test-time adaptive object detection methods for VLM-based either rely on costly backpropagation or bypass semantic misalignment via external memory, none directly and efficiently align text and vision in a training-free manner. To address this, we propose Reward-Guided Semantic Evolution (RGSE), a training-free framework that directly refines the text embeddings at test time. Inspired by evolutionary search, RGSE treats text embedding adaptation as a semantic search process: it perturbs text embeddings as candidate variants, evaluates them via cosine similarity with current and historical high-confidence visual proposals as a reward signal, and fuses them into a refined embedding through reward-weighted averaging. Without any backpropagation, RGSE achieves state-of-the-art performance across multiple detection benchmarks while adding minimal computational overhead. Our code will be open source upon publication.

• The paper introduces a novel RGSE framework that refines text embeddings at test time to correct semantic misalignment between visual and textual features.
• It employs an evolutionary search-inspired method—perturbing embeddings and using reward-weighted averaging—to improve detection performance without backpropagation.
• Experiments on benchmarks like FoggyCityscapes and COCO-C show significant mAP gains with minimal computational overhead.

Reward-Guided Semantic Evolution for Test-time Adaptive Object Detection

Introduction

This paper introduces Reward-Guided Semantic Evolution (RGSE), a framework designed for enhancing test-time adaptive object detection in vision-LLMs, specifically targeting the correction of semantic misalignment between text embeddings and visual embeddings under distribution shifts during testing. The RGSE framework aims to refine text embeddings directly, leveraging an evolutionary search-inspired method to enhance detection performance without the need for backpropagation, maintaining computational efficiency. The technique balances minimal computational overhead with achieving state-of-the-art performance across several benchmarks.

Methodology

Semantic Misalignment and Test-time Adaptation: Vision-LLMs, such as CLIP and Grounding DINO, align image and text representations in a shared semantic space to allow for zero-shot generalization across novel categories. However, performance degradation occurs under distribution shifts due to misalignment between text embeddings and shifted visual embeddings. Traditional test-time adaptation methods either involve computationally expensive backpropagation or bypass semantic misalignment by leveraging external memory. RGSE addresses these limitations by directly refining text embeddings in an efficient, training-free manner.

RGSE Process Overview: RGSE involves a four-stage process:

1. Perturbation: Generates multiple candidate text embeddings by adding Gaussian noise to the embeddings of active categories.
2. Reward Evaluation: Computes a reward for each perturbed embedding based on its cosine similarity with current and historical high-confidence visual proposals.
3. Refinement: Fuses perturbed embeddings into a refined text embedding via reward-weighted averaging.
4. Memory Update: Updates a class-specific cache with high-confidence visual proposals.

   Figure 1: Comparison between previous methods and RGSE. (a) Previous methods either rely on computationally expensive backpropagation to update text embeddings or bypass semantic misalignment by maintaining a separate visual cache. (b) RGSE treats text embedding adaptation as an evolutionary search, which enables direct and efficient correction of semantic misalignment.

Experimental Results

Benchmark Evaluation: RGSE was evaluated on FoggyCityscapes, PASCAL-C, and COCO-C datasets, demonstrating robust improvements across various challenging scenarios including diverse weather conditions and synthetic corruption types. It consistently outperformed existing state-of-the-art methods, showing substantial gains in mAP metrics.

Efficiency and Hyperparameter Sensitivity: Through detailed ablation studies, RGSE is shown to maintain efficiency, with negligible increases in computational time and memory compared to other techniques relying on backpropagation. Sensitivity analyses also revealed its robustness to variations in hyperparameters like perturbation number and historical reward weights.

Qualitative Evaluation

Qualitative results further substantiate RGSE's capacity to correct semantic misalignments, demonstrating more accurate object detection under severe domain shifts. Visual comparisons with baseline models illustrate RGSE’s effectiveness in reducing false positives and recovering missed detections.

Figure 2: Overview of Reward-Guided Semantic Evolution (RGSE). Given a test image, we first obtain the initial outputs from Grounding DINO: region proposals $\{\bm{v}_m, \bm{b}_m, \bm{p}_m^{\text{init}, l_m\}_{m=1}^M$, and text embeddings $\{\bm{t}_k\}_{k=1}^K$. To address semantic misalignment under domain shift, RGSE refines the original text embeddings through four forward-pass stages: (1) Perturbation: Gaussian noise is added to active text embeddings; (2) Reward evaluation: each perturbed embedding is scored based on cosine similarity with current and historical high-confidence visual proposals; (3) Refinement: only candidates outperforming the parent are fused via reward-weighted averaging; (4) Memory update: high-confidence proposals are added to a class-specific cache for future adaptation.

Discussion

Implications and Future Research: RGSE offers significant practical implications, particularly in real-time applications where computational efficiency is vital. It paves the way for more sophisticated test-time adaptation frameworks in vision-LLMs, emphasizing lightweight approaches without the trade-off in performance. Future research could explore adaptive strategies, optimizing perturbation and reward evaluation processes for greater precision, and extending RGSE’s capabilities to other areas requiring semantic alignment.

Conclusion

Reward-Guided Semantic Evolution presents a robust solution for the test-time adaptation challenge in vision-language object detection, avoiding heavy computational techniques and achieving superior detection performance in dynamically shifting environments. By refining embeddings through evolutionary principles, RGSE secures its position as a leading framework for semantic adaptation, contributing substantially to the development of efficient, adaptable AI systems.