Storing Less, Finding More: How Novelty Filtering Improves Cross-Modal Retrieval on Edge Cameras

Abstract: Always-on edge cameras generate continuous video streams where redundant frames degrade cross-modal retrieval by crowding correct results out of top-k search. This paper presents a streaming retrieval architecture: an on-device epsilon-net filter retains only semantically novel frames, building a denoised embedding index; a cross-modal adapter and cloud re-ranker compensate for the compact encoder's weak alignment. A single-pass streaming filter outperforms offline alternatives (k-means, farthest-point, uniform, random) across eight vision-LLMs (8M-632M) on two egocentric datasets (AEA, EPIC-KITCHENS). Combined, the architecture reaches 45.6% Hit@5 on held-out data using an 8M on-device encoder at an estimated 2.7 mW.

• The paper introduces a streaming ε-net filter that retains semantically novel frames, enabling efficient cross-modal retrieval on edge cameras.
• It interleaves compact on-device encoding with cloud-based re-ranking, achieving up to 9.6 percentage points improvement in top-5 retrieval metrics.
• The approach reduces bandwidth usage and preserves privacy by transmitting only embeddings, demonstrating scalability on low-power devices.

Semantic Redundancy Filtering for Efficient Cross-Modal Retrieval on Edge Cameras

Motivation and Problem Statement

Edge devices such as AR glasses, body cameras, and warehouse robots generate continuous egocentric video streams, overwhelming storage and transmission infrastructure due to frame redundancy. This redundancy not only increases resource demands but actively impairs retrieval performance, as repeated frames crowd out semantically salient moments in top-$k$ cross-modal searches. The paper addresses fundamentally query-agnostic, streaming frame selection for cross-modal image-text retrieval, targeting battery-limited platforms incapable of running large vision-language encoders.

Architecture and Methodological Contributions

The proposed architecture interleaves on-device filtering and cloud-side search, decoupling frame selection from query processing. On-device, a compact encoder transforms frames into embeddings; a streaming $\varepsilon$-net filter retains only semantically novel frames, enforcing explicit separation via cosine distance threshold $\tau$. At query time, a Locked-image Tuning (LiT) adapter projects text queries into the embedding space, and a cloud re-ranker further optimizes alignment using a larger frozen encoder. Critically, the architecture operates solely on embeddings: raw images never leave the device, minimizing bandwidth and preserving privacy.

Figure 1: Architecture overview showing stages of on-device encoding/filtering and cloud-based cross-modal re-ranking.

The $\varepsilon$-net streaming filter implements single-pass frame selection. For each incoming embedding, the filter checks if its maximum cosine similarity with previously retained embeddings is less than $\tau$. If so, the frame is retained, ensuring the resulting set covers all moments while avoiding redundancy. This streaming protocol obviates the need for offline clustering or query-guided selection.

Empirical Evaluation

The denoising effect is validated across eight vision-language retrieval models (8M–632M parameters) using Aria Everyday Activities (AEA) and EPIC-KITCHENS datasets—both comprised of annotated egocentric video sequences. Filtering via novelty selection consistently outperforms offline baselines (k-means, farthest-point, uniform, random) and even full-frame sets in Hit@5 retrieval metrics. Filtering yields the most substantial recovery in compact models with weak cross-modal alignment; for instance, CLIP B/32 retrieves 9.6 percentage points more events after filtering, while MobileCLIP improves by 8.9 points.

Figure 2: Top-5 retrieval performance for various models, with novelty filtering outperforming offline baselines in most cases.

Bootstrap resampling confirms robustness, with the novelty gain holding across methods and model scales. Experimental sweeps of the cosine threshold $\tau$ demonstrate that, when tuned to dataset-specific event durations, the improvement generalizes to short events (EPIC-KITCHENS): for $\tau \geq 0.94$, filtering wins in 6 out of 7 videos.

Alignment Enhancement and Further Gains

Compact encoders are inherently limited in cross-modal alignment. The architecture addresses this via a lightweight LiT MLP, trained to project text queries into the compact image embedding space. The combination of filtering and adaptation yields multiplicative gains: TinyCLIP, for example, improves from 11.0% to 21.3% in top-5 retrieval. Fine-tuning both the encoder and adapter jointly, using synthetic dense captions, pushes held-out performance to 25.0% (AEA) and 6.0% (EPIC-KITCHENS).

Scalability, Privacy, and Deployment Implications

Profiling on edge hardware (Hexagon v69 NPU) shows the TinyCLIP encoder plus filter both run at <3 mW, consuming just 1% of the NPU cycles at 5 FPS, well below typical edge device power budgets (~200 mW). The embedding-only pipeline enables transmission rates of ≤43 KB/min, easily within BLE 4.0 capacity, compared to ~45 MB/min for raw frames. By transmitting only embeddings, visual privacy is inherently preserved—no reconstructable visual content leaves the device.

End-to-End Retrieval and Re-ranking

Final system performance is constrained by the compact on-device encoder, but cloud-side re-ranking using a large model (SigLIP 2) improves held-out hit rates to 45.6% (Hit@5) and 77.9% (Hit@50), surpassing baseline retrieval on the full, unfiltered frame sets, while requiring $5.7\times$ fewer re-ranker candidates at comparable hit rates.

Figure 3: Cumulative improvement with filtering, adaptation, and re-ranking; reduced candidates needed for equivalent hit rates.

Theoretical Implications and Context

Removing semantic redundancy from the retrieval index alters geometric bias in nearest-neighbor search, especially for compact models where alignment errors are exacerbated by cluster dominance. The streaming $\varepsilon$-net is provably coverage-complete and model-agnostic, decoupling frame selection from retrieval and query processing. Unlike prior approaches—offline deduplication, shot-boundary detection, and query-guided summarization—the method requires only local embedding comparisons and single-pass processing, satisfying real-world constraints for always-on, battery-limited devices.

Limitations and Proposed Extensions

Dataset scope is limited to indoor environments; external validation on industrial or outdoor streams is needed. The fixed threshold $\tau$ is not self-adaptive, leading to compression drift and reduced coverage for exceptionally long videos or varying event granularity. Domain adaptation and fine-tuning are still bottlenecked by training data diversity, reflected in train/test gaps. Incorporating multi-sensor stream gating or adaptive thresholding could further extend efficacy and efficiency.

Conclusion

Semantic redundancy filtering via streaming $\varepsilon$0-nets yields measurable improvement in cross-modal retrieval for edge device pipelines, particularly when paired with lightweight alignment adapters and scalable cloud-side re-ranking. The modular architecture supports independent upgrades, rigorous privacy constraints, and practical system requirements. Theoretical and practical implications suggest broad applicability in edge AI deployments, with future research poised to address adaptation, robustness, and domain generalization.