SpotVLM: Cloud-Edge Real-Time VLM System

• SpotVLM is a cloud-edge collaborative framework for real-time vision-language reasoning that uses Context Transfer to repurpose delayed cloud outputs as high-quality historical context.
• It employs a Context Replacement Module and a Visual Focus Module to integrate precise LVLM outputs with fast SVLM inference in latency-sensitive tasks like autonomous driving and video captioning.
• The framework improves semantic and visual grounding by transferring historical context, yielding superior performance across multiple real-time tasks even under variable cloud delays.

Searching arXiv for the specified paper and closely related work names mentioned in the provided data. SpotVLM is a cloud-edge collaborative framework for real-time vision-language reasoning that is organized around the idea of Context Transfer: delayed outputs from a cloud Large Vision-LLM (LVLM) are not treated as expired answers, but as high-quality historical context for future edge-side Small Vision-LLM (SVLM) inference. The framework targets latency-sensitive streaming settings such as autonomous driving, gesture recognition, and video captioning, where inference must be continuous over temporally correlated frames and where cloud latency fluctuations make conventional collaboration strategies unreliable. In this formulation, delayed LVLM outputs provide both semantic history and visual guidance, and SpotVLM operationalizes this through a Context Replacement Module and a Visual Focus Module in a zero-shot deployment setting (Qian et al., 18 Aug 2025).

1. Problem formulation and deployment setting

SpotVLM is motivated by real deployment scenarios in which a system processes a stream of frames under strict latency constraints while answering visual-language queries in real time. The paper identifies three established deployment choices and their limitations. Cloud-only LVLMs offer stronger reasoning and perception, but network transmission, queueing, and inference latency make them unreliable for real-time use. Edge-only SVLMs are faster, but their reasoning and grounding accuracy is much lower. Partitioned LVLMs and offloading systems reduce compute burden, but still fundamentally depend on timely cloud returns; if a cloud response arrives several frames late, it can no longer directly serve the current frame (Qian et al., 18 Aug 2025).

The central observation is that video streams exhibit temporal correlation. A delayed answer about frame $T-d$ can therefore remain useful when frame $T$ is being processed. SpotVLM treats the cloud model’s delayed output as a reusable source of semantic and spatial priors rather than as a failed response. This differs from prior collaboration patterns that use cloud output as the final answer only if it arrives “in time,” and otherwise discard it or fall back to the small model. The framework is thus designed for environments in which latency is variable rather than merely high, and in which the utility of a cloud response depends not only on arrival time but also on its residual contextual value (Qian et al., 18 Aug 2025).

A plausible implication is that SpotVLM redefines the notion of timeliness in cloud-edge VLM systems: an answer may cease to be temporally valid as a direct response while remaining useful as structured historical evidence for subsequent reasoning.

2. Context Transfer as a collaboration paradigm

The paper terms the proposed paradigm Context Transfer. Its defining distinction is that LVLM output is reused as contextual prior even when it arrives too late to answer the original frame directly. In this view, collaboration is not limited to task partitioning or answer substitution; instead, the cloud model contributes delayed intelligence that persists into future inference (Qian et al., 18 Aug 2025).

Two mechanisms are emphasized as the basis for the performance gain. First, high-quality textual history matters. Small models benefit from better context, but if they generate their own history, that history is often noisy or low-quality. Replacing self-generated history with LVLM-generated context provides the SVLM with stronger semantic anchors. Second, grounding matters. LVLMs can identify salient image regions, and those regions can be reused to help the SVLM focus on the correct parts of the current frame while suppressing irrelevant visual tokens.

This formulation makes Context Transfer more than a latency workaround. It is a reuse strategy for delayed outputs in temporally correlated streams. The cloud model is not merely a backup executor for difficult frames; it becomes a producer of historical priors that shape downstream local reasoning. This suggests a broader systems interpretation in which the cloud provides asynchronous semantic supervision for the edge model without requiring additional fine-tuning.

3. Architecture: Context Replacement Module and Visual Focus Module

SpotVLM implements Context Transfer through two zero-shot modules: the Context Replacement Module (CRM) for textual context and the Visual Focus Module (VFM) for visual grounding (Qian et al., 18 Aug 2025).

CRM manages cached historical outputs from the cloud LVLM. The edge system stores responses from earlier frames according to

$$\mathcal{H}_{T-d}^{LVLM} \leftarrow LVLM(I_{T-d}, Q_{T-d}),$$

where $I_{T-d}$ and $Q_{T-d}$ are the image and question for frame $T-d$. If the cloud response is delayed by at most a threshold $\delta$, the corresponding SVLM historical content is replaced: $$\text{if } \Delta t = T - (T-d) \leq \delta,\quad \mathcal{H}_{T-d}^{SVLM} \leftarrow \mathcal{H}_{T-d}^{LVLM}.$$ Instead of naively concatenating all prior question-answer pairs, SpotVLM compresses the history into a shorter prompt,

$$\tilde{\mathcal{H}} = Compress(\mathcal{H}_{\leq T-1}),$$

and the current SVLM input becomes

$$Input_{T}^{SVLM} = \{ I_T, Q_T, \tilde{\mathcal{H}} \}.$$

In functional terms, CRM replaces noisy self-generated history with LVLM-derived historical context and thereby improves the semantic conditioning of edge inference.

VFM handles the visual side of collaboration. The LVLM output includes $T$0, namely attended image regions. These are exploited through two components. In Grounding-Guided Visual Representation (GGVR), the ROI from the historical frame is mapped to image patches: $T$1 Only task-relevant patches are retained, reducing visual tokens and computation while emphasizing salient content.

In Grounding-Tracked Visual Representation (GTVR), the ROI from frame $T$2 is used to help localize the corresponding object in frame $T$3. The category feature is computed as

$T$4

followed by cosine similarity between this feature and each patch of the current frame: $T$5 The category-wise similarity maps are averaged,

$T$6

converted into a visual weight,

$T$7

and used in a gated token update: $T$8

Taken together, CRM and VFM instantiate a dual transfer mechanism: textual history is improved by replacing low-quality local summaries with LVLM outputs, and visual processing is refined by propagating cloud-derived saliency into patch selection and feature weighting. The paper presents both modules as architectural augmentations rather than training-based adaptations (Qian et al., 18 Aug 2025).

4. Runtime pipeline and zero-shot operating mode

SpotVLM is presented as a zero-shot collaborative method using pretrained LVLM and SVLM backbones without additional fine-tuning. At runtime, the edge device performs two operations in parallel: immediate local SVLM inference for real-time output, and selective cloud upload of frames to the LVLM. Because cloud responses may arrive late, the edge system stores them and reuses them as historical context for later frames (Qian et al., 18 Aug 2025).

The runtime decision rule is explicit. If the cloud output arrives within 1 second, it can be used directly for the current frame. If it arrives too late, it is still cached and used as historical context for future SVLM inference. The LVLM produces both a textual answer $T$9 and an ROI $$\mathcal{H}_{T-d}^{LVLM} \leftarrow LVLM(I_{T-d}, Q_{T-d}),$$0, which are injected into the edge-side pipeline through CRM and VFM, respectively.

The paper states that, in implementation, two previous frames are kept as context, and the cloud feedback is treated as the farthest frame in the SVLM historical context. The algorithmic steps are described as follows: capture current frame $$\mathcal{H}_{T-d}^{LVLM} \leftarrow LVLM(I_{T-d}, Q_{T-d}),$$1 and query $$\mathcal{H}_{T-d}^{LVLM} \leftarrow LVLM(I_{T-d}, Q_{T-d}),$$2; run local SVLM inference immediately; optionally upload selected frames to the cloud LVLM; when an LVLM result returns, cache its answer and ROI with the original timestamp and either replace the corresponding historical context if the latency is below threshold or store it for future contextual use; compress historical dialogue into a short prompt; and use LVLM ROI to select relevant patches, compute similarity-based focus for the current frame, and update visual tokens before SVLM reasoning (Qian et al., 18 Aug 2025).

Because the method introduces no new learning objective or fine-tuning loss, its contribution is architectural and algorithmic rather than optimization-based. This suggests that its empirical behavior depends primarily on backbone quality, latency regime, and the temporal stability of semantic and spatial cues across frames.

5. Experimental protocol, tasks, and baselines

The evaluation covers three real-time tasks across four datasets under real-time constraints. The tasks are real-time multi-object recognition, real-time gesture recognition, and real-time video frame caption / action recognition (Qian et al., 18 Aug 2025).

For multi-object recognition, the datasets are nuScenes and BDD100K MOT. The nuScenes configuration uses 150 scenes, 20 s each, sampled at 1 FPS, with 10 object categories. The BDD100K MOT configuration uses 200 scenes, 40 s each, also sampled at 1 FPS, with 10 object categories. Gesture recognition is evaluated on IPN Hands, which contains 14 gesture classes, 52 videos, sampled at 2 FPS. Video frame caption / action recognition is evaluated on Actions for Cooking Eggs (ACE), with 8 cooking actions, 10 videos, sampled at 1 FPS (Qian et al., 18 Aug 2025).

For multi-object recognition, the reported metrics are Micro-F1, Macro-F1, and 0-1 Exact Match (0-1 EM). For gesture recognition and video frame captioning, the metric is Accuracy (Acc). A frame is treated as having empty output if the system cannot return a valid result within 1 second. This evaluation protocol directly couples model quality with response-time constraints and therefore reflects the intended operating regime of the framework.

The baselines are SVLM Only (Local), LVLM Only (Cloud), Distributed VLM, LAECIPS, ADAS, and SpotVLM. In the main comparison, all methods use the same backbone pair: Qwen2.5-VL-3B as SVLM and Qwen2.5-VL-72B as LVLM. The edge implementation uses a GeForce RTX 4090 (24 GB) (Qian et al., 18 Aug 2025).

6. Quantitative results, ablations, and latency behavior

The reported main result is that SpotVLM is best across all evaluated datasets and metrics (Qian et al., 18 Aug 2025). The comparative results are summarized below.

Dataset / task	Baselines	SpotVLM
nuScenes	SVLM only: 58.19 / 37.70 / 16.10; LVLM only: 45.76 / 32.04 / 20.75; LAECIPS: 59.84 / 39.64 / 24.29; ADAS: 60.00 / 41.35 / 18.11	69.96 / 48.19 / 27.32
BDD100K MOT	SVLM only: 62.12 / 31.06 / 6.37; LVLM only: 49.65 / 25.85 / 20.28; LAECIPS: 62.47 / 32.14 / 21.05; ADAS: 63.87 / 33.12 / 10.63	71.19 / 36.26 / 23.88
IPN Hand	SVLM only: 36.41; LVLM only: 23.91; LAECIPS: 39.40; ADAS: 39.27	45.11
ACE	SVLM only: 23.43; LVLM only: 61.14; LAECIPS: 48.71; ADAS: 55.43	62.75

On nuScenes, the ablation study isolates module effects by uploading all frames. The baseline parallel inference yields 66.87 / 43.07 / 24.39. Adding CRM only yields 71.46 / 48.64 / 29.98. Adding CRM + GGVR yields 72.15 / 50.85 / 30.93. Adding CRM + GGVR + GTVR yields 73.02 / 51.56 / 33.17. The paper’s interpretation is that CRM contributes the largest gain, confirming the value of better historical textual context; GGVR improves efficiency by trimming irrelevant visual tokens; and GTVR further improves consistency in tracking object categories across frames (Qian et al., 18 Aug 2025).

An adaptability study reports that the method generalizes across model combinations, including MiniCPM-o-2.6-int4 + Qwen2.5-VL-72B and Qwen2.5-VL-3B + GPT-4o, with SpotVLM improving over the baseline in both cases. This indicates that the collaboration strategy is not specific to one model pair. A plausible implication is that the method’s gains derive from its interaction protocol rather than from an idiosyncratic coupling of specific architectures.

The latency analysis on nuScenes simulates different cloud delays. At 1 second latency, SpotVLM matches the LVLM upper bound. From 2 seconds onward, LVLM results can only be used as context, but SpotVLM still beats SVLM-only. Performance gradually declines as latency grows, but remains better than the local model even at 9 seconds delay. These findings support the claim that delayed cloud outputs remain useful as historical priors when frames are temporally correlated (Qian et al., 18 Aug 2025).

7. Assumptions, limitations, and significance

SpotVLM depends on several stated assumptions. It relies on temporal continuity between adjacent frames, since delayed outputs are useful only when scenes evolve smoothly. It assumes that the LVLM provides not only answers but also ROI localization, which the framework can exploit. It also assumes that a delay threshold $$\mathcal{H}_{T-d}^{LVLM} \leftarrow LVLM(I_{T-d}, Q_{T-d}),$$3 is available for deciding whether an LVLM output is still temporally useful, and that the edge device can perform feature matching and token updates efficiently enough for real-time operation (Qian et al., 18 Aug 2025).

The paper also identifies limitations. It does not present a learned adaptive scheduler; upload scheduling is relatively simple and based on latency history or thresholding. Performance degrades as latency becomes very large because the relevance of past context weakens. The method is evaluated without extra fine-tuning, which supports zero-shot deployment but may leave performance untapped relative to learned coordination policies. Visual tracking relies on similarity between historical and current features and may therefore be less reliable under abrupt viewpoint or scene changes (Qian et al., 18 Aug 2025).

Within the paper’s framing, the principal practical implication is a design principle for latency-aware VLM systems: if cloud results are late, they should not be discarded but converted into context. The stated consequences include more robust real-time reasoning under unstable networks, better utilization of expensive cloud inference, lower need for frequent cloud calls, and a path toward collaborative VLM systems that are not only accuracy-aware but latency-aware. More generally, SpotVLM shifts cloud-edge collaboration from synchronous answer delivery toward asynchronous contextual augmentation, with delayed LVLM outputs functioning as reusable semantic and grounding priors rather than as binary successes or failures (Qian et al., 18 Aug 2025).