STER-VLM: Spatio-Temporal Traffic VLM

• STER-VLM is a traffic-focused vision–language framework that decomposes captioning into spatial-invariant and temporal-variant components to enhance pedestrian video understanding.
• It employs phase-aware frame selection and best-view filtering to efficiently process multi-camera, multi-phase traffic videos while reducing computational load.
• The model leverages lightweight LoRA-based adaptation of Qwen2.5-VL-7B for structured captioning and visual question answering through curated visual and textual prompts.

STER-VLM, introduced in “STER-VLM: Spatio-Temporal With Enhanced Reference Vision-LLMs” (Nguyen-Nhu et al., 19 Aug 2025), is a traffic-focused vision–language modeling framework for pedestrian-centric video understanding. It is designed to address two limitations identified for generic VLMs in real-world traffic analysis: insufficient fine-grained spatio-temporal understanding in multi-camera, multi-phase videos, and heavy computational demands when many frames and views are processed naïvely. The framework combines caption decomposition, temporal frame selection with best-view filtering, reference-driven understanding, and curated visual/textual prompts, and is built around LoRA-based adaptation of Qwen2.5-VL-7B for structured captioning and visual question answering.

1. Scope, task setting, and conceptual definition

STER-VLM operates on pedestrian-centric traffic videos segmented into five phases: prerecognition, recognition, judgment, action, and avoidance (Nguyen-Nhu et al., 19 Aug 2025). Its inputs include raw videos, per-phase segments, per-frame pedestrian and vehicle bounding boxes, 3D gaze vectors, and ground-truth captions and QAs. Its outputs are detailed structured captions and answers to VQA queries.

The framework is motivated by a specific failure mode of conventional multi-frame VLM usage. According to the paper, too many frames can cause memory overflow and temporal inconsistencies, while too few frames can produce short or under-informative captions. The method also targets the computational overhead of prior modular systems that require training and deploying multiple large models. STER-VLM addresses this by using a compact pipeline that separates spatial and temporal content, restricts visual input to a minimal but sufficient subset of frames and views, and injects reference text as auxiliary guidance rather than architectural expansion (Nguyen-Nhu et al., 19 Aug 2025).

A central design choice is the decomposition of captioning into two semantically disjoint components: a spatial-invariant component covering appearance, static environment, weather, and road conditions, and a temporal-variant component covering actions, relative positions, motion, and phase-specific dynamics. This decomposition is aligned to WTS annotation conventions and functions as the organizing principle for the rest of the system.

2. System architecture and model organization

The architecture comprises two specialized captioners and a composition model, all based on Qwen2.5-VL-7B with LoRA adaptation (Nguyen-Nhu et al., 19 Aug 2025). One captioner is trained for spatial-invariant description, the other for temporal-variant description, and a composition model merges and refines the two outputs into a single structured caption. The paper also describes a two-stage training strategy in which captioning is learned first, and the resulting visual-text grounding is then transferred to VQA.

The adaptation regime is deliberately lightweight. Qwen2.5-VL-7B is fine-tuned via LoRA with rank 64, and only MLP layers are updated. The spatial and temporal captioners are trained independently for 3 epochs with batch size 4, 4-step gradient accumulation, AdamW learning rate $2 \times 10^{-4}$, gradient checkpointing, cosine schedule, and warm-up ratio 10%. The composition model uses gradient accumulation of 6 steps and learning rate $1 \times 10^{-4}$. VQA is initialized from the best captioning checkpoints and trained for 1 epoch at learning rate $5 \times 10^{-5}$. All experiments are run in bf16 on one NVIDIA RTX A6000 GPU (Nguyen-Nhu et al., 19 Aug 2025).

The paper does not introduce new loss terms such as contrastive, alignment, or reference-grounding objectives. Adaptation is described as instruction-tuning style fine-tuning with LoRA. This makes the method architecturally conservative: performance gains are attributed primarily to data structuring, frame/view selection, prompting, and reference injection rather than to new trainable fusion modules or auxiliary losses.

3. Caption decomposition, prompting, and reference-driven guidance

Caption decomposition is performed by Qwen2.5-72B-Instruct, which acts as a caption parser under few-shot prompting (Nguyen-Nhu et al., 19 Aug 2025). For each scenario and phase, the system collects captions, instructs the parser to extract spatial-invariant attributes into one consolidated caption, and separately extracts temporal-variant behavior and relations into a second caption. The resulting image–text pairs are then used to train the two specialist captioners.

The spatial-invariant component includes attributes such as gender, age group, height, clothing colors and types, weather, road material and condition, traffic volume, and the presence of street lights or sidewalks. The temporal-variant component includes action, attention, and location, such as pose, movement, gaze direction, awareness, and relative position to vehicles or obstacles. The paper’s qualitative example explicitly separates a description like “A male pedestrian in his 30s, approximately 170 cm tall… dressed in a white jacket… The weather was clear… road made of asphalt…” from a temporal description such as “He was observed in a squatting position… diagonally positioned to the left in front of a vehicle… appeared unaware of the vehicle…” (Nguyen-Nhu et al., 19 Aug 2025).

STER-VLM further uses curated visual and textual prompts. Visual-aware prompting adds colored bounding boxes, red circular dots, and rendered 3D gaze lines to direct attention to relevant regions. Textual role-play prompts provide structured hints. Ablation results indicate that the spatial model benefits most from textual hints alone, scoring 31.847, while visual-only prompting scores 17.113 and combined prompting scores 30.656. For the temporal model, combined visual and textual prompting scores 29.185, compared with 25.245 for visual-only and 28.964 for textual-only. Removing Appearance hints causes the largest spatial drop, $-9.2$, while removing Action causes the largest temporal drop, $-1.29$ (Nguyen-Nhu et al., 19 Aug 2025).

Reference-driven guidance is implemented with Qwen2.5-VL-72B, which generates per-frame textual references summarizing key objects, relations, and actions. These references are appended to the prompts as hints. The paper emphasizes that they do not override model outputs and do not introduce new trainable modules. Their effect is measurable: on the WTS+BDD combined setting, adding references improves BLEU-4 from 0.232 to 0.234, METEOR from 0.441 to 0.443, ROUGE-L from 0.430 to 0.431, CIDEr from 0.984 to 1.023, and caption score from 30.061 to 30.257 (Nguyen-Nhu et al., 19 Aug 2025).

4. Temporal frame selection and best-view filtering

A distinctive feature of STER-VLM is its phase-aware frame selection strategy. The method uniformly samples three frames per phase to maintain temporal coverage while constraining memory and compute. For multi-camera inputs, it performs best-view filtering by selecting frames and cameras with the largest informative content according to bounding-box area criteria (Nguyen-Nhu et al., 19 Aug 2025).

The paper formalizes this selection as

$$\mathbf{I}_{\text{phase}} = \begin{dcases} [\mathbf{I}_{best},\ \mathbf{I}_{addition}] & \text{if phase} \in [1, 2, 3] \ [\mathbf{I}_{1},\ \mathbf{I}_{2},\ \mathbf{I}_{3}] & \text{if phase} \in [4, 5] \end{dcases}$$

For spatial-invariant captioning, the system selects the first frames of phases 1–2 for environment context and the frames with the largest pedestrian bounding box in phases 3–4 for appearance. For temporal captioning, phases 4–5 use three uniformly sampled frames from the camera with the largest sum of pedestrian and vehicle bounding-box areas. In phases 1–3, the pair $[\mathbf{I}_{best},\ \mathbf{I}_{addition}]$ is used, where $\mathbf{I}_{best}$ has the largest pedestrian bounding box and $\mathbf{I}_{addition}$ has the largest sum of pedestrian and vehicle bounding-box areas (Nguyen-Nhu et al., 19 Aug 2025).

The computational cost of best-view selection is described as linear in the number of frames times cameras per phase. The paper’s qualitative analysis states that single-frame inference tends to miss appearance details, whereas selected multi-frame inputs produce more coherent and detailed captions. This suggests that STER-VLM treats temporal coverage not as dense frame ingestion but as constrained, annotation-aware evidence selection.

5. Datasets, metrics, and empirical performance

The framework is evaluated on WTS and BDD PC 5K (Nguyen-Nhu et al., 19 Aug 2025). WTS contains 249 scenarios and 810 videos from dashcams and traffic surveillance, with five phases per video, 3D gaze, pedestrian and vehicle bounding boxes, detailed captions, and per-phase QA. BDD PC 5K consists of 3,402 pedestrian-centric dashcam videos filtered from BDD100K and similarly organized into five phases with per-phase captions and QA pairs.

Evaluation uses BLEU-4, METEOR, ROUGE-L, and CIDEr. The captioning score is a weighted average that gives full weight to BLEU-4, METEOR, and ROUGE-L and a lower weight to CIDEr, averaged across internal and external subsets. The final evaluation score combines captioning and VQA accuracy equally and scales the result to 100 (Nguyen-Nhu et al., 19 Aug 2025).

On WTS, the baseline versus full pipeline comparison shows CIDEr increasing from 0.822 to 0.938, while caption score changes from 29.160 to 28.595. On BDD, BLEU-4 improves from 0.246 to 0.252, METEOR from 0.449 to 0.462, ROUGE-L from 0.429 to 0.452, CIDEr from 1.070 to 1.108, and caption score from 30.792 to 31.920. On the combined setting, CIDEr increases from 0.946 to 1.023 and caption score from 29.975 to 30.258 (Nguyen-Nhu et al., 19 Aug 2025). The mixed behavior on WTS reflects a metric trade-off later discussed by the authors: decomposition increases semantic richness and CIDEr but can reduce BLEU-4, METEOR, or ROUGE-L in internal subsets because phrasing diverges from reference surface forms.

The VQA stage benefits substantially from the two-stage training protocol. Direct fine-tuning yields 75.367, whereas the captioning-to-VQA transfer strategy yields 81.507, corresponding to $+6.14\%$ (Nguyen-Nhu et al., 19 Aug 2025). In the AI City Challenge 2025 Track 2, STER-VLM obtained a test score of 55.655 and ranked 7th as team AIO_GENAI4E, with neighboring teams scoring between 52.148 and 60.039.

6. Relation to other uses of the term “STER-VLM”

The exact name “STER-VLM” is explicitly attached to the traffic-video framework of (Nguyen-Nhu et al., 19 Aug 2025), but the same string is used interpretively in several adjacent arXiv discussions to denote broader VLM design patterns. This creates a recurring ambiguity in secondary usage.

Usage of “STER-VLM”	Paper	Mapping described in the source
Explicit named framework	“STER-VLM: Spatio-Temporal With Enhanced Reference Vision-LLMs” (Nguyen-Nhu et al., 19 Aug 2025)	Traffic-focused captioning and VQA with decomposition, best-view filtering, references, and curated prompts
Spatio-Temporal Embodied Reasoning VLM	“VLM-3R” (Fan et al., 26 May 2025)	Interpreted as a VLM with implicit 3D tokens, SVVF, and temporal spatial reasoning from monocular video
Selective/Semantic Trajectory Experience Replay with a VLM	“VLM-Guided Experience Replay” (Sharony et al., 2 Feb 2026)	Interpreted as clip-level semantic replay prioritization using a frozen VLM
Student–Teacher Enhanced Refinement VLM	“Annotation Free Spacecraft Detection and Segmentation using Vision LLMs” (Hicsonmez et al., 4 Feb 2026)	Interpreted as teacher pseudo-labeling, refinement, and student distillation
Typographical variant of SteerVLA	“SteerVLA” (Gao et al., 9 Feb 2026)	Interpreted as VLM-guided steering of a VLA driving policy via meta-actions and reasoning traces
Closely related streaming 3D VLM need	“Stream3D-VLM” (Yu et al., 5 Jun 2026)	Online 3D spatial understanding with streaming control, VSFI, and GAVC

A common misconception is therefore to treat “STER-VLM” as a single stable acronym across the literature. The available sources suggest instead that one should distinguish between the explicitly named traffic-analysis model of (Nguyen-Nhu et al., 19 Aug 2025) and broader retrospective uses of the term as shorthand for spatio-temporal, steerable, replay-guided, or teacher–student VLM systems.

7. Limitations, trade-offs, and research position

The limitations identified for STER-VLM are primarily methodological rather than architectural. First, decomposition introduces metric trade-offs: CIDEr improves, but BLEU-4, METEOR, and ROUGE-L can decline in internal subsets because the generated wording departs from reference phrasing (Nguyen-Nhu et al., 19 Aug 2025). Second, prompt sensitivity remains significant. Visual prompts can degrade performance when bounding boxes are incomplete or missing, and the spatial model in particular benefits more from textual hints than from visual-only prompting.

Third, reference reliability is explicitly limited. The large VLM generating per-frame references can misinterpret subtle details, so the framework uses references only as hints. Fourth, the fixed two-part decomposition is less flexible than finer categorical schemes such as the five-way split used in Divide and Conquer. The paper describes this as a simplicity–expressiveness trade-off: STER-VLM reduces the number of models and training complexity, but a fixed spatial-versus-temporal partition may constrain expressiveness (Nguyen-Nhu et al., 19 Aug 2025).

In related-work terms, the framework is positioned against CityLLaVA, Divide and Conquer, LLaVA-ST, and accident-understanding systems such as SeeUnsafe, AccidentGPT, SafePLUG, and ScVLM. Relative to these, STER-VLM is characterized by a simpler spatial/temporal split, phase-aware frame sampling, best-view filtering, and reference-driven prompting without heavy architectural changes. The authors identify future directions including automatic reference validation or consistency checking, dynamic decomposition that adapts to scenario complexity, end-to-end alignment losses for reference grounding, improved view selection under extreme occlusions, and expanded multimodal inputs such as LiDAR or HD maps as in MAPLM (Nguyen-Nhu et al., 19 Aug 2025).