---
title: 'TSV360 Dataset: Text-Driven Saliency in 360 Videos'
url: https://www.emergentmind.com/topics/tsv360-dataset
type: topic
---

# TSV360 Dataset: Text-Driven Saliency in 360 Videos

Searching arXiv for the primary paper and closely related context papers.
Querying the arXiv API directly for the cited paper IDs to verify titles and context.
TSV360 is a dataset for text-driven saliency detection in 360-degree videos. Introduced together with the TSalV360 method, it provides aligned triplets of equirectangular projection (ERP) frames, textual descriptions of salient objects or events, and ground-truth saliency maps, thereby supporting both training and objective evaluation of customized, text-conditioned saliency prediction in omnidirectional video [2509.26208]. Its central contribution is to bridge two previously separate lines of work: 360-degree video saliency datasets built from gaze or head-movement signals, and text-driven saliency resources developed for still images or non-omnidirectional video.

## 1. Task definition and motivation

TSV360 is designed specifically for the task of text-driven saliency detection in 360° videos. In this setting, saliency is conditioned not only on visual conspicuity but also on a user-provided textual description of the desired object or event. The motivating examples given for this paradigm are requests such as “focus on the drummer” or “watch the orange cat,” which require a model to privilege semantically specified content rather than all globally salient regions [2509.26208].

This framing differs from earlier 360° saliency prediction work. Existing datasets such as Sports-360, VR-EyeTracking, PVS-HM, and Pano2Vid support training and evaluation with visual-only signals, including fixations and head movements, but do not provide textual supervision. Conversely, text-driven saliency datasets such as SJTU-TIS and Ego4D target images or egocentric video with text, rather than the geometry and viewing mechanics of omnidirectional video. TSV360 is therefore positioned as a response to a concrete gap: prior 360° saliency methods seek all salient regions, whereas text-conditioned navigation and summarization in 360° video require saliency maps that reflect user intent.

A recurrent misconception is that text-driven saliency necessarily requires saliency labels obtained under text-prompted viewing. TSV360 does not adopt that strategy. Its fixation data are generic rather than text prompted, and the dataset associates these unbiased saliency maps with semantically relevant descriptions after the fact. The stated rationale is to avoid biasing attention toward non-salient content while still enabling text-grounded modeling.

## 2. Composition, scale, and data organization

The dataset contains 160 360° videos, each up to 60 seconds long, with diverse content including music shows, sports games, short films, and documentaries [2509.26208]. These videos are sourced from VR-EyeTracking and Sports-360. The original fixation data were collected with HTC VIVE head-mounted displays, using head and gaze directions from 45 participants in VR-EyeTracking and eye fixations from 27 participants in Sports-360.

From these sources, TSV360 provides approximately 16,000 triplets. Each triplet comprises an ERP frame, a textual description of the salient object or event visible at that moment, and a corresponding ground-truth saliency map. The dataset is organized at the frame level with associated text and saliency, although it also supports sequence-based modeling. In the accompanying experiments, input sequences of $F = 8$ ERP frames are used, and the saliency map is predicted for the last frame of the sequence. Thus, the annotated frame functions as the temporal anchor, while the surrounding sequence supplies context.

The experimental setup reported for the dataset uses videos downsampled to 16 fps, ERP inputs at resolution $960 \times 1920$, and ground-truth saliency maps at $480 \times 960$. The paper states that these settings reflect the experimental configuration and are also natural resolutions for ERP content and corresponding saliency maps. Officially, TSV360 does not prescribe a single fixed train/validation/test partition; instead, the paper adopts five-fold cross-validation, using 80% of the data for training and 20% for testing in each fold.

## 3. Annotation pipeline and label construction

The annotation pipeline begins with fixation maps from VR-EyeTracking and Sports-360. These are standardized by applying a Gaussian filter with fixed standard deviation $\sigma = 5^\circ$, following prior work, in order to ensure consistent saliency maps across videos [2509.26208]. A fine-tuned 2D production or summarization pipeline is then applied to ERP videos and their saliency maps to separate distinct events and generate individual event-specific saliency maps.

Several concrete modifications define this stage. DBSCAN is replaced with HDBSCAN, described as better for variable-density clusters and requiring no parameter tuning. Haversine distance is used to create spatial-temporal sub-volumes, with the stated purpose of better preserving spatial relationships in 360° scenes than Euclidean distance. Salient regions are detected by clustering intensity and distance; spatio-temporal sub-volumes are formed; abrupt changes are reduced by filling missing frames; fields of view around salient regions are extracted; and 2D fragments are stitched chronologically. A fine-tuned step then produces distinct saliency maps for each salient event within a scene from the original fixation-derived saliency map.

Textual descriptions are obtained from the resulting 2D event-focused video fragments. Each fragment is densely captioned per second using LLaVA-Next-7B with the prompt: “Briefly describe what is depicted in the video, using one sentence.” Quality control is substantial. Approximately 49% of source videos were discarded because they lacked multiple identifiable salient objects or events and therefore offered insufficient semantic diversity. Approximately 40% of generated saliency-map-and-caption pairs were removed when no identifiable object or event was present. About 65% of the remaining captions were manually curated to remove unclear or repetitive phrasing and to increase semantic diversity and relevance.

The dataset also applies two augmentation procedures. Temporal window-shifting samples multiple overlapping ERP-frame sequences for briefly appearing events, increasing the number of training examples. Text paraphrasing adds semantically aligned paraphrases for the same event to diversify textual expression. The paper further states that inter-annotator agreement is not applicable: saliency derives from aggregated human fixation records, while captions are first machine-generated and then curated, so quality control relies on filtering and manual curation rather than annotator-consensus statistics.

## 4. Geometric representation and 360°-specific processing

TSV360 is built around ERP imagery, but its intended use explicitly accounts for spherical distortion. For 360° processing, each ERP frame is projected into $T = 18$ tangent images of size $224 \times 224$ pixels, each with $\mathrm{FOV} = 80^\circ$ [2509.26208]. These tangent images function as local viewports over the sphere and are intended to mitigate ERP distortion. One configuration reported in the implementation samples four latitudes, $(-67.5^\circ, -22.5^\circ, 22.5^\circ, 67.5^\circ)$, with 3, 6, 6, and 3 patches respectively, for a total of 18 viewports.

Decoder outputs have spatial size $56 \times 56$ per tangent view. These outputs are re-projected to ERP resolution $240 \times 480$ and then upsampled bilinearly to $480 \times 960$ so that they align with the ground-truth saliency map resolution. On the text side, annotations are single-sentence descriptions generated per second of event-focused 2D clips. When CLIP is used, the model consumes tokenized local text features with length $L_t = 77$. The paper does not specify file formats or a JSON schema, and it states only that textual annotations are tied to timestamps and event segments resulting from the event extraction pipeline.

The accompanying TSalV360 method uses this dataset to support a particular multimodal architecture: CLIP-based global and local vision-language features, similarity estimation, and viewport spatio-temporal cross-attention. The similarity estimation module computes cosine similarity between global visual features and global textual features, producing scores that weight local visual features. Viewport spatio-temporal cross-attention first performs temporal attention across frames per viewport, then spatial attention across viewports per frame, and finally cross-attention in which visual tokens query textual tokens. The cross-attention formulation given in the paper is

$$
\begin{aligned}
&\mathbf{Q = W^Q} \cdot {\mathbf{Z'}^{(m)}_i , \quad \text{where } i \in [1, \dots, N] \\
&\mathbf{K, V =  W^K} \cdot \mathbf{T_L}_j, \mathbf{W^V} \cdot \mathbf{T_L}_j  \quad \text{where } j \in [1, \dots, L_t] \\
&\text{CrossAttention}(\mathbf{Q, K, V}) = \text{SoftMax}(\mathbf{QK^\top} / \sqrt{d_k}) \mathbf{V} \cdot \mathbf{W^O}
\end{aligned}
$$

where $N = F \times T$ is the total number of visual tokens, $\mathbf{W^Q}$, $\mathbf{W^K}$, and $\mathbf{W^V}$ are learned projections, $\mathbf{W^O}$ is the output projection, $d_k$ is the textual feature dimension, and $\mathbf{T_L}$ denotes local textual features.

## 5. Evaluation protocol and reported results

The evaluation protocol for TSV360 uses five-fold cross-validation, with 80% of the data for training and 20% for testing in each fold, and results reported as averages across folds [2509.26208]. The reported metrics are CC (Correlation Coefficient), SIM (Similarity), and KLD (Kullback–Leibler Divergence), computed per predicted-versus-ground-truth saliency-map pair.

The paper compares a visual-only baseline, SalViT360, against the text-conditioned TSalV360 model. SalViT360 achieves CC $0.382 \pm 0.024$, SIM $0.189 \pm 0.013$, and KLD $18.486 \pm 0.427$. TSalV360 achieves CC $0.541 \pm 0.027$, SIM $0.395 \pm 0.018$, and KLD $11.718 \pm 0.650$. The paper attributes the improvement to several design choices used together: CLIP as a vision-language backbone for visual and text features, replacement of viewport self-attention with viewport spatio-temporal cross-attention (VSTCA), introduction of similarity estimation (SimEst) to weight visual features by text alignment, adoption of sigmoid activation in the decoder, and the addition of hierarchical skip connections.

These results are presented as evidence that TSV360 is sufficient not merely for dataset construction but also for benchmarking customized text-driven saliency detection in 360° videos. A plausible implication is that the dataset’s event-specific coupling of saliency and text enables objective discrimination between models that merely predict generic attention and models that respond to semantic intent.

## 6. Relation to prior resources, access, and limitations

TSV360 is closely related to earlier 360° saliency datasets but differs in the supervision it provides. Sports-360, VR-EyeTracking, PVS-HM, and Pano2Vid supply gaze- or head-movement-derived saliency signals without text, so they cannot directly support training or evaluation of text-driven saliency models [2509.26208]. The dataset also has a methodological connection to 360-VSumm, a separate dataset for 360-degree video summarization, because the fine-tuned event-extraction stage is derived from a 2D production or summarization pipeline; 360-VSumm itself addresses 360° video summarization into 2D form rather than text-driven saliency [2406.02991].

The literature represented here also reveals a nomenclature issue. The acronym “TSV360” is not exclusive to the text-driven saliency dataset of TSalV360: the alias is also reported for a time-scale modification audio-quality dataset hosted on IEEE Dataport [2006.00848]. In practice, precise citation by title or arXiv identifier is therefore important to avoid ambiguity.

Access details for TSV360 remain limited in the paper. The code and experimental implementation for TSalV360 are publicly available at `https://github.com/IDT-ITI/TSalV360`, but the paper does not provide a direct download URL or license details for the dataset itself. It explicitly advises researchers to consult the repository or contact the authors regarding availability and usage terms.

The main limitations are also stated directly. The saliency maps are text-agnostic because they are derived from generic fixation behavior rather than explicit search tasks; this preserves true visual saliency but may differ from attention patterns under prompted viewing. Captions are machine-generated and then curated, so they may still reflect biases of both the captioning model and human curators. Event imbalance is mitigated through temporal window-shifting and paraphrasing, yet extremely rare events may remain underrepresented. Exact distributions for sentence length, category counts, and file layout are not reported. Recommended use therefore follows the paper’s protocol: sequence-based text-conditioned saliency prediction with ERP-to-tangent preprocessing, reprojection back to ERP space, and evaluation with CC, SIM, and KLD under five-fold cross-validation [2509.26208].

Source: https://www.emergentmind.com/topics/tsv360-dataset