Mid-360: 360° Video Summarization Benchmark

• Mid-360 is a framework that transforms 360° videos into 2D summaries by selecting salient events and generating undistorted previews.
• It leverages the 360-VSumm dataset and adapts state-of-the-art 2D summarizers, demonstrating substantial performance gains upon retraining.
• The methodology emphasizes balancing coverage and redundancy through constrained subset selection and saliency-aware feature reweighting.

Searching arXiv for the specified paper and the baseline methods to ground the article in current paper metadata. Mid-360, as used in the 360-VSumm study, denotes a 360° video summarization setting in which 360° video content is transformed into concise 2D-video summaries that can be consumed via traditional devices such as TV sets and smartphones. In this setting, summarization is performed not directly on raw equirectangular video but on 2D previews produced from salient events, and the resulting summaries are trained and evaluated against human-generated annotations. The associated work introduces the 360-VSumm dataset, formulates summarization as a constrained subset-selection problem, adapts two state-of-the-art 2D-video summarization methods as baselines, and uses their behavior to characterize the differences between flat-video and genuinely 360° summarization workflows (Kontostathis et al., 2024).

1. Conceptual scope

The central object in Mid-360 is the conversion of 360° video into a concise 2D summary. The cited work defines this as the transformation of 360-degree video content to concise 2D-video summaries that can be consumed via traditional devices, such as TV sets and smartphones (Kontostathis et al., 2024). This places Mid-360 at the intersection of panoramic video understanding, view selection, and temporal summarization.

A key design choice is that the summarization target is a 2D video assembled from selected narrow field-of-view material rather than a summary expressed in native spherical coordinates. The paper therefore treats the problem as a bridge between 360° capture and conventional video consumption. This also explains why the initial baselines are adapted from 2D-video summarization rather than from native 360° scene-understanding pipelines.

The work explicitly argues that 360° video summarization is not a trivial “drop-in” extension of 2D summarization. Off-the-shelf models pre-trained on flat-video corpora do not transfer effectively to multi-event spherical inputs, and this observation functions as one of the main empirical motivations for the Mid-360 setting (Kontostathis et al., 2024).

2. Dataset construction and annotation regime

The dataset underlying Mid-360 is 360-VSumm, built on top of the publicly available VR-EyeTracking corpus. From the original 208 high-definition 360° clips, each 20–60 seconds long and covering indoor/outdoor scenes, sports, concerts, documentaries, and short films, the pipeline performs salient-event detection by clustering high-saliency pixels in equirectangular frames, grouping them into space-time sub-volumes, cropping 4:3 FOV fragments, and sewing them into a concise 2D “preview” video. The corpus is then filtered for dynamic content longer than 1 minute, yielding 40 two- to four-minute 2D videos (Kontostathis et al., 2024).

Ground truth is obtained from 15 paid annotators aged 24–44. Each annotator watches every 2D video and selects the “most important” 15% of its sub-fragments. Videos are pre-segmented into 2-second clips so that annotators can pick and revise individual pieces until their summary meets the 15% time budget. The annotation workflow is mediated by a lightweight Java Swing-based tool with four explicit functions: a row of clickable dashes representing each 2-second sub-fragment beneath a video player; an always-visible indicator of how many fragments must be chosen, colored red or gray if the annotator is over- or under-budget; a “Check Summary” button that stitches and plays only the chosen clips in a smaller preview window; and standard play/pause, frame-step, 10-frame skip, and scroll bar controls for fine-tuning fragment boundaries before saving a binary selection mask to disk (Kontostathis et al., 2024).

From these annotations, the dataset derives four kinds of per-video outputs: 15 binary frame-level summaries with values $1=$ selected and $0=$ not; an averaged “mean” ground-truth score between 0 and 1 for supervised training; the list of event fragments and sub-fragments detected by the pipeline; and the original saliency score of each frame, averaged over its salient regions (Kontostathis et al., 2024).

This annotation design is significant because it yields both discrete human summaries and continuous supervision signals. A plausible implication is that the dataset supports both strict subset-selection evaluation and softer importance-learning objectives without requiring separate annotation protocols.

3. Formalization as subset selection

Mid-360 casts 360° summarization as a formal selection problem over a 2D video produced from a 360° clip. If the 2D video consists of $N$ contiguous sub-fragments indexed by $i=1,\ldots,N$, and $f_i \in \mathbb{R}^d$ is the visual embedding of the $i$-th sub-fragment, then summarization consists of choosing a subset $S \subseteq \{1,\dots,N\}$ of size $|S|=k$, with $k=0.15N$, such that the selected clips best cover the important content. The paper uses GoogleNet pool5 features with $d=1{,}024$ at 2 fps (Kontostathis et al., 2024).

The objective is written as

$0=$0

where $0=$1 is a frame-level importance score and $0=$2 quantifies visual overlap (Kontostathis et al., 2024).

This formulation exposes the canonical tension between coverage and redundancy. The representativeness term favors fragments with high importance, while the redundancy penalty discourages selecting visually overlapping content. In Mid-360, that tension is particularly relevant because 360° footage frequently contains multiple simultaneous or partially overlapping events, making it insufficient to optimize only for saliency or only for diversity.

The two baseline methods instantiate this objective differently. Supervised PGL-SUM learns $0=$3 by minimizing cross-entropy against human labels:

$0=$4

Unsupervised CA-SUM chooses $0=$5 by minimizing the reconstruction loss of the full video using only the selected frames plus a diversity regularizer:

$0=$6

where $0=$7 projects onto the subspace spanned by $0=$8 (Kontostathis et al., 2024).

4. Adapted baseline architectures

The baseline systems in Mid-360 are two attention-based state-of-the-art 2D summarizers applied to the 2D “flattened” videos. The first is PGL-SUM, attributed to Apostolidis et al. and described as combining global and local multi-head self-attention to model both coarse and fine dependencies, plus sinusoidal positional encoding. In the reported adaptation, the weights are re-initialized with Xavier with gain $0=$9 and bias $N$0, and the best results are obtained with 2 global heads, 8 local heads, and 8 local attention modules. The second is CA-SUM, also attributed to Apostolidis et al., which uses a “concentrated” attention mechanism focused on non-overlapping diagonal blocks of size 70, with a loss that trades off summary length against diversity via a regularizer $N$1 (Kontostathis et al., 2024).

The modifications for spherical input are intentionally limited. Rather than feeding raw equirectangular frames, which contain severe distortion at poles, the pipeline pre-extracts salient 4:3 NFOV crops, namely the 2D previews, so that both models see only “normal”, undistorted imagery. The study also experiments with saliency-aware feature reweighting, using each frame’s ground-truth saliency $N$2 to transform the embedding as

$N$3

This mechanism is the basis of the reported saliency-aware variants (Kontostathis et al., 2024).

The common training protocol is fixed across methods: videos are downsampled to 2 fps; GoogleNet pool5 features are extracted; optimization uses Adam; the batch is the full video; and training runs for 600 epochs. For PGL-SUM the hyper-parameters are $N$4, dropout $N$5, and $N$6. For CA-SUM they are $N$7 and $N$8 (Kontostathis et al., 2024).

These choices indicate that the paper is not primarily proposing a native spherical architecture. Instead, it establishes a controlled benchmark in which 2D summarizers are transplanted into a 360°-derived setting with minimal changes, thereby isolating the effect of the data regime and the saliency prior.

5. Evaluation protocol and empirical results

Evaluation follows standard 5-fold cross-validation over the 40 videos, with 80% train and 20% test splits. Summaries are assessed using the F-score against each of the 15 human summaries. If $N$9 is the set of frames in a machine summary and $i=1,\ldots,N$0 the set of frames selected by one annotator, then

$i=1,\ldots,N$1

For each video, the maximum F-score over the 15 annotators is reported, and the final score is the average across videos (Kontostathis et al., 2024).

The transfer setting shows that models pre-trained on SumMe or TVSum perform at or below random on 360-VSumm.

Setting	Model	F-score
Pre-trained transfer	PGL-SUM (SumMe)	33.7%
Pre-trained transfer	PGL-SUM (TVSum)	33.2%
Pre-trained transfer	CA-SUM (SumMe)	36.5%
Pre-trained transfer	CA-SUM (TVSum)	35.4%
Pre-trained transfer	Random summarizer	35.2%

After re-training and tuning on the 40-video 360-VSumm splits, performance improves substantially.

Setting	Model	F-score
Trained on 360-VSumm	PGL-SUM	47.9%
Trained on 360-VSumm	CA-SUM	45.3%
Saliency-aware	PGL-SUM-sal	48.2% (+0.3)
Saliency-aware	CA-SUM-sal	46.6% (+1.3)

The qualitative analysis reported in the paper aligns with the aggregate scores. In one clip containing three overlapping stadium events, CA-SUM’s unsupervised loss latches onto the most visually dense region and omits the third event entirely, whereas PGL-SUM’s learned importance selects fragments from all three events. Adding saliency re-weighting leads PGL-SUM-sal to produce the most balanced, human-like overviews (Kontostathis et al., 2024).

A common misconception is that strong 2D summarizers can simply be transferred to 360° content. The reported pre-trained results directly counter that view: pre-training on SumMe or TVSum is insufficient, and in several cases the scores are at or below the random summarizer. In the Mid-360 setting, this is one of the clearest empirical indications that spherical, multi-event video introduces a distribution shift not resolved by conventional flat-video pre-training.

6. Analytical implications and prospective directions

The paper draws three principal conclusions from the Mid-360 experiments. First, 360° video summarization is not a trivial extension of 2D summarization. Second, re-training on 360-VSumm yields a 10–15% absolute gain, which the authors interpret as evidence that epoch-length data from real 360° footage is essential. Third, the uplift from saliency-aware feature weighting, up to $i=1,\ldots,N$2 F, indicates that gaze-derived annotations can guide summarizers to the most viewer-relevant views (Kontostathis et al., 2024).

The work further characterizes Mid-360 summarization as benefiting from four ingredients already present in the benchmark: a dense human-annotated dataset reflecting multi-event, dynamic 360° scenes; an evaluation framework that measures recall across multiple ground truths; architectural extensions such as positional encodings and concentrated attention for selecting a temporally sparse yet semantically balanced subset; and incorporation of saliency priors to correct for spatial distortions and viewer focus (Kontostathis et al., 2024).

The future directions are explicitly framed around native 360° summarization rather than continued reliance on flattened 2D proxies. The paper states that truly native 360° summarizers should jointly optimize spatial view selection, meaning which NFOV to show, and temporal summarization, meaning which subshots to include. It identifies three promising directions: spherical convolutions or graph-based sampling over equirectangular meshes; reinforcement-learning agents that mimic head movements; and multi-view transformers that attend both across time and across the $i=1,\ldots,N$3 steradians of available content (Kontostathis et al., 2024).

This suggests that Mid-360 is best understood not as a final architecture but as an intermediate benchmark regime. Its immediate contribution lies in establishing a human-annotated dataset, a formal optimization view, and baseline transfer results that quantify the gap between 2D summarization practice and the requirements of spherical video understanding.