VPCT: ViewPort Context for Immersive Media

• VPCT is a context-aware framework that enhances viewport decisions by fusing cues from language, saliency, and user behavior beyond simple head-motion extrapolation.
• In 360° virtual tourism video, a subtitle-based VPCT model uses CNN, LSTM, and Seq2Seq to fuse saliency maps, subtitle indicators, and head trajectories for accurate future viewport prediction.
• For 360° image compression, a transformer-based VPCT module leverages intra-view and inter-view attention to inject cross-viewport global priors, achieving significant bit savings without quality loss.

Searching arXiv for the cited VPCT-related papers to ground the article in current records. ViewPort ConText (VPCT) denotes a class of context-aware methods for immersive media in which viewport-related decisions are improved by incorporating information beyond raw head-motion extrapolation. In the literature, the term has two explicit instantiations. In 360-degree virtual tourism video, VPCT is a subtitle-based viewport prediction model that combines ground-truth saliency, subtitle-derived navigation cues, and past head movement trajectory to predict future viewport positions (Jing et al., 2022). In 360° image compression, VPCT is a transformer-based ViewPort ConText module that injects cross-viewport global prior information into the entropy model of a neural codec in order to recover global spherical context lost by viewport extraction (Liao et al., 24 Mar 2026). Related work extends the same general idea—viewport decisions depend on context from language, visual content, user behavior, and cross-view correlations—even when the acronym itself is not formally used (Chopra et al., 2021, Dharmasiri et al., 2021, Zhang et al., 2023, Li et al., 2023).

1. Terminological scope and conceptual definition

In the strictest sense, VPCT refers to the method introduced in "Subtitle-based Viewport Prediction for 360-degree Virtual Tourism Video" (Jing et al., 2022). There, the model is a multi-input deep learning framework for future viewport/head-orientation prediction in guided tourism video, and its distinctive feature is the use of subtitle/navigation context in addition to saliency and head trajectory. The paper’s central claim is that subtitles in guided 360° tourism video often contain navigation cues such as “in front of you,” “on the left,” or “look at the bridge,” which alter user attention and thus improve prediction accuracy when modeled explicitly (Jing et al., 2022).

A second, formally named VPCT appears in "Viewport-based Neural 360° Image Compression" (Liao et al., 24 Mar 2026). In that work, VPCT stands for ViewPort ConText, a transformer-based module embedded in a neural compression pipeline. Its function is not to predict future gaze, but to model cross-viewport global prior information during entropy coding, so that later viewports can exploit information from previously encoded ones (Liao et al., 24 Mar 2026).

Beyond these two explicit usages, several papers adopt the same contextual premise without defining an official VPCT acronym. PARIMA models future viewport as a function of both prior viewport state and the trajectories of prime objects in the video (Chopra et al., 2021). MFTR performs viewport prediction via multi-modal fusion of head history, eye movements, and video content (Zhang et al., 2023). STVP combines saliency and trajectory information for volumetric video streaming (Li et al., 2023). By contrast, "Robust and Resource-efficient Machine Learning Aided Viewport Prediction in Virtual Reality" states explicitly that VPCT is not the official name of the proposed method, and that the paper instead proposes a meta-learning based viewport prediction paradigm with two separate models (Jiang et al., 2022).

This suggests that "VPCT" functions both as a paper-specific method name and as a broader descriptor for viewport-context modeling in immersive media.

2. Subtitle-aware VPCT for 360-degree virtual tourism video

The VPCT model of (Jing et al., 2022) is designed for 360-degree virtual tourism videos, a domain in which subtitles are not merely accessibility aids but often encode guidance about where viewers should look. The model therefore treats viewport prediction as a fusion problem over three information sources: ground-truth saliency derived from users’ past head trajectories, subtitle context, and past head movement trajectory (Jing et al., 2022).

At time $t$, the model uses four inputs: a ground-truth saliency heat map, a subtitle indicator $f_{SI}^{(t)}$, subtitle navigation text, and past head movement trajectory $C^{(t)}$ (Jing et al., 2022). The saliency branch does not use content-based saliency; instead it constructs ground-truth saliency maps from user head trajectories: $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$ with

$$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$

where $C_{v,u}^{(t)}=(\varphi,\theta)$, $O(*)$ is orthodromic distance on the unit sphere, and $\sigma=\pi/30$ (Jing et al., 2022). This saliency heat map is processed by a VGG16-based CNN to produce $f_{GTS}^{(t)}$ (Jing et al., 2022).

The subtitle branch contains two features. The subtitle indicator is defined as

$f_{SI}^{(t)} = 1$

if a subtitle exists at time $f_{SI}^{(t)}$0, otherwise $f_{SI}^{(t)}$1 (Jing et al., 2022). The navigation feature $f_{SI}^{(t)}$2 is extracted from subtitle text with a multi-layer LSTM; if there is no subtitle or no navigation content, $f_{SI}^{(t)}$3 is set to zero (Jing et al., 2022). The extracted features are fused as

$f_{SI}^{(t)}$4

then paired with past trajectory history and fed into a Seq2Seq network (Jing et al., 2022). Using a sliding window of $f_{SI}^{(t)}$5 past timesteps,

$f_{SI}^{(t)}$6

the model predicts $f_{SI}^{(t)}$7 future viewport positions,

$f_{SI}^{(t)}$8

with $f_{SI}^{(t)}$9 and $C^{(t)}$0 (Jing et al., 2022).

The loss function is MSE (Jing et al., 2022). Evaluation uses $C^{(t)}$1, $C^{(t)}$2, and orthodromic distance, with lower values indicating better prediction (Jing et al., 2022). On five guided tourism videos—London, Barcelona, Vienna, Varadero, and Holguin—the proposed model reports 58.6053 for $C^{(t)}$3, 8.1483 for $C^{(t)}$4, and 0.3095 \pm 0.4601 for average orthodromic distance, outperforming MM18, NOSSDAV17, and IEEENL20 on all reported metrics (Jing et al., 2022).

The experimental setting is tightly specified. The subtitles are separate from the video and move with the viewport in the VR display, 30 subjects watched using a Facebook Oculus Quest 2, head trajectories were recorded in Unity at 0.5-second intervals, and orientation was converted from Euler angles to spherical coordinates $C^{(t)}$5 (Jing et al., 2022). The total subtitle duration is 17 minutes, about 74% of the total video duration (Jing et al., 2022). The paper further reports that trajectories in the subtitle group were more clustered and goal-directed, while trajectories without subtitles were more scattered (Jing et al., 2022).

3. VPCT as cross-viewport context in neural 360° image compression

In (Liao et al., 24 Mar 2026), VPCT has a different but related meaning. The paper replaces the conventional pipeline of projecting a spherical image into a single 2D plane with viewport extraction, motivated by the claim that ERP introduces both oversampling and distortion (Liao et al., 24 Mar 2026). However, compressing multiple viewports independently causes global information loss, because inter-viewport redundancy and overlap are no longer directly visible to a standard 2D codec (Liao et al., 24 Mar 2026).

VPCT is introduced as the mechanism that restores this missing global context. The paper defines the entropy-model improvement as

$C^{(t)}$6

where $C^{(t)}$7 is local information from the current viewport codec and $C^{(t)}$8 is global cross-viewport information produced by VPCT (Liao et al., 24 Mar 2026). Thus VPCT is not a standalone codec; it is a context module that improves the entropy model of canonical learned image compression structures (Liao et al., 24 Mar 2026).

The module operates on a sequence of quantized latent features from multiple viewports,

$C^{(t)}$9

and is organized as a transformer stack whose basic layer contains an intra-view block and an inter-view block (Liao et al., 24 Mar 2026). The intra-view block models causal dependencies inside the current viewport using masked self-attention: $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$0 with a causal mask $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$1 (Liao et al., 24 Mar 2026). The inter-view block performs cross-view attention between the current viewport and previously encoded viewports: $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$2 with a cross-view causal mask $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$3 to ensure that only already encoded viewports are used as prior information (Liao et al., 24 Mar 2026).

The viewport-based pipeline uses default viewport centers

$$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$4

with $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$5 field of view in both directions to cover the sphere (Liao et al., 24 Mar 2026). Training uses CompressAI, 19,590 360° images, Adam, 200 epochs, batch size 8, learning rate $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$6, and MSE distortion, with $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$7 in

$$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$8

(Liao et al., 24 Mar 2026). The test datasets are LIC360, Flickr360, CVIQ, and SaliencyVR (Liao et al., 24 Mar 2026).

The main reported result is that the viewport-based pipeline achieves 14.01% average bit savings compared to the best-performing existing 360° compression methods without compromising quality (Liao et al., 24 Mar 2026). The paper also reports that VPCT adds only about 2.30% FLOPs and 2.32% MAC/pixel while improving compression substantially (Liao et al., 24 Mar 2026). A notable qualitative finding is that later viewports benefit more from VPCT, because they can use earlier viewports as context (Liao et al., 24 Mar 2026).

4. Related context-aware formulations in viewport prediction

Several papers operationalize the same general principle as VPCT—viewport behavior depends on content, motion, or behavioral context—while using different model names.

PARIMA treats viewport prediction as a content-aware streaming control problem in which future viewports depend on both user context and video content context represented by the trajectories of prime objects (Chopra et al., 2021). The system performs one-time server-side preprocessing with YOLOv3, converts equirectangular frames to cube-map projection, tracks detected objects using spherical centroid tracking, predicts a smooth intermediate viewport sequence with ARIMA, refines it with Passive-Aggressive Regression, and updates the model online after each chunk (Chopra et al., 2021). The paper reports that object trajectories contribute, on average, about 33.8% of the final viewport prediction (Chopra et al., 2021).

MFTR reformulates viewport prediction as tile classification rather than direct coordinate regression (Zhang et al., 2023). It uses historical head orientations, eye movements, and future video frames as context; extracts temporal features with LSTMs and a Temporal Transformer; visual features with MobileNetV2 and a Visual Transformer; and then performs temporal-visual fusion with a transformer stack (Zhang et al., 2023). For each future timestamp, MFTR outputs a $$GTS_{v,x,y}^{(t)}=\frac{1}{N}\sum_{u}GTS_{v,u,x,y}^{(t)}, \quad t\in [0,T]$$9 tile score map $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$0, applies a threshold $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$1, and chooses the final viewport as the region containing the most user-interested tiles (Zhang et al., 2023). On PVS-HM at 5 seconds, it reports AP 0.730 and AO 0.787 (Zhang et al., 2023).

STVP extends context-aware viewport prediction to volumetric video / point cloud video (Li et al., 2023). It combines spatial-temporal saliency detection with LSTM-based head trajectory prediction, using Uniform Random Sampling (URS) to preserve inter-frame mapping and an attention-based fusion of saliency and trajectory: $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$2

$$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$3

(Li et al., 2023). The reported results include 82.09% point-level MIoU, 88.10% tile-level MIoU, 90.17% accuracy, 83.72% precision, and 96.80% recall (Li et al., 2023).

A different but related line of work uses viewport traces themselves as context for content organization. "Viewport-Aware Dynamic 360° Video Segment Categorization" aggregates 88 videos, over 3700 viewport traces, and over 142 hours of viewing logs, clusters 2-second chunks of viewport traces using a 15-dimensional feature vector, chooses $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$4 viewport clusters and $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$5 video categories using Davies-Bouldin analysis, and reports 28.32% improvement in similarity of viewport distributions within clusters compared to static genre-based categorization (Dharmasiri et al., 2021). This paper treats viewer behavior as a context descriptor for video segments rather than as a direct predictive signal.

5. Robustness, adaptation, and non-VPCT nomenclature

Not all viewport-context work uses VPCT as an official term. "Robust and Resource-efficient Machine Learning Aided Viewport Prediction in Virtual Reality" states explicitly that the paper does not explicitly define a module or method named “VPCT” and that the closest conceptual match is its viewport prediction framework (Jiang et al., 2022). That framework focuses on robustness, especially the worst-performing predictions, rather than on subtitle context or cross-viewport entropy modeling.

The method uses a double-meta-learning paradigm with two separate models: a viewing direction prediction model $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$6 and a minimum prefetch size prediction model $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$7 (Jiang et al., 2022). Both are meta-trained offline and adapted online per user using Reptile-style updates. The prefetch angle is predicted via

$$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$8

where $$GTS_{v,u,x,y}^{(t)}=\exp\left ( -\frac{O(C_{v,u}^{(t)},P_{x,y}^{(t)})^2}{2\sigma ^2} \right ),$$9 is the actual viewport angle and $C_{v,u}^{(t)}=(\varphi,\theta)$0 is the predicted angular error (Jiang et al., 2022).

The training structure uses previous 100 samples as input, the current sample as label, a 100 ms time step, and mean squared loss, with a leave-one-user-out style within each video (Jiang et al., 2022). On a dataset with 48 users, 18 panoramic videos, and 5 categories, the paper reports strong worst-case gains: worst successful prefetch rate improved from 35% to 89% in experiment 1 and from 75% to about 93% in experiment 2 (Jiang et al., 2022). Full adaptation reaches IWP up to 54% in experiment 1 and up to 18% in experiment 2 (Jiang et al., 2022).

This usage is significant because it marks a boundary condition for the VPCT label. The paper is directly concerned with viewport prediction context and personalization, yet it rejects VPCT as a formal acronym for its own method. A plausible implication is that “VPCT” has become a useful interpretive shorthand across neighboring research threads, even when individual papers do not adopt it.

6. Common methodological themes and limitations

Across the cited literature, VPCT-style methods share a common methodological structure: they incorporate contextual signals that are external to simple motion extrapolation. In (Jing et al., 2022), the additional context is subtitle/navigation text; in (Liao et al., 24 Mar 2026), it is cross-viewport global prior information; in (Chopra et al., 2021), it is prime-object trajectories; in (Zhang et al., 2023), it is a fusion of head history, eye movements, and video content; and in (Li et al., 2023), it is spatial-temporal saliency plus trajectory.

A second recurrent theme is that context is often fused with sequential predictors. The subtitle-based VPCT uses CNN + LSTM + Seq2Seq (Jing et al., 2022). PARIMA combines ARIMA and Passive-Aggressive Regression with online adaptation (Chopra et al., 2021). MFTR employs multiple transformer stacks for within-modality and cross-modality dependencies (Zhang et al., 2023). STVP fuses saliency and trajectory through learned attention (Li et al., 2023). The compression VPCT uses transformer-based intra-view and inter-view attention inside the entropy model (Liao et al., 24 Mar 2026).

The limitations are equally consistent. The subtitle-based VPCT is evaluated on a specific domain, guided virtual tourism video, and subtitle usefulness depends on whether subtitles contain navigation information (Jing et al., 2022). It also relies on ground-truth saliency from collected trajectories, rather than purely content-driven saliency (Jing et al., 2022). PARIMA requires one-time server-side object preprocessing and assumes that head movement depends strongly on prime-object trajectories (Chopra et al., 2021). STVP is evaluated on only four 8iVFB sequences and its temporal saliency depends heavily on URS preserving inter-frame mapping (Li et al., 2023). The meta-learning framework of (Jiang et al., 2022) improves robustness, but its formulation is not itself a VPCT module or acronym.

These constraints indicate that VPCT is not a single unified architecture. Rather, it is a family of context-enriched modeling strategies whose specific form depends on the immersive-media task: viewport prediction, tile selection, segment categorization, or compression.

7. Significance for immersive media systems

The significance of VPCT lies in its challenge to the assumption that immersive-media viewport behavior can be modeled adequately from historical head motion alone. The subtitle-based VPCT shows that, in guided tourism video, language is a strong attention cue and that subtitle context provides both direct benefit through explicit features and indirect benefit by shaping users’ head trajectories (Jing et al., 2022). PARIMA shows that object trajectories can materially influence online viewport prediction and downstream bitrate allocation (Chopra et al., 2021). MFTR argues that tile-level prediction yields better robustness and interpretability than direct trajectory regression (Zhang et al., 2023). STVP demonstrates that saliency and trajectory can be fused effectively in volumetric video, where the viewport is a 3D region rather than a 2D crop (Li et al., 2023). The compression VPCT shows that viewport context is not only a predictive cue but also a coding prior that can reduce bit consumption in neural 360° image compression (Liao et al., 24 Mar 2026).

Taken together, these works establish VPCT as an important organizing idea for immersive-media research: viewport-related decisions are improved when models exploit the contextual structure of language, content, user behavior, and cross-view dependencies. This suggests that the long-term development of viewport-aware systems will continue to move away from single-signal extrapolation and toward explicitly structured context modeling across modalities and tasks.