Percep360: Spherical Perception in 360 Media
- Percep360 is a unified research paradigm addressing spherical distortions and viewport-dependent perception in 360° media.
- It spans methods from blind quality assessment with multi-axis attention to native spherical modeling for depth, segmentation, and object detection.
- Applications include optimizing streaming, enhancing audiovisual evaluations, and improving immersive VR/AR experiences in panoramic scenes.
Searching arXiv for the cited Percep360-related papers to ground the article in the current record. Percep360 is a research umbrella for perception-centered methods, datasets, and system designs for omnidirectional media and panoramic scene understanding. In the materials associated with this term, Percep360 denotes a common technical objective: aligning 360-degree representation, inference, and evaluation with the actual constraints of spherical sensing, head- or viewport-dependent observation, and multi-modal quality of experience. Across the cited works, Percep360 spans blind omnidirectional image quality assessment, perception-aware streaming, multi-task 360 scene understanding, spherical dense prediction, panoramic object and affordance grounding, multimodal audiovisual evaluation, and 360-degree reasoning with multimodal LLMs (Yan et al., 26 Feb 2025). This breadth suggests that Percep360 is best understood not as a single model, but as a unifying paradigm for distortion-aware, behavior-aware, and geometry-aware perception over full-sphere data.
1. Conceptual scope and problem setting
Percep360 arises from a set of recurring technical difficulties that distinguish 360-degree imagery and video from planar media. Omnidirectional content is spherical but is typically stored in equirectangular projection (ERP), which introduces severe, non-uniform geometric stretching near the poles; user perception is also viewport-dependent in head-mounted display viewing, so perceptual relevance depends on where a viewer looks and when (Yan et al., 26 Feb 2025). A related survey frames the same problem more broadly: 360-degree media combine spherical geometry, projection-induced distortions, user-driven viewport mechanics, and HMD-specific rendering constraints, so perception, assessment, and compression must all be reformulated relative to conventional 2D workflows (Li et al., 2019).
The same geometric difficulty appears in 360 perception tasks beyond quality assessment. ERP warps the sphere onto a plane, degrading feature quality for geometry and semantics, while the ultra-wide field of view creates a need for strong global perception (Ai et al., 2024). Native spherical modeling has therefore emerged as a distinct direction, with geodesic icosphere or hexasphere discretizations used to avoid ERP’s boundary artifacts and pole distortion in dense prediction tasks such as depth estimation and semantic segmentation (Benny et al., 2024). In detection settings, ERP distortion and wrap-around discontinuities complicate standard bounding-box formulations, motivating spherical boxes, spherical anchors, and spherical IoU definitions tailored to omnidirectional scenes (Zhao et al., 2019).
Percep360 also extends to multimodal and embodied settings. A panoptic formulation appears in 360+x, which combines third-person panoramic and front views with egocentric monocular/binocular video, multi-channel audio, directional binaural delay, location data, and text descriptions to study “how daily information is accessed in the real world” (Chen et al., 2024). In audiovisual quality research, 360-specific evaluation is necessary because immersive quality depends jointly on ERP video, spatial audio, reproduction setup, and task design (Fela et al., 2022). In embodied perception, holistic affordance grounding in 360-degree indoor environments introduces additional challenges of semantic dispersion and cross-scale alignment over the sphere (Zhu et al., 10 Mar 2026).
2. Perceptual quality assessment and viewing-behavior modeling
A major Percep360 thread concerns quality prediction for omnidirectional images under realistic viewing conditions. Blind OIQA is needed when pristine references are unavailable, especially for production VR/AR pipelines and large-scale repositories (Yan et al., 26 Feb 2025). In this setting, Max360IQ introduces a blind omnidirectional image quality assessment model with multi-axis attention, multi-scale feature integration, and deep semantic guidance, and it is explicitly positioned as suitable for a Percep360 pipeline for 360° content quality monitoring and QoE optimization (Yan et al., 26 Feb 2025). Its end-to-end flow uses viewport extraction, a backbone with stacked multi-axis attention, multi-scale feature integration, and quality regression with deep semantic guidance. On JUFE, Max360IQ improves over Assessor360 by 3.5% PLCC and 3.6% SRCC; on OIQA and CVIQ it achieves SRCC 0.9704 and 0.9809, respectively (Yan et al., 26 Feb 2025).
A complementary line emphasizes that perceptual quality depends on viewing behavior rather than a single deterministic viewport path. “Perceptual Quality Assessment of 360° Images Based on Generative Scanpath Representation” introduces a Generative Scanpath Representation that aggregates multiple hypothetical users under a specified viewing condition defined by starting point and exploration time (Sui et al., 2023). The framework uses ScanDMM to generate scanpaths, converts gaze-centered spherical-tangent patches into a compact spatiotemporal representation, and predicts quality with X-Clip-B/32. On JUFE, GSR-X achieves SRCC 0.818 and PLCC 0.830, while ablations reach SRCC 0.853 and PLCC 0.859 when using spherical tangent patch extraction and behavior-aware GSR (Sui et al., 2023). The same work reports that GSR-based full-reference models remain competitive while reducing preprocessing from approximately 127.3 s per reference/distorted pair to approximately 0.238 s in the cited comparison (Sui et al., 2023).
A related full-reference perspective models panoramas explicitly as moving-camera videos. “Perceptual Quality Assessment of Omnidirectional Images as Moving Camera Videos” identifies starting point and exploration time as key VR viewing conditions, and reports that starting point is significant, distortion type is significant, exploration time alone is not significant, but the interaction of starting point and exploration time is strong (Sui et al., 2020). Its framework converts a panorama into a viewport video along viewing trajectories and applies 2D full-reference IQA with temporal pooling. On its proposed database, behavior-conditioned models improve substantially over projection-only and viewport baselines; for example, O-DISTS reaches overall PLCC approximately 0.660 and SRCC approximately 0.613, and Gaussian recency weighting can raise O-DISTS to approximately 0.84/0.84 in the reported ablation (Sui et al., 2020). This suggests that recency-aware temporal pooling is not merely an implementation detail but a structural component of perceptual fidelity in 360 quality modeling.
Percep360 also includes depth-specific perceptual quality. “Perceptual Depth Quality Assessment of Stereoscopic Omnidirectional Images” proposes a no-reference Depth Quality Index for stereoscopic omnidirectional images, using interocular discrepancy, LAB channels, adaptive equator viewports, and wavelet-subband statistics (Zhou et al., 2024). On SOLID, DQI achieves SROCC approximately 0.9299, PLCC approximately 0.9482, and KROCC approximately 0.7814 for depth quality prediction, and combining DQI with existing IQA features improves overall 3D omnidirectional quality prediction, reaching SROCC approximately 0.9301 and PLCC approximately 0.9359 with local MS-SSIM viewports plus DQI (Zhou et al., 2024).
3. Perception-aware streaming and multimodal quality of experience
Percep360 also encompasses systems that optimize delivery rather than only post hoc assessment. “Pano: Optimizing 360° Video Streaming with a Better Understanding of Quality Perception” argues that perceived quality in 360° video is not simply proportional to field-of-view size (Guan et al., 2019). Pano models quality with a 360°-specific JND-aware PSPNR that depends on viewpoint moving speed, recent luminance change, and depth-of-field disparity. User studies in that work show that when the viewpoint moves greater than , users tolerate about 50% more distortion in background pixels than at rest; luminance changes of at least 200 gray levels often enable approximately 50% higher distortion tolerance for approximately 5 seconds; and depth-of-field differences of at least 0.7 diopters typically raise JND by about 50% (Guan et al., 2019).
That perceptual model then drives variable-sized tiling and quality adaptation. The reported outcome is that, compared with state-of-the-art techniques, Pano can save 41–46% bandwidth without any drop in perceived quality, or raise perceived quality by 25%–142% without using more bandwidth (Guan et al., 2019). Component-wise ablations report bandwidth reductions of 17% from JND-awareness, 11% additional from the 360-specific multipliers, and 17% additional from variable-sized tiling at MOS approximately 5 (Guan et al., 2019). In a Percep360 interpretation, this is a delivery-side analogue to behavior-aware OIQA: perceptual factors are made operational in resource allocation.
The same multimodal concern appears in audiovisual quality databases and predictors. “Perceptual Evaluation on Audio-visual Dataset of 360 Content” builds a 360 audiovisual dataset with 16 AV stimuli, 8K ERP video, and 4th-order AmbiX HOA audio, and conducts separate audio, video, and AV subjective experiments (Fela et al., 2022). In that study, VMAF 4K gives the best video correlation with subjective scores, with PLCC 0.919 and SRCC 0.957, while ViSQOLaswb gives the best audio correlation, with PLCC 0.924 and SRCC 0.938 (Fela et al., 2022). It also reports that audio quality differences become perceptually meaningful in AV ratings only when video quality is sufficiently high, specifically when resolution is at least 4K or QP is at most 28 (Fela et al., 2022).
A machine-learning extension turns these signals into predictive AV quality models. “Perceptual Evaluation of 360 Audiovisual Quality and Machine Learning Predictions” benchmarks multiple linear regression, decision tree, random forest, and support vector machine on combinations of video and audio quality metrics (Fela et al., 2021). The best configuration uses VMAF and AMBIQUAL with support vector machine under k-Fold cross-validation, reaching PCC , SROCC , and RMSE (Fela et al., 2021). This indicates that Percep360 is not limited to visual geometry and viewport behavior; it also includes modality fusion strategies for predicting perceived audiovisual quality from heterogeneous objective measures.
4. Geometry-aware 360 scene understanding and dense perception
Percep360 includes a second major family: dense and semantic understanding of panoramic scenes. Elite360M targets end-to-end 360 multi-task learning by combining ERP with distortion-free, spatially continuous icosahedron projection points and fusing them through Bi-projection Bi-attention Fusion and Cross-task Collaboration (Ai et al., 2024). It jointly predicts depth, surface normals, and semantic segmentation. On Matterport3D, Elite360M with ResNet-34 uses 29.41M parameters and 85.07G FLOPs, while InvPT++ uses 379.22M parameters and 1009.00G FLOPs; Elite360M achieves comparable depth and better normals, though semantic segmentation remains slightly below UniFuse-M and InvPT++ (Ai et al., 2024). Ablations show that adding B2F over an ERP baseline improves depth AbsRel by 5.23% with ResNet-34 and 12.13% with ResNet-18 at only about 1.02M additional parameters and +6.06G FLOPs (Ai et al., 2024).
SphereUFormer advances the same goal by discarding ERP entirely in favor of native spherical transformers on icosphere or hexasphere meshes (Benny et al., 2024). It introduces Spherical Local Self-Attention with neighborhood-restricted attention and learned relative positional bias, plus spherical up/downsampling. On Stanford2D3D at rank 7, SphereUFormer reaches MAE .165, MRE .071, and for depth, and for semantic segmentation reaches Acc 88.6 and mIoU 72.2; on Structured3D it reaches MAE .142, MRE .045, , and segmentation Acc 95.8 and mIoU 53.0 (Benny et al., 2024). The paper explicitly attributes gains to improved pole behavior and removal of ERP boundary effects (Benny et al., 2024).
Monocular 360 depth estimation receives a different treatment in PanoGabor. “Revisiting 360 Depth Estimation with PanoGabor: A New Fusion Perspective” argues that perspective-based features eventually need to be unified into ERP, reintroducing distortions, and addresses this with a PanoGabor-based fusion module (Shen et al., 2024). PGFuse combines two cubemap groups, channel-wise and spatial-wise unidirectional fusion, latitude-aware PanoGabor filters, and a spherical gradient constraint. On Matterport3D with ResNet-34, PGFuse achieves AbsRel 0.1067, SqRel 0.0858, RMSE 0.4624, and , outperforming Elite360D and UniFuse in the cited comparison; analogous gains are reported on Stanford2D3D and Structured3D (Shen et al., 2024). The ablation from baseline to full model improves RMSE from 0.5021 to 0.4624 and from 87.29 to 89.06 (Shen et al., 2024).
These models collectively show a stable Percep360 principle: dense understanding of spherical scenes requires either better cross-projection fusion or native spherical operators. A plausible implication is that Percep360 architectures can be arranged on a spectrum from ERP-centric correction to sphere-native computation, with both ends aiming at the same perceptual invariant: global consistency without projection-induced semantic drift.
5. Object, affordance, and multimodal reasoning over the full sphere
Beyond dense geometry, Percep360 includes scene-level recognition, detection, grounding, and reasoning. Reprojection R-CNN defines a standard object detection framework for 360° images using spherical bounding boxes, spherical anchors, spherical IoU, a Spherical RPN on ERP, and a tangent-plane refinement stage (Zhao et al., 2019). On VOC360, Rep R-CNN reaches 71.88 mAP at 127 ms per image with proposals and 71.65 mAP at 178 ms with ; on COCO-Men, it reaches 81.48 mAP at 178 ms with 0 (Zhao et al., 2019). This demonstrates that Percep360 object reasoning can combine ERP for global proposal efficiency with perspective reprojection for geometric fidelity.
A more scene-level formulation appears in PanoAffordanceNet, which introduces “Holistic Affordance Grounding in 360° Indoor Environments” and the 360-AGD dataset (Zhu et al., 10 Mar 2026). The model combines a Distortion-Aware Spectral Modulator and an Omni-Spherical Densification Head, and trains with pixel-wise, distributional, and region-text contrastive objectives. On 360-AGD, PanoAffordanceNet achieves Easy split KLD 1.270, SIM 0.506, NSS 4.490 and Hard split KLD 1.306, SIM 0.474, NSS 4.398, substantially outperforming OOAL and OS-AGDO (Zhu et al., 10 Mar 2026). The full ablation on the Hard split improves from 1.475 / 0.416 / 4.196 for the baseline to 1.306 / 0.474 / 4.398 for the complete model (Zhu et al., 10 Mar 2026).
Percep360 also reaches high-level multimodal reasoning with multimodal LLMs. “360° Image Perception with MLLMs: A Comprehensive Benchmark and a Training-Free Method” introduces 360Bench with 643 ERP images at 7296 × 3648 and 1,532 multiple-choice samples across seven subtasks (Tran et al., 17 Mar 2026). It reports that the best proprietary MLLM, Gemini Pro 2.5, reaches 46.5% overall, while the best open-source baseline, Qwen2.5-VL-7B on CMP, reaches 38.1% (Tran et al., 17 Mar 2026). The training-free Free360 framework decomposes reasoning into entity identification, attribute extraction, inter-entity relation detection, and entity-view relation detection using a scene graph and hybrid ERP/CMP transformations. Free360 improves Qwen2.5-VL-7B from 38.1% on CMP and 37.3% on ERP to 45.3% overall in the hybrid setting, with especially large gains on viewer-centric spatial reasoning, up to +22.9 on SR-OV in the cited comparison (Tran et al., 17 Mar 2026).
Panoptic multi-view multimodal perception appears in 360+x. The dataset contains 2,152 videos grouped into 232 scenes, about 244,000 seconds of data, four synchronized viewpoints, and multiple modalities including video, audio, directional binaural delay, GPS, weather, and text (Chen et al., 2024). On scene classification, all three views with video, audio, and directional features achieve average precision 80.62, outperforming all smaller combinations; on TriDet-based temporal action localisation, all views plus V+A+D with 360+x pretraining reach average mAP 19.1 in the cited ablation (Chen et al., 2024). This indicates that Percep360 is not only spherical in geometry but also panoptic in viewpoint aggregation.
6. Distortion taxonomy, human factors, and broader implications
Percep360 research repeatedly returns to distortion taxonomy and human factors. “Visual Distortions in 360-degree Videos” provides a pipeline-wide review covering capture, stitching, projection, resampling, compression, delivery, and HMD display (Azevedo et al., 2019). It highlights ERP stretching near poles, cubemap face seams, stitching ghosting, exposure mismatch, radial blocking near ERP poles, tile-boundary artifacts, and HMD-specific effects such as chromatic aberration, screen-door effect, and persistence smear (Azevedo et al., 2019). The review emphasizes that the perceptual impacts of many of these distortions remain open research questions, and positions such characterization as foundational for benchmarking and psychovisual study design (Azevedo et al., 2019).
Human factors in immersive evaluation are explored directly in socioemotional and privacy-sensitive settings. “Methodology to Assess Quality, Presence, Empathy, Attitude, and Attention in 360-degree Videos for Immersive Communications” validates a joint methodology for long-form 360 communication scenarios (Orduna et al., 2021). In that work, quality ratings vary significantly with QP, and social presence depends significantly on content, with “Study in Spain” reaching social presence 1 versus 4.748 and 4.752 for the other two contents; spatial presence remains high without significant content or condition effects (Orduna et al., 2021). This extends Percep360 beyond classical QoE into presence, attention, and attitude.
A more specific human-factor example appears in facial anonymization. “Investigating the Perception of Facial Anonymization Techniques in 360° Videos” compares blurring, black boxes, and face-swapping on screen and in HMD (Wöhler et al., 2024). Blocking is identified at 91.25% on screen and 96.25% in HMD; blurring at 86.25% and 85.0%; face-swapping at 32.5% and 42.5% (Wöhler et al., 2024). The study reports that face-swapping is most realistic and least disruptive but is perceived as less effective anonymization than blocking or blurring, while presence-related realism in HMD is reduced most strongly by blocking (Wöhler et al., 2024). This indicates a recurring Percep360 trade-off: methods that preserve immersion and photorealism may weaken explicit perceptions of privacy protection.
Finally, Percep360 also reaches data generation. “Hallucinating 360°: Panoramic Street-View Generation via Local Scenes Diffusion and Probabilistic Prompting” defines Percep360 as a panoramic generation framework for autonomous driving that targets coherence and controllability (Teng et al., 9 Jul 2025). On the reported validation results, Percep360 obtains BRISQUE 20.24, PIQE 11.44, SSIM 0.16, FID 14.43, drivable IoU 0.25, and mean mIoU 0.13, and synthetic augmentation from Percep360 improves OneBEV drivable IoU from 0.593 to 0.610 and mean mIoU from 0.470 to 0.495 (Teng et al., 9 Jul 2025). This suggests a further expansion of the concept: Percep360 can also denote the generation of controllable panoramic data that improve downstream perception.
Across these strands, Percep360 denotes a technically coherent research direction: perception over 360-degree media should be spherical in geometry, attentive to human viewing behavior, aware of modality interactions, and evaluated under immersive conditions rather than inherited planar assumptions. The cited works differ in task and architecture, but they converge on the same methodological thesis: robust omnidirectional perception depends on matching representation, inference, and evaluation to the physical and perceptual structure of the sphere.