Foveated Vision Interface (FOVI) Overview

• Foveated Vision Interface (FOVI) is a perceptual interface principle that allocates high resolution near fixation while reducing detail in the periphery.
• It underpins diverse applications—from adaptive VR displays and streaming to deep learning models—optimizing both computational load and perceptual quality.
• Recent advances integrate gaze, attention, and task relevance to jointly optimize latency, bandwidth, and image fidelity across sensing, rendering, and AI inference systems.

Searching arXiv for the cited FOVI and related foveation papers to ground the article in current literature. A Foveated Vision Interface (FOVI) is a visual sensing, display, or processing interface that allocates spatial fidelity non-uniformly, concentrating resolution or computation near the fovea and reducing it with eccentricity, often in a gaze-contingent manner and sometimes in a way that explicitly matches human acuity. In the display literature, a foveated display is defined broadly as either a system that steers or moves display content according to gaze direction or a system that varies resolution or perceived resolution in a way that matches human visual acuity (Spjut et al., 2019). Subsequent work generalizes that idea beyond displays to streaming, computational imaging, active perception, robot learning, and deep vision models, where FOVI denotes an interface in which the visual representation itself is reshaped around fixation, attention, or task relevance rather than delivered on a uniform grid (Blauch et al., 3 Feb 2026).

1. Conceptual scope and taxonomy

The conceptual core of FOVI is the coupling of nonuniform spatial allocation with either gaze contingency or acuity matching. The standardization work on foveated displays treats these as two orthogonal axes: a resolution contingent classification, which compares the display’s Resolution Distribution Function (RDF) to the viewer’s Acuity Distribution Function (ADF), and a gaze contingent classification, which quantifies how stable that resolution profile remains under gaze change (Spjut et al., 2019). This yields the combined A–D and 1–4 taxonomy, where Class A is acuity matched, Class B is foveally matched, Class C is peripherally matched, and Class D is non-acuity matched, while Class 1 through Class 4 range from fully foveated to non-foveated.

This taxonomy is important because it decouples the idea of foveation from any specific implementation substrate. A FOVI may be realized as a gaze-steered display, a nonuniform renderer, a bandwidth-adaptive streaming stack, a foveated thermal imager, or a vision model that ingests retina-like samples. This suggests that FOVI is best understood not as a single algorithmic family but as a perceptual interface principle: align delivered visual detail with where acuity or task-relevant information is highest, and avoid over-provisioning the periphery.

A common misconception is that FOVI is synonymous with dynamic eye-tracked VR rendering. The literature is broader. The display definition explicitly includes systems without eye tracking if their resolution profile matches acuity (Spjut et al., 2019), while later work shows FOVI-like organization in thermal meta-optics, robot policies, and deep neural interfaces (Saragadam et al., 2022). A plausible implication is that “interface” is the more fundamental term: the system mediates between world, sensor, model, and observer through a spatially varying allocation policy.

2. Biological and mathematical foundations

FOVI inherits its structure from the nonuniformity of biological vision. Across the surveyed literature, the recurring premise is that acuity and contrast sensitivity peak near fixation and decrease with eccentricity, so uniform sampling wastes bandwidth, pixels, or FLOPs outside the fovea (Jabbireddy et al., 2022). The display framework captures this with the ADF, modeled as a piecewise function of eccentricity $e$:

$S$5

where $F$ is foveal acuity, $e_0$ is the constant-resolution foveal extent, and $S$ is the peripheral roll-off slope (Spjut et al., 2019). The companion RDF describes what the interface actually provides; the comparison $RDF(e) \ge ADF(e)$ defines acuity matching.

Several later systems instantiate this same principle with different mathematical objects. In EyeNexus, gaze-driven encoding uses a Gaussian quantization-offset field centered at gaze:

$$QO(i,j) = QO_{max} - QO_{max} \times \exp{\left(-\frac{(Distance(i,j))^2}{2C^2}\right)}$$

where the controller $C$ widens or narrows the effective foveation region according to network state (Wu et al., 15 Sep 2025). In the CUDA foveated renderer, blur width is derived from an eccentricity-dependent contrast-threshold model using Geisler and Perry’s formulation, producing an eccentricity-dependent $\sigma$ for Gaussian blur (Malkin et al., 2020). In the biologically inspired deep-vision interface, cortical magnification is made explicit through

$M(r)=\frac{1}{r+a}, \qquad w(r)=\log(r+a)+C$

so that uniform sampling in cortical space induces nonuniform retinal sampling in visual space (Blauch et al., 3 Feb 2026).

The literature also argues that eccentricity alone is not sufficient. The attention-aware contrast sensitivity model introduces an attention-dependent threshold curve

$t_a(e) = p_0 \sqrt{e} + p_1$

and rescales a baseline CSF by an attention gain $F$0, showing that peripheral sensitivity drops when attention is concentrated at the fovea (Krajancich et al., 2023). This directly challenges eccentricity-only FOVI models. A plausible implication is that a mature FOVI should be not only gaze-aware but also attention-aware, because the usable peripheral budget depends on what the observer is doing at fixation.

3. Interface architectures across rendering, display, and streaming

In VR and AR, FOVI most commonly appears as a rendering or transport interface that preserves high fidelity near gaze and reduces effort elsewhere. The survey literature organizes these implementations by where foveation enters the pipeline: screen-based, object-based, and optics-based methods, as well as static versus dynamic gaze dependence (Jabbireddy et al., 2022). This broad taxonomy matters because different bottlenecks dominate in different systems: shading cost, geometry cost, bandwidth, display hardware, or latency.

EyeNexus is a representative closed-loop streaming realization. It combines Foveated Spatial Compression (FSC) with Foveated Video Encoding (FVE), using real-time gaze tracking from the headset, mapping gaze into frame coordinates, applying AADT (Axis-aligned Distorted Transfer), then assigning per-macroblock Quantization Offset (QO) through NVENC in CONSTQP mode (Wu et al., 15 Sep 2025). Its distinctive feature is that the foveation controller $F$1 is both gaze-centered and network-aware, with AIMD-like updates using $F$2, $F$3, and $F$4. Under WiFi, it reports mean end-to-end latency of 69.4 ms, versus 77.7 ms for GCC and 79.3 ms for FovOptix; under mobile conditions it reports 69 ms, versus 74 ms for FovOptix and 237 ms for CGFVE. The abstract reports latency reduction of up to 70.9% and perceptual visual-quality improvement of up to 24.6%, while the user study reports improvements of up to 48% in playability and visual quality and states that motion sickness was eliminated (Wu et al., 15 Sep 2025).

The rendering literature highlights a complementary implementation pattern: post-process emulation rather than renderer redesign. The CUDA-optimized foveated visual system is explicitly an emulated foveated renderer that sits on top of an existing image stream and applies eccentricity-dependent Gaussian blur in real time (Malkin et al., 2020). It divides the frame into square fragments mapped to CUDA blocks, stores kernels in constant memory, tiles in shared memory, and performs separable horizontal and vertical convolution. On an NVIDIA GTX 1060, the kernel foveates a $F$5 image in 1.65 ms, and the full pipeline takes 4.4 ms with pinned memory, corresponding to about 165 Hz. Its main technical claim is that a very large number of pooling regions can be supported without stitching or blending artifacts.

A separate line of work addresses a bottleneck earlier in the pipeline: gaze tracking itself. FovealNet models the end-to-end TFR stack with

$F$6

and argues that mean gaze error is an inadequate target because the long tail in the error distribution inflates the rendered foveal radius and thus rendering latency (Liu et al., 2024). Its event-based cropping removes over 64.8% of irrelevant pixels, token pruning reduces ViT computation, and a latency-aware loss directly optimizes the downstream cost of gaze error. Under its performance-aware objective, FovealNet (0.0) reduces 95th-percentile error from 8.21° to 2.31°, maximum error from 24.2° to 5.22°, achieves at least 1.42× speedup compared to previous methods, and reports a 13% increase in perceptual quality for foveated output (Liu et al., 2024).

These systems collectively show that in VR-class FOVI, the operative problem is not merely where to place a high-resolution inset. It is a joint optimization over gaze estimation, perceptual falloff, bitrate allocation, GPU load, and temporal stability. The survey literature reinforces that artifacts in foveated systems are often temporal rather than purely spatial, motivating phase alignment, temporal anti-aliasing, reprojection, and learned reconstruction (Jabbireddy et al., 2022).

4. FOVI as sensing and computational imaging

FOVI is not restricted to display-side adaptation. The thermal meta-optics system demonstrates that the sensing function itself can be foveated (Saragadam et al., 2022). Its frontend uses three components: a foveal metalens, a perifoveal metalens, and a freely rotating linear polarizer. The foveal path has focal length $F$7 mm and diameter 75 mm; the perifoveal path has focal length $F$8 mm and diameter 25 mm. Because the two metalenses are polarization selective and complementary, the sensor records a temporal mixture

$F$9

where $e_0$0 is the high-resolution narrow-FoV foveal image and $e_0$1 is the lower-resolution wide-FoV perifoveal image (Saragadam et al., 2022).

The backend solves an inverse problem, either with least squares when mixing ratios are known or with a deep image prior (DIP) using two untrained U-Nets with skip connections, plus total variation and gradient exclusion loss (Saragadam et al., 2022). The prototype achieves 12 fps raw capture and, in the synchronized least-squares mode, 40 fps for a $e_0$2 image with less than 20 ms CPU time per frame; the robust DIP mode is much slower, with up to 50 seconds per video frame. The system reports a large-aperture, 1:6 foveal expansion, and in one analyzed setup a 9:1 foveation expansion when the scene is placed 300 mm from the foveal lens. The “in the wild” demonstrations on rooftops, parking lots, cars, moving humans, and heaters matter because they show that foveated sensing can preserve wide contextual awareness while concentrating optical acuity centrally.

The active-gaze scene-exploration work extends this sensing perspective into sequential perception. Here the camera is foveated at each fixation, a pretrained YOLOv3 detector is applied to the foveated image, and outputs are calibrated with a Foveal Observation Model using class- and distance-conditioned Dirichlet distributions across 7 distance levels (Dias et al., 2022). Evidence is fused into a body-centered semantic map using sum, product, or modified Kaplan updates, and the next fixation is chosen by minimizing expected map uncertainty. On 50 randomly selected images from the COCO 2017 validation set, active gaze control improves detection F1-score by about 2–3 percentage points after 10 fixations and reduces the number of required gaze shifts to about one third of random search (Dias et al., 2022).

A concise way to view these systems is that they implement FOVI “upstream” of display: the interface no longer merely presents a foveated image to an observer, but acquires, multiplexes, or explores the world through a foveated sensing policy.

Domain	Representative system	Key mechanism
VR streaming	EyeNexus	FSC + FVE with gaze- and bandwidth-aware control
Thermal imaging	All-silicon meta-optics	Polarization-multiplexed foveal/perifoveal capture
Active scene exploration	Foveal camera + semantic map	Uncertainty-driven next-best fixation

5. FOVI in deep vision and robotics

Recent work treats FOVI as an input interface for machine learning systems rather than only a human-facing display. The most explicit formulation is the biologically inspired interface that samples visual space in a retina-like pattern and remaps it onto a uniformly dense, V1-like sensor manifold (Blauch et al., 3 Feb 2026). Receptive fields are defined as k-nearest-neighborhoods (kNNs) on that manifold, and a novel kernel-mapping procedure aligns irregular neighborhoods to a common Cartesian reference frame, enabling kNN-convolution with weight sharing. This avoids the local anisotropy of log-polar and warped Cartesian mappings. In its CNN instantiation, the paper reports an inverted U-shaped relation between performance and foveation strength, with best ImageNet-1K performance around $e_0$3 under a 64×64 sample budget drawn from 256×256 ambient images, using 4 random fixations for training and up to 20 fixations for validation. In the ViT adaptation, FOVI-ViT-H+ @ 64 uses about 1/16 of the pixels and about 1/3 of the GFLOPs of the full-resolution baseline while reaching over 96% of the baseline accuracy in a single fixation (Blauch et al., 3 Feb 2026).

A different deep-learning formulation makes the foveated sensor itself differentiable. The active-vision architecture based on graph convolution introduces a foveated sampling layout inspired by Vogel’s sunflower model, an attention mechanism that predicts the next fixation from a soft saliency map, and a graph-convolution operator defined over irregularly sampled neighborhoods (Killick et al., 2023). On ImageNet-100, its foveated graph ConvNeXt exceeds uniform ConvNeXt by 2.5% top-1 accuracy at the same $e_0$4 pixel budget, and the sequential model with 3 fixations reaches 76.5%, above Learning to Zoom’s 75.8%. On ST-MNIST it reports about 11% gain over the uniform downsampling baseline, reinforcing the claim that active foveation is especially useful under large scale variation (Killick et al., 2023).

Robot learning work applies the same logic to tokenization. In the AV-ALOHA-based system, human operators teleoperate a bimanual robot with an active-vision arm while a Meta Quest Pro records gaze. Predicted gaze is then used to center a foveated patch tokenization in a ViT, with dense patches near gaze and sparse patches in the periphery (Chuang et al., 21 Jul 2025). The paper compares a Foveated pattern with 20 total tokens against a uniform 18 × 18 grid with 324 tokens and a 4 × 5 grid with 20 coarse tokens. The reduction from 324 to 20 corresponds to 16.2× fewer tokens, or about 94% fewer tokens. Reported ViT latency is 243.8 ms for Fine, 17.6 ms for Coarse, and 16.4 ms for Foveated; ViT GFLOPs are 1905.4, 126.9, and 115.6, respectively. Full-policy results on a single RTX 3090 show about 7× faster training and about 3× faster inference, with the strongest gains on distractor robustness and precision-heavy tasks such as ThreadNeedle and PourTestTube (Chuang et al., 21 Jul 2025).

Taken together, these works imply a shift in emphasis. Classical foveation often asks how to preserve perceived image quality for a human viewer. Deep-learning FOVI instead asks how to preserve task-relevant information density for a model under strict compute budgets. The common structure is still foveal/peripheral differentiation, but the downstream objective becomes classification, imitation learning, or representation transfer rather than display fidelity.

6. Evaluation, controversies, and open problems

Evaluation in FOVI spans psychophysics, system metrics, and task performance, and the literature repeatedly warns against overreliance on a single criterion. The display framework proposes pixel deficit, pixel waste, and RDF efficiency as technology-agnostic metrics grounded in ADF–RDF mismatch (Spjut et al., 2019). Streaming systems use end-to-end latency, perceptual image quality, and user studies. Machine-learning systems use top-1 accuracy, FLOPs, memory, token count, or policy success. A plausible implication is that FOVI has no universal scalar objective because the interface sits between perception, hardware, and task.

One important controversy is whether eccentricity-only foveation is perceptually faithful. The attention-aware model shows that when attention is diverted to a central RSVP task, peripheral thresholds can increase by almost $e_0$5 at $e_0$6 and more than $e_0$7 at $e_0$8 in the medium condition, and by up to $e_0$9 in the high condition (Krajancich et al., 2023). In a separate foveation-tolerance study with 30 adults, acceptable foveation slopes increase significantly for medium and high attention relative to low attention, with $S$0 after Bonferroni correction. This indicates that many existing FOVI models may be conservative because they overestimate peripheral sensitivity under fovea-centric tasks.

Another debate concerns how to predict human scene understanding under foveated viewing. The Foveated Scene Understanding Map (F-SUM) treats comprehension as a function of fixation-dependent access to semantics, using a VLM to describe foveated views and comparing those descriptions to gold-standard unfoveated captions (Wen et al., 19 May 2025). Over 277 scenes, the aggregate F-SUM score correlates with average human reaction time at $S$1 and with number of saccades at $S$2; it also correlates with description accuracy at $S$3 for 2 saccades and $S$4 for 4 saccades, exceeding standard clutter, image-complexity, and language-entropy baselines (Wen et al., 19 May 2025). This suggests that FOVI quality cannot be reduced to local blur visibility alone; the spatial distribution of semantically decisive information also matters.

Across the literature, several open problems recur. Dynamic systems remain sensitive to gaze latency and foveation error (Jabbireddy et al., 2022). Realistic attention measurement is still unresolved (Krajancich et al., 2023). Thermal and snapshot foveated imaging remain constrained by narrowband operation, approximately static scenes, and ill-posed recovery without temporal mixing (Saragadam et al., 2022). Deep-vision FOVI has not yet fully integrated saccade policy learning and temporal accumulation (Blauch et al., 3 Feb 2026). In VR, the strongest systems are increasingly closed-loop: gaze tracker, controller, renderer or codec, and quality predictor are co-designed rather than optimized independently (Liu et al., 2024).

The cumulative picture is that FOVI is moving from a local rendering heuristic toward a general systems framework for perceptually and computationally efficient active vision. Its mature form is likely to be multimodal and closed-loop: gaze-contingent, bandwidth-aware, attention-aware, and task-aware, with the fovea not merely rendered sharply but treated as the control locus through which sensing, transport, and inference are organized.