Eye-Perspective Rendering Techniques

• Eye-Perspective Rendering (EPR) is a technique that aligns rendered imagery with the viewer’s eye nodal point, providing accurate perspective compared to fixed camera models.
• It corrects distortions in parallax, disparity, and registration by dynamically accounting for the eye’s rotation and nodal displacement in near-eye displays and augmented reality systems.
• EPR is applied across VR, AR, and panoramic rendering pipelines using real-time eye tracking and proxy geometries, though challenges remain in calibration and latency minimization.

Searching arXiv for the cited Eye-Perspective Rendering papers and closely related work. Eye-Perspective Rendering (EPR) denotes rendering strategies that place the effective center of projection at the viewer’s eye perspective rather than at a fixed device-centered, headset-centered, or capture-centered camera. In near-eye displays, the canonical formulation updates the virtual camera so that it coincides with the eye’s instantaneous nodal point, not merely its center of rotation; in image-based augmented reality and panorama systems, the same principle appears as reprojection from a camera frame or capture viewpoint into the user’s eye frame or an adapted eye height. Across these variants, EPR addresses distortions in perspective, disparity, motion parallax, and registration that arise when rendered imagery is generated from the wrong viewpoint (Linton, 2019, Konrad et al., 2019, Emsenhuber et al., 15 Sep 2025).

1. Geometric basis and optical rationale

The geometric motivation for EPR is the separation between the eye’s center of rotation $C$ and its center of projection, typically identified with the anterior or front nodal point $N$. One treatment models the nodal point as approximately $6\,\text{mm}$ anterior to $C$, so that when the eye rotates by angle $\theta$, the physical nodal point translates laterally relative to the display window. In that model, a static virtual camera is correct only for straight-ahead gaze; EPR removes the induced angular and disparity errors by treating $N$ as the virtual camera position and recomputing the asymmetric viewing frustum around that moving point (Linton, 2019).

A closely related formulation in ocular parallax rendering uses the Gullstrand-Emsley schematic eye, with the front nodal point approximately $7.7\,\text{mm}$ anterior to the center of rotation. When the eye rotates by horizontal angle $\theta$ and vertical angle $\phi$, the nodal point becomes

$N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$

and projection of a world point $N$0 through $N$1 produces image coordinates

$N$2

Relative to zero eye rotation, the ocular-parallax image shift is $N$3 and $N$4; for small rotations and assuming $N$5, one can linearize to $N$6 and $N$7. This identifies ocular parallax as a depth-dependent retinal shift caused by the non-coincidence of rotation and projection centers (Konrad et al., 2019).

In optical see-through (OST) augmented reality, the same geometric issue appears at a larger systems level. Image-based annotations are computed on a world-facing camera displaced from the user’s eye, typically by $N$8–$N$9. Without eye-center reprojection, augmentations appear in front of or behind the intended object, producing parallax error, stereo conflict, and misregistration. EPR reprojects these annotations through the eye frame so that the displayed augmentation better coincides with what the user sees directly through the optics (Emsenhuber et al., 15 Sep 2025).

This suggests that EPR is best understood not as a single algorithm but as a family of viewpoint-correction techniques whose common invariant is the use of the eye, rather than the device, as the effective rendering origin.

2. The distortion family addressed by EPR

A detailed analytical treatment identifies 14 distinct gaze-contingent distortions that can occur when virtual cameras remain fixed while the eye rotates in near-eye displays. Using $6\,\text{mm}$0 for the vector from the eye’s rotation center to its nodal point, with $6\,\text{mm}$1, and $6\,\text{mm}$2 for the optical distance of the display, the nodal-point shift under eye rotation is

$6\,\text{mm}$3

From this shift follow several classes of error (Linton, 2019).

Visual direction of fixated objects: the retinal angular error is approximately $6\,\text{mm}$4. In VR, objects appear to require a corrective microsaccade; in AR, virtual objects at zero disparity are mis-directed relative to real ones.

Vergence insufficiency: both eyes incur equal and opposite $6\,\text{mm}$5, reducing binocular convergence by

$6\,\text{mm}$6

The stated consequence is near distances overestimated in VR and zero-disparity alignment achieved too far in AR.

Visual direction of peripheral objects (horizon tilt): a peripheral point at screen height $6\,\text{mm}$7 experiences the same nodal shift, changing its perceived elevation to

$6\,\text{mm}$8

Downward gaze makes the static horizon appear tilted upward.

Binocular disparity of non-fixated objects: for a peripheral object at depth $6\,\text{mm}$9,

$C$0

Diplopia cues are mis-reported and virtual depth cues conflict.

Relative visual angle between virtual objects and relative binocular disparity between virtual objects: EPR also targets changes in angular separation and in disparity differences between object pairs, with the latter approximated by

$C$1

Gaze-contingent visual direction (“pre-shifting”): a simple 2D pre-shift of the image by $C$2 removes gross horizon tilt but retains parallax errors for nearer points.

Gaze-contingent disparities: after pre-shifting, the effective inter-nodal distance reduces by approximately $C$3, producing

$C$4

This is singled out as the sole significant error for $C$5.

Motion-parallax and proprioceptive distortions: the analysis further includes missing motion parallax from eye movements, compounded conflicts from combined head and eye movements, mismatched parallax under head movements alone, proprioceptive cue-conflict, angular motion errors for moving objects, and distortion of stereoscopic 3D motion trajectories.

The practical importance of these terms depends strongly on optical distance. In a typical HMD optical model, the display’s optical distance is approximately $C$6, the display-to-nodal physical separation is approximately $C$7–$C$8 but optically magnified to appear at $C$9, the eye nodal-point offset is approximately $\theta$0, and off-axis lens distortions are static and can be pre-warped independently of gaze direction. Because directional errors scale as $\theta$1, most are negligible for modern HMDs; disparity errors instead scale as $\theta$2 and remain sizeable only when $\theta$3 is small, especially below $\theta$4 (Linton, 2019).

For the concrete example $\theta$5, $\theta$6, $\theta$7, and $\theta$8,

$\theta$9

In AR this shifts a virtual object at $N$0 by approximately $N$1 toward the observer relative to a real object at the same distance (Linton, 2019).

3. Platform-specific formulations

In stereoscopic VR, EPR is frequently realized as ocular parallax rendering. Per-frame eye tracking estimates fixation in head-centered coordinates, nodal points are computed for each eye, the usual eye-space translation of $N$2 is replaced by a translation of $N$3 in the view-to-eye transform, and an off-axis projection matrix is updated each frame so that the rendered stereo images incorporate the instantaneous eye perspective. In the reported prototype, this was implemented on an HTC Vive Pro with $N$4 pixels per eye and a $N$5 binocular eye tracker, with end-to-end gaze-to-render latency of approximately $N$6 (Konrad et al., 2019).

In near-field augmented reality, the same nodal-point correction primarily targets gaze-contingent disparity rather than global distance compression. The analytical result is explicit: for most modern HMDs with $N$7, only gaze-contingent disparity remains above human detection thresholds, primarily in near-field AR. This focuses EPR on applications such as precise registration in reaching space, assembly, and surgery rather than on generic far-field VR viewing (Linton, 2019).

In OST HMDs, the dominant problem is not only intraocular geometry but camera-eye misregistration. A software EPR pipeline reprojects camera-derived annotations onto proxy geometry and then re-renders them from the eye. Three proxy choices have been reported on Microsoft HoloLens 2: Plane-Proxy EPR with a single fixed-depth plane, Mesh-Proxy EPR using the SLAM reconstruction, and Gaze-Proxy EPR with a plane repositioned to the depth of the gaze target. Their common projection model uses calibrated camera intrinsics $N$8, eye intrinsics $N$9, and world transforms $7.7\,\text{mm}$0 and $7.7\,\text{mm}$1, with sampling in camera space and rendering in eye space (Emsenhuber et al., 15 Sep 2025).

A related handheld AR line treats the problem as user-perspective rendering rather than eye-perspective rendering. Here, video see-through output is corrected by estimating head or eye position relative to the device and rendering via an off-axis projection from that viewpoint instead of the back-facing camera. Adaptive User-Perspective Rendering reduces the cost of continuous head tracking by combining lightweight KLT-based motion estimation with intermittent full 3D face tracking, while preserving the eye-centric camera model (Mohr et al., 2017).

In pre-captured omnidirectional environments, the analogous viewpoint variable is eye height. Because the capture position is fixed, EPR requires synthesis of a new spherical panorama as if the camera had been at a different vertical offset $7.7\,\text{mm}$2. A multitask depth-plus-segmentation network, an omnidirectional-aware layered depth image, and semantic-guided inpainting together provide a means of adapting a single $7.7\,\text{mm}$3 image to altered eye height (Feng et al., 2023).

Off-center panorama rendering offers a further related formulation. Standard equirectangular-to-perspective rendering is distortion-free only when the virtual camera is at the panorama’s origin. Once the camera moves away from the center, barrel distortions appear. A modified pipeline using cylindrical intersection and an optimized dolly-zoom step significantly reduces those distortions, enabling more convincing “walk around” interaction in a single panorama, although without true occlusions or parallax (Peng et al., 2021).

4. Rendering pipelines and computational realizations

The minimal gaze-contingent EPR pipeline for near-eye displays is straightforward. One implementation guideline is: track eye orientation quaternion $7.7\,\text{mm}$4, compute nodal translation

$7.7\,\text{mm}$5

update virtual camera position $7.7\,\text{mm}$6, define an asymmetric frustum from the display window corners minus $7.7\,\text{mm}$7, and leave camera orientation fixed, aligned to head tracking, while translating only the camera position. Reported computational cost for projection-matrix recomputation is negligible, approximately $7.7\,\text{mm}$8 per frame, with eye-tracker update rates of $7.7\,\text{mm}$9–$\theta$0, eye-tracking latency of $\theta$1, frustum update of $\theta$2, and total added latency below $\theta$3 (Linton, 2019).

The ocular-parallax implementation refines this by explicitly computing fixation-based nodal points for each eye:

$\theta$4

Each vertex is then transformed by

$\theta$5

followed by an off-axis projection matrix whose frustum planes depend on $\theta$6, $\theta$7, $\theta$8, and the virtual image distance $\theta$9 (Konrad et al., 2019).

In OST HMD EPR, the pipeline is explicitly two-stage. For a world point $\phi$0,

$\phi$1

The color at $\phi$2 is sampled from the world-camera image and assigned to proxy geometry at $\phi$3, after which the proxy is re-rendered from the eye. Plane-Proxy EPR uses a single infinite plane parallel to the eye image plane; Mesh-Proxy EPR uses the SLAM mesh as projector target; Gaze-Proxy EPR moves a proxy plane to the current gaze depth, estimated by intersecting the fused gaze ray with the mesh. On HoloLens 2, Plane-Proxy incurs negligible overhead and easily exceeds $\phi$4, whereas Mesh-Proxy and Gaze-Proxy both report approximately $\phi$5 end-to-end, with Gaze-Proxy additionally showing approximately $\phi$6–$\phi$7 delay on gaze-plane updates (Emsenhuber et al., 15 Sep 2025).

Pre-captured $\phi$8 eye-height adaptation requires a substantially heavier image-synthesis pipeline. The reported system ingests an equirectangular RGB frame, predicts depth and semantic segmentation through a multitask network combining an equirectangular branch and a cubemap branch with feature exchange, constructs an omnidirectional-aware layered depth image by cutting connectivity at depth or semantic discontinuities, reprojects each layer from original camera center $\phi$9 to new center $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$0, and fills disocclusions using two partial-convolution U-Net style inpainting networks, one for depth and one for color with semantic guidance. Runtime is reported as $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$1 at $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$2, compared with OmniNeRF at more than $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$3 and $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$4 (Feng et al., 2023).

For off-center panoramas, the key computational change is replacement of spherical intersection by cylindrical intersection, which preserves vertical lines, followed by an optimization of a scalar offset $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$5 along the viewing ray to minimize a line-linearity distortion objective over a projected grid. The paper reports Google Ceres for the optimization, with less than $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$6 per pose on a 6-core CPU, average approximately $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$7 and peak approximately $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$8 on a GTX 1650 Ti, suitable for interactive use (Peng et al., 2021).

5. Empirical findings

The empirical literature does not support a single universal effect of EPR; instead it reports task- and modality-dependent benefits.

Paper	Setting	Reported result
(Konrad et al., 2019)	VR psychophysics	Ocular parallax is detectable and improves ordinal depth and realism
(Feng et al., 2023)	Pre-captured $N(\theta,\phi)=C+R(\theta,\phi)\cdot (N-C),$9 VR	Correct eye-height adaptation preserves immersion and improves distance accuracy
(Mohr et al., 2017)	Handheld AR	Adaptive user-perspective rendering lowers error and workload
(Emsenhuber et al., 15 Sep 2025)	OST HMD AR	Mesh-Proxy and Gaze-Proxy outperform fixed Plane-Proxy

In VR psychophysics, ocular parallax rendering produced a consistent detection threshold of approximately $N$00 at $N$01 correct, independent of absolute depth. In an ordinal depth task with neighboring textured surfaces separated by $N$02 or $N$03, static rendering was approximately chance at $N$04, whereas correct ocular parallax yielded $N$05 at $N$06 and $N$07 at $N$08; repeated-measures ANOVA reported significant main effects of rendering mode and separation, and post-hoc Tukey tests confirmed correct parallax better than static at $N$09. In a realism judgment, correct parallax was chosen over static in $N$10 of trials, but in an absolute egocentric depth task based on reaching, there was no significant difference between static stereo and ocular parallax rendering (Konrad et al., 2019).

The analytical EPR paper on depth perception in VR and AR explicitly states that it does not report a dedicated user study of EPR versus static rendering. It instead situates EPR against related literature in which near distances are overestimated by $N$11–$N$12 in static HMDs, adding eye-height or virtual offsets can reduce bias by approximately $N$13, and gaze-contingent disparity compensation theoretically reduces near-field AR placement errors from approximately $N$14–$N$15 to approximately $N$16. Its central claim is therefore theoretical and optical rather than behavioral: for typical HMDs, most gaze-contingent distortions are negligible except near-field disparity errors (Linton, 2019).

In pre-captured immersive environments, the user study used $N$17, four scenes, distances of $N$18–$N$19, and within-subjects eye-height offsets $N$20. Verbal estimates showed no significant change at $N$21 versus $N$22, but significant under-estimation at $N$23 with $N$24. Presence and ownership ratings indicated that $N$25 reduced ownership, whereas $N$26 and $N$27 preserved higher ownership. Quantitatively, on SynDepth360 the reported image synthesis metrics were SSIM $N$28, PSNR $N$29, and LPIPS $N$30, compared with OmniNeRF at SSIM $N$31, PSNR $N$32, and LPIPS $N$33 (Feng et al., 2023).

In handheld AR, a within-subjects study with $N$34 and $N$35 trials compared device-perspective rendering, full user-perspective rendering, fixed user-perspective rendering, and adaptive user-perspective rendering. Mean touch error was $N$36 px for device perspective, $N$37 px for full user perspective, $N$38 px for fixed user perspective, and $N$39 px for the adaptive method. NASA-TLX and SEQ similarly favored the adaptive condition, and preference votes were highest for it. Performance measurements on an HTC M8 over $N$40 frames reported frame times of approximately $N$41 for full user-perspective rendering, $N$42 for the adaptive method, and $N$43 for fixed user-perspective rendering (Mohr et al., 2017).

In OST AR on HoloLens 2, an $N$44-participant within-subject study compared Plane-Proxy, Gaze-Proxy, and Mesh-Proxy under wall-mounted and table orientations. Mean accuracy was $N$45 for Mesh-Proxy, $N$46 for Gaze-Proxy, and $N$47 for Plane-Proxy. Mean task completion time was $N$48 for Mesh-Proxy, $N$49 for Gaze-Proxy, and $N$50 for Plane-Proxy. Mean TLX overall was $N$51, $N$52, and $N$53, respectively. Reported alignment error at $N$54 was $N$55 for Mesh-Proxy and $N$56 for Gaze-Proxy. Participants ranked Mesh-Proxy first, Gaze-Proxy second, and Plane-Proxy third, although some ranked Gaze-Proxy equal to Mesh-Proxy in the wall condition (Emsenhuber et al., 15 Sep 2025).

6. Limitations, misconceptions, and research directions

A common misconception is that EPR is a general remedy for all distance-perception biases in VR. The analytical near-eye result is more restrictive: when the optical distance of the display exceeds $N$57, most gaze-contingent distortions are negligible, and only gaze-contingent disparities remain materially important, especially for near objects in AR. The same source states that in VR alone the benefits for perceived distance are modest, even if EPR can enhance natural parallax cues and reduce microsaccadic conflict (Linton, 2019).

Another misconception is that any off-center reprojection is equivalent to full 3D scene reconstruction. The panorama work is explicit that cylinder-based off-center projection and dolly-zoom reduce distortion and preserve vertical lines, but provide no true 3D occlusions or parallax; large translations still break the illusion (Peng et al., 2021). Similarly, fixed Plane-Proxy EPR in OST AR is computationally cheap but misaligns everywhere except on a single depth plane and was described by participants as “too annoying to adjust” or “actively hindering” (Emsenhuber et al., 15 Sep 2025).

Calibration and latency are recurring limits. Ocular-parallax rendering notes that latency of approximately $N$58 and eye-tracker precision of approximately $N$59 limit the crispness of micro-parallax, and that higher-rate trackers at $N$60–$N$61 with sub-degree accuracy would sharpen the effect (Konrad et al., 2019). OST Gaze-Proxy EPR reports perceptible $N$62–$N$63 update latency when the proxy plane jumps to a new gaze depth, while also requiring eye-calibration error at or below $N$64 and camera-intrinsics error at or below $N$65 pixel for precise registration (Emsenhuber et al., 15 Sep 2025).

Scene representation is likewise a bottleneck. Mesh-Proxy EPR depends on the quality and update rate of the SLAM mesh; pre-captured $N$66 eye-height adaptation currently operates at approximately $N$67 once layered reprojection and inpainting are included, and specular surfaces as well as temporal consistency in video remain unresolved. Reported future extensions include GPU fusion or tile-based reprojection with neural approximate inpainting, hybridization with reflective-region latent NeRFs, 3D U-Net extensions with optical-flow-guided hole masks, and higher-resolution encoders such as Swin-Transformer combined with latent diffusion inpainting (Feng et al., 2023).

A final research direction concerns accommodation-correct EPR. Ocular parallax rendering is described as a minimal-overhead instantiation of EPR; for “true full EPR,” the reported extension is to combine gaze-contingent projection with dynamic-focus optics or light-field or holographic displays so that the eye receives not only the correct projection center but also the correct focal distance for each depth layer (Konrad et al., 2019). This suggests that current EPR systems are best viewed as partial corrections within a broader program of display-side and rendering-side viewpoint fidelity.