---
title: 'RTGaze: Real-Time Gaze Tracking & Redirection'
url: https://www.emergentmind.com/topics/rtgaze
type: topic
---

# RTGaze: Real-Time Gaze Tracking & Redirection

RTGaze refers to multiple families of methods and systems addressing the challenges of real-time gaze tracking, estimation, and redirection. These methods cover gaze-controllable neural rendering, dynamic target tracking in AR environments, and lightweight model-based eye tracking. The term encompasses both neural image synthesis pipelines and frameworks for high-speed, robust eye-gaze inference, as evidenced in three major strands of the literature: real-time neural gaze redirection from single images [2511.11289], dynamic gaze monitoring and depth-level estimation for transparent displays and AR [2406.18595], and event-driven gaze tracking with lightweight segmentation [2201.07367].

## 1. Real-Time 3D-Aware Gaze Redirection from a Single Image

RTGaze [2511.11289] introduces a real-time, 3D-aware gaze redirection framework to synthesize photorealistic face images with controllable eye movements from only a single input image and a 2D gaze prompt $(g = (\text{pitch},\ \text{yaw}))$. The system targets the long-standing challenge of combining 3D consistency, high-quality synthesis, and inference efficiency for practical AR/VR and telepresence.

The pipeline consists of the following core stages:

- **Gaze-controllable representation learning:** Dual encoders extract high-frequency features (a CNN capturing eyelash and iris texture) and low-frequency features (a DeepLabV3 backbone with ViT encoder for global facial structure). The gaze prompt is projected by an MLP and injected via cross-attention into the high-frequency branch, then fused into a gaze-controllable latent code.
- **Triplane decoding and volumetric neural rendering:** The latent code is decoded to a triplane feature tensor (as in EG3D, representing three orthogonal feature planes). A neural renderer using ray marching and volume rendering synthetizes the redirected image for any camera pose.
- **Geometric prior distillation:** Depth maps from a pretrained 3D-GAN portrait generator serve as a geometric teacher. The system uses L1 loss between teacher and student depth maps for shape regularization.

At inference, only a single feedforward pass (no GAN inversion) is required. The end-to-end network achieves $\sim$0.06s per $512{\times}512$ image on an NVIDIA 3090 GPU, supporting real-time rates ($16$ FPS) with an $800{\times}$ speedup compared to previous 3D-aware neural radiance field (NeRF) approaches.

## 2. Dynamic Gaze Target Tracking and Depth-Level Estimation for Transparent Displays

A distinct RTGaze framework [2406.18595] addresses gaze monitoring for AR/transparent display (TD) applications, such as automotive heads-up displays. It combines dynamic 2D widget (“gaze target”) tracking with categorical depth estimation of human gaze (on-plane, out-plane-near, out-plane-far).

Key architectural modules include:

- **Dynamic Quadtree Target Tracking:** The display region is managed by a Quadtree data structure, enabling $O(\log n)$ widget insert/delete and sub-millisecond gaze queries. Each gaze point is mapped to its leaf node and overlapping/focused widget(s) are resolved by geometric containment and z-order priority.
- **Multi-Stream Self-Attention Depth Inference:** Parallel neural streams process gaze rotation vectors, eye positions, and the intersection/distance data. Intra-stream and inter-stream self-attention capture modality- and cross-modality dependencies, culminating in a lightweight softmax classifier for discrete depth assignment.

Latency is sub-millisecond on both CPUs and automotive-grade SoCs (e.g., TI TDA4 VM), with measured mean accuracy of $97.1\%\pm1.2\%$ and inference speed at $0.0007$s per sample (quantized). The framework supports highly dynamic, multi-widget AR interfaces and robustly segments gaze between the display and real-world background.

## 3. Event-Driven Gaze Tracking with Lightweight Segmentation

The RTGaze method [2201.07367] targets high-frequency, resource-constrained environments, such as near-eye AR/VR cameras operating at $>$30 Hz on mobile processors. The architecture is built around “Auto-ROI”:

- **Software-emulated event camera:** A binary event map is generated by thresholding normalized pixel intensity differences between consecutive frames, directly mimicking a hardware event camera’s output.
- **ROI prediction network:** A compact convolutional neural network (3 conv + 2 FC layers, $\approx$4.2K parameters) predicts the next region-of-interest from the event map, prior ROI, and edge maps.
- **ROI-based segmentation:** Only the predicted ROI (covering $18{-}32\%$ of pixels) is processed by a U-Net-style encoder-decoder with depthwise separable convolutions. When the eye is still, segmentation is extrapolated, reducing the compute load.
- **Model-based gaze vector estimation:** The segmented ROI is fit to a geometric eye model for 3D gaze estimation.

The complete pipeline achieves sub-0.5$^\circ$ mean gaze errors and $>30$ Hz real-time operation with a $5.5{\times}$ compute speedup over state-of-the-art full-frame gaze networks.

## 4. Mathematical Formulations and Losses

Each RTGaze variant incorporates distinct mathematical machinery:

- **Neural gaze redirection [2511.11289]:** The composite loss combines mask-guided 2D image reconstruction ($\mathcal{L}_R$) over eye and face masks, a depth distillation loss ($\mathcal{L}_D$) for matching teacher/student geometries, and a perceptual VGG-based loss ($\mathcal{L}_P$), weighted as
  $$
  \mathcal{L}_\text{total} = \alpha\, \mathcal{L}_R + \beta\, \mathcal{L}_D + \gamma\, \mathcal{L}_P.
  $$
- **Depth estimation [2406.18595]:** The categorical cross-entropy loss supervises the multi-class depth classifier, with intra- and inter-stream self-attention:
  $$
  \mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^C y_{i,c} \log \hat{p}_{i,c}.
  $$
- **Event-driven tracking [2201.07367]:** Event maps are generated as
  $$
  E_{t+1}(x,y) = \Phi\left(\frac{|F_t(x,y) - F_{t+1}(x,y)|}{F_t(x,y)}, \sigma\right).
  $$
  The segmentation loss is a composite of cross-entropy, edge, and shape consistency, and the ROI predictor is trained with mean-squared error.

## 5. Quantitative Evaluation and Benchmarking

Evaluation across all RTGaze systems uses standardized datasets, latency metrics, and accuracy scores.

**Gaze Redirection [2511.11289]:**

| Metric     | RTGaze | GazeNeRF | HeadNeRF    |
|------------|--------|----------|-------------|
| FID        | 38.3   | 81.8     | 69.5        |
| PSNR (dB)  | 19.0   | 15.45    | –           |
| LPIPS      | 0.262  | 0.291    | –           |
| SSIM       | 0.715  | 0.733    | –           |
| Time (s)   | 0.061  | 60.06    | –           |
| Gaze Error (°) | 9.05  | 6.94  | –           |
| ID Score   | 60.7   | 45.2     | –           |

RTGaze provides real-time performance with state-of-the-art FID and identity retention versus NeRF-based baselines, albeit with a slightly higher angular gaze error compared to GazeNeRF.

**Dynamic Tracking [2406.18595]:**  
Achieves $97.1\%$ classification accuracy for depth, $0.31\,\text{ms}$ latency for 12 dynamic widgets, and maintains sub-millisecond end-to-end processing.

**Event-Driven Tracking [2201.07367]:**  
End-to-end gaze error $<0.5^\circ$ at $32.5$ Hz ($40.2$ Hz for the small segmentation variant), with $5.7\times$ speedup over RITnet on the TEyeD dataset.

## 6. Ablation Studies, Limitations, and Prospects

All three research lines present systematic ablations:

- **Gaze feature injection in neural gaze redirection** is most effective in high-frequency branches; other injection points degrade FID or convergence [2511.11289]. 3D geometric distillation substantially improves identity and visual quality.
- **Self-attention modules in AR gaze depth estimation** are critical: removing intra-stream or inter-stream attention leads to $\sim$20–35% drops in classification accuracy [2406.18595].
- **ROI and event map feedback** in eye-tracking are essential for speed and mIoU; omitting these cues or using non-learning baselines increases error or compute [2201.07367].

Reported limitations include reliance on frontal priors, sensitivity to rapid head movements or widget overlaps, and imperfect robustness to occlusion and low-light scenarios. Prospective directions span multi-pose or self-supervised geometry learning, explicit eyeball modeling, temporal consistency for video, 3D widget layouts, and in-sensor integration for further latency and energy savings.

## 7. Contributions and Impact

RTGaze, in its multiple instantiations:

- Demonstrates efficient, real-time 3D-aware gaze synthesis for image redirection, with hybrid latent codes and geometric teacher distillation ([2511.11289]).
- Enables low-latency, high-accuracy attentional monitoring and interaction in AR environments with robust widget tracking and depth discrimination ([2406.18595]).
- Establishes a scalable paradigm for event-driven eye tracking optimized for mobile and embedded hardware, supporting rapid updates and low-power consumption ([2201.07367]).

The cross-domain applicability underlines RTGaze as an evolving suite of gaze analysis and manipulation tools for AR/VR, telepresence, mobile interaction, and automotive displays.

Source: https://www.emergentmind.com/topics/rtgaze