EyeSeg: Advances in Eye Image Segmentation

• EyeSeg is a framework for dense, per-pixel semantic segmentation of eye images that differentiates key components like background, sclera, iris, and pupil for applications such as gaze estimation and AR/VR.
• It leverages semi-supervised learning and domain-specific augmentations to mitigate challenges like label scarcity, occlusion, and motion blur in off-axis and varied imaging conditions.
• Geometry-aware approaches, including EllSeg and CondSeg, further refine segmentation by predicting full elliptical structures and integrating uncertainty metrics to boost downstream performance.

EyeSeg is associated in recent arXiv literature with semantic segmentation of eye images and with two specific frameworks bearing that name: a semi-supervised method for eye-part segmentation under label scarcity (Chaudhary et al., 2021) and an uncertainty-aware AR/VR segmentation framework designed for blur, occlusion, and domain shift (Peng et al., 13 Jul 2025). Across the broader literature, the underlying task is dense pixelwise labeling of cropped eye images into classes such as background, sclera or eye region, iris, and pupil, typically as an intermediate representation for pupil-center estimation, ellipse fitting, glint localization, avatar eye rendering, and gaze estimation in XR systems (Wu et al., 2019).

1. Problem setting and task definition

Eye segmentation is treated as a dense per-pixel semantic labeling problem in which structural eye components are explicitly separated for downstream geometric or appearance-based inference. In off-axis VR/MR settings, this task is difficult because the eye is imaged obliquely, with large pupil eccentricity, partial occlusion by eyelids and eyelashes, distracting reflections, and challenging near-eye imagery (Wu et al., 2019). The 2025 EyeSeg paper identifies three additional failure modes for deployment in AR/VR: motion blur, eyelid occlusion, and cross-domain gaps between training and deployment conditions (Peng et al., 13 Jul 2025).

The label space is not completely uniform across papers. EyeNet formulates segmentation as four-class dense prediction over Background, Sclera, Iris, and Pupil, with a grayscale input of size 160×120 and a 4-channel dense prediction map followed by a per-pixel softmax (Wu et al., 2019). The 2025 EyeSeg framework describes its cropped, closed-set formulation in terms of background, eye, iris, or pupil (Peng et al., 13 Jul 2025). OpenEDS2020 frames the segmentation subset as semantic eye segmentation with three foreground classes plus background, and its reported per-class segmentation scores are Background: 0.971, Sclera: 0.674, Iris: 0.882, Pupil: 0.835, with Average: 0.841 (Palmero et al., 2020).

The motivation is consistent across these works. The 2021 semi-supervised EyeSeg paper states that accurate and robust identification of pupil, iris, and sclera is central to modern appearance-based eye tracking because pupil center and iris geometry are used for gaze estimation, and accurate ellipse fitting improves robustness under eyelash or eyelid occlusion, reflections on cornea or glasses, off-axis camera placement, heavy makeup, and variation in illumination and gaze direction (Chaudhary et al., 2021). This suggests that “EyeSeg” is not merely a labeling exercise but a structural representation layer for later gaze-related inference.

2. Datasets, annotation regimes, and benchmark structure

Public benchmarks differ sharply in scale, sparsity, and intended use. OpenEDS2020 explicitly emphasizes temporal structure and sparse labels, whereas MagicEyes emphasizes scale, demographic diversity, and comprehensive annotation for multi-task XR pipelines (Palmero et al., 2020, Wu et al., 2020).

Dataset	Segmentation structure	Reported benchmark details
OpenEDS2020	200 sequences at 5 Hz from 74 different subjects	about 29,476 frames, 2,605 semantic segmentation masks, baseline mIoU 0.841 (Palmero et al., 2020)
OpenEDS-2019	8916 train, 2403 validation, 1440 test images	used by semi-supervised EyeSeg with mean IoU on validation (Chaudhary et al., 2021)
MagicEyes	587 subjects; Device 1 has 62K train and 18K test segmentation labels	EyeNet reports 97.29% averaged trace of the confusion matrix (Wu et al., 2020, Wu et al., 2019)

OpenEDS2020 is notable for its sparse-label protocol. Starting from roughly 600K images organized into 594 temporal sequences, the authors randomly chose 11,476 images for hand annotation, defined the sequence label ratio

$$R^i = \frac{N_{label}^i}{N_{total}^i} \times 100\%,$$

targeted about 5%, subsampled from 100 Hz to 5 Hz, and then selected the top 200 sequences sorted by decreasing label ratio, yielding about 29,476 frames and 2,605 semantic segmentation masks (Palmero et al., 2020). In evaluation, 5 annotated samples per sequence are hidden as test samples, resulting in roughly 5% labeled data for training and about 3% for testing. The paper explicitly positions this as a benchmark for temporal label propagation, spatio-temporal models for real-time video inference, few-shot learning, geometry-constrained semantic segmentation, and co-segmentation (Palmero et al., 2020).

Its annotation rules are unusually detailed. The right eye was horizontally flipped to align tear ducts with left-eye annotations; tear ducts are included as part of the eye region; in partial occlusion, annotators labeled only the visible parts of the pupil and iris; eyelashes are assigned to the underlying pupil or iris region; and during blinks, only a thin sliver of the eye region is labeled (Palmero et al., 2020). Label quality was checked by annotator agreement, with no labels produced by at least two annotators falling below 80% mIoU.

MagicEyes, by contrast, is a large-scale mixed-reality dataset collected with real XR devices. The dataset paper reports 587 subjects, over 3 million images total, over 800,000 gaze target-labeled images, and 80,000 images with human-labeled ground truth in the abstract (Wu et al., 2020). The EyeNet paper, which benchmarks segmentation on MagicEyes, states that the overall annotation corpus contains 87,000 annotated images for tasks including segmentation, glint presence or absence, glint 2D position, and pupil 2D position (Wu et al., 2019). Segmentation labels were produced with polygon marking and ellipse fitting tools, with manual polygons for background, sclera, iris, and pupil (Wu et al., 2019).

3. Semi-supervised EyeSeg

The 2021 EyeSeg paper formulates eye-part segmentation as learning

$f: \mathcal{X} \rightarrow \mathcal{Y},$

with labeled examples $\mathcal{X}_l, \mathcal{Y}_l$, unlabeled examples $\mathcal{X}_u$, and $k < m$ (Chaudhary et al., 2021). Its baseline supervised model, denoted $\mathcal{S}_L$, uses only labeled data and optimizes

$$\mathcal{L}_{sup} = \mathcal{L}_{CEL}(\lambda_1 + \lambda_2 \mathcal{L}_{BAL}) + \lambda_3 \mathcal{L}_{SL}.$$

The paper argues that standard consistency-based semi-supervision from classification is insufficient for segmentation because the output is spatially structured: when the input is rotated or translated, the label map must transform accordingly. It therefore proposes two frameworks. The first, $\mathcal{SSL}_D$, uses domain-specific augmentations that do not change pixel locations, specifically Gamma correction and CLAHE. For each unlabeled image $x_u$, it averages predictions over $A$ augmentations,

$f: \mathcal{X} \rightarrow \mathcal{Y},$0

and imposes an $f: \mathcal{X} \rightarrow \mathcal{Y},$1 consistency penalty,

$f: \mathcal{X} \rightarrow \mathcal{Y},$2

with total loss

$f: \mathcal{X} \rightarrow \mathcal{Y},$3

The second framework, $f: \mathcal{X} \rightarrow \mathcal{Y},$4, extends this to spatially varying transformations by predicting in transformed space and inverse-mapping the output: $f: \mathcal{X} \rightarrow \mathcal{Y},$5 and optimizing

$f: \mathcal{X} \rightarrow \mathcal{Y},$6

The geometric transforms are rotation in $f: \mathcal{X} \rightarrow \mathcal{Y},$7 and translation in $f: \mathcal{X} \rightarrow \mathcal{Y},$8 pixels, applied with 50% probability for rotation and 80% probability for translation (Chaudhary et al., 2021).

The implementation uses RITnet as backbone, input size $f: \mathcal{X} \rightarrow \mathcal{Y},$9, Adam, learning rate $\mathcal{X}_l, \mathcal{Y}_l$0, batch size 8 with 4 labeled and 4 unlabeled samples, and 250 epochs (Chaudhary et al., 2021). On OpenEDS-2019, the headline low-label result is that with only 48 labeled images, the two semi-supervised frameworks improve segmentation performance by 0.38% and 0.65% over the supervised baseline. The paper also reports that $\mathcal{X}_l, \mathcal{Y}_l$1 improves the baseline by up to 0.21% in one low-label setting, and $\mathcal{X}_l, \mathcal{Y}_l$2 improves further by up to 0.33% over $\mathcal{X}_l, \mathcal{Y}_l$3. For class-wise IoU in the extreme 4 labeled images regime, the gain from baseline to $\mathcal{X}_l, \mathcal{Y}_l$4 is around 4.08% for pupil and 4.48% for iris (Chaudhary et al., 2021). This suggests that unlabeled eye images are most valuable when labeled subject diversity or annotation volume is severely limited.

4. Geometry-aware reformulations: EllSeg and CondSeg

A major line of work adjacent to EyeSeg redefines the segmentation target so that it is more directly aligned with geometric inference. EllSeg does this by replacing visible-part segmentation with segmentation of the entire elliptical structures of the pupil and iris, despite occlusion (Kothari et al., 2020). Instead of predicting only what is visible, the network predicts a 3-class semantic segmentation over background, iris, and pupil, where the iris and pupil masks correspond to full ellipse regions. The motivation is explicit: conventional pipelines segment visible parts, extract edges, and fit ellipses afterward, but ellipse fitting is brittle when eyelids, eyelashes, reflections, or off-axis placement remove too much contour evidence (Kothari et al., 2020).

EllSeg is evaluated with DenseElNet, RITnet, and DeepVOG backbones, and combines a segmentation loss

$\mathcal{X}_l, \mathcal{Y}_l$5

with a center-of-mass loss $\mathcal{X}_l, \mathcal{Y}_l$6 (Kothari et al., 2020). Its abstract reports at least 10% and 24% increases in pupil and iris center detection rate respectively within a two-pixel error margin compared to standard eye-parts segmentation. The paper also reports dataset-specific gains in detection rate for EllSeg over PartSeg, including 11% pupil-center improvement on NVGaze, 12% on RIT-Eyes, 29% iris-center improvement on NVGaze, and 25% on RIT-Eyes (Kothari et al., 2020).

CondSeg further systematizes the geometric viewpoint by introducing two priors: an ellipse prior, in which projected pupil and iris boundaries are modeled by ellipses parameterized as

$\mathcal{X}_l, \mathcal{Y}_l$7

and a condition prior, in which the visible pupil or iris is the intersection of the full ellipse with the eye-region mask (Jia et al., 2024). The key relation is

$\mathcal{X}_l, \mathcal{Y}_l$8

CondSeg converts ellipse parameters into differentiable soft masks and trains against visible segmentation labels through binary cross entropy, thereby estimating full ellipses without explicitly annotating full ellipses (Jia et al., 2024).

On OpenEDS-2019, CondSeg reports pupil 90.91 and iris-region 94.37 IoU, with center errors 1.48 for pupil and 3.42 for iris; on OpenEDS-2020, it reports pupil 86.80 and iris-region 91.83 IoU, with center errors 1.61 and 5.91 (Jia et al., 2024). The paper explicitly states that CondSeg is generally competitive with direct regression while requiring no ellipse labels. A plausible implication is that geometry-aware supervision can reduce the annotation gap between visible masks and the full structures needed by downstream gaze pipelines.

5. Segmentation as a component of multi-task, synthetic-to-real, and temporal gaze systems

In EyeNet, eye segmentation is both a standalone task and the representation-learning anchor for a broader off-axis XR pipeline (Wu et al., 2019). The segmentation branch is trained first because it provides “the richest semantic information” and is “the most complicated supervised task to train accurately.” It is then jointly optimized with pupil center localization and glint localization, after which glint predictions are used for model-based supervision of the cornea branch (Wu et al., 2019). The segmentation decoder is described as similar to SegNet and U-Net, and the branch is optimized with pixel-wise cross-entropy,

$\mathcal{X}_l, \mathcal{Y}_l$9

On MagicEyes, EyeNet reports a confusion-matrix averaged trace of 97.29%, with diagonal entries 96.25 for pupil, 99.03 for iris, 96.71 for sclera, and 97.18 for background (Wu et al., 2019).

A different integration strategy appears in “Gaze Estimation with Eye Region Segmentation and Self-Supervised Multistream Learning,” where segmentation is not the final objective but a structural conditioning signal for gaze regression (Mahmud et al., 2021). That pipeline trains a U-Net type model on 60,000 synthetic eye images from UnityEyes to predict two masks—iris and visible eyeball—then uses the predicted masks on real images as inputs to a three-stream gaze network: raw eye image, visible eyeball mask, and iris mask. The raw-eye encoder is pretrained with SimCLR-style contrastive learning, and the full multistream model regresses pitch and yaw with MSE loss (Mahmud et al., 2021). On EYEDIAP, the reported mean angular errors are 6.29 and 6.48 for the multistream network under LOSO and 5-fold, improving to 6.15 and 6.34 with SSL pretraining (Mahmud et al., 2021).

OpenEDS2020 complements these model-centric papers by explicitly constructing a temporal benchmark in which only about 5% of frames are labeled, while future methods are expected to use short-range continuity to propagate information to neighboring frames (Palmero et al., 2020). The baseline deliberately does not exploit temporal information; it uses a lightweight encoder-decoder loosely based on SegNet, with 7 downscaler blocks, 7 upscaler blocks, separable convolution, LeakyReLU, multiplicative skip connections, 40K parameters, and 300 KB model size, trained on 1,605 images for 150 epochs with Adam, initial learning rate 0.004, and batch size 128 (Palmero et al., 2020). This suggests that the literature uses EyeSeg both as an explicit segmentation endpoint and as a scaffold for richer temporal or multi-task eye understanding.

6. Uncertainty-aware EyeSeg and the boundary of the concept

The 2025 EyeSeg framework adds an explicit uncertainty model on top of a deterministic segmentation backbone (Peng et al., 13 Jul 2025). Its pipeline has three components: an eye detector that crops the eye patch, a deterministic segmentation network $\mathcal{X}_u$0 producing latent features $\mathcal{X}_u$1, and an uncertainty-aware projection head $\mathcal{X}_u$2 that predicts a diagonal covariance matrix in latent space. After eye detection and cropping, the label set is closed, and the paper defines a class-dependent prior centered at

$\mathcal{X}_u$3

The standard segmentation loss is

$\mathcal{X}_u$4

Its main theoretical statement is Theorem 1, which says that for a pixel $\mathcal{X}_u$5, the trace of the optimal covariance equals the squared Euclidean distance between the latent code and the corresponding class center: $\mathcal{X}_u$6 The practical uncertainty score for a whole eye patch is

$\mathcal{X}_u$7

The model then uses this scalar to reject unreliable outputs or to weight and fuse multiple gaze estimates (Peng et al., 13 Jul 2025).

The reported empirical results are strong. On OpenEDS, Ours(E) achieves 98.6 iris MIoU and 97.8 pupil MIoU, with average MIoU 94.8. The same variant is reported as substantially cheaper than several baselines: 1.53G FLOPs versus 39.41G FLOPs for DeepLabv3+, 16.57G FLOPs for RITNet, and 21.15G FLOPs for Pylids (Peng et al., 13 Jul 2025). For uncertainty-based filtering, EyeSeg achieves retained MIoU around 89.32–89.72 in-domain, 86.56–87.15 cross-domain, 89.23–89.60 under occlusion, and 88.04–88.36 under blur (Peng et al., 13 Jul 2025). The paper explicitly states that these uncertainty scores outperform baselines such as Ensemble, EvPSNet, Dudes, and BiTrans on difficult cases.

A common simplification is to equate EyeSeg with eye tracking as a whole. The cited literature does not support that equivalence. Some pipelines make segmentation foundational, but others do not use explicit segmentation at all. “Towards End-to-end Video-based Eye-Tracking” states that it contains no explicit eye segmentation network, no mask prediction, and no region-level semantic segmentation objective; instead, it uses normalized 128×128 eye patches, temporal recurrence, and screen-content-conditioned refinement (Park et al., 2020). The biosignal-based systems earEOG and ElectraSight are even further from image segmentation: earEOG uses 14 electrodes around the ears in a headphone form factor and shows strong horizontal but weak vertical eye-tracking performance (King et al., 8 Jun 2025), while ElectraSight is a fully onboard smart-glasses system using hybrid contact and contactless EOG/QVar sensing with 81% accuracy for 10 classes and 92% for 6 classes (Schärer et al., 2024). This suggests that EyeSeg is best understood not as a synonym for all eye tracking, but as one technically specific and highly influential representation strategy within a broader eye-sensing landscape.