---
title: 'EyeSeg: Advances in Eye Image Segmentation'
url: https://www.emergentmind.com/topics/eyeseg
type: topic
---

# EyeSeg: Advances in Eye Image Segmentation

EyeSeg is associated in recent arXiv literature with semantic segmentation of eye images and with two specific frameworks bearing that name: a semi-supervised method for eye-part segmentation under label scarcity [2103.09369] and an uncertainty-aware AR/VR segmentation framework designed for blur, occlusion, and domain shift [2507.09649]. Across the broader literature, the underlying task is dense pixelwise labeling of cropped eye images into classes such as background, sclera or eye region, iris, and pupil, typically as an intermediate representation for pupil-center estimation, ellipse fitting, glint localization, avatar eye rendering, and gaze estimation in XR systems [1908.09060].

## 1. Problem setting and task definition

Eye segmentation is treated as a dense per-pixel semantic labeling problem in which structural eye components are explicitly separated for downstream geometric or appearance-based inference. In off-axis VR/MR settings, this task is difficult because the eye is imaged obliquely, with large pupil eccentricity, partial occlusion by eyelids and eyelashes, distracting reflections, and challenging near-eye imagery [1908.09060]. The 2025 EyeSeg paper identifies three additional failure modes for deployment in AR/VR: motion blur, eyelid occlusion, and cross-domain gaps between training and deployment conditions [2507.09649].

The label space is not completely uniform across papers. EyeNet formulates segmentation as four-class dense prediction over **Background**, **Sclera**, **Iris**, and **Pupil**, with a grayscale input of size **160×120** and a **4-channel dense prediction map** followed by a per-pixel softmax [1908.09060]. The 2025 EyeSeg framework describes its cropped, closed-set formulation in terms of **background, eye, iris, or pupil** [2507.09649]. OpenEDS2020 frames the segmentation subset as semantic eye segmentation with three foreground classes plus background, and its reported per-class segmentation scores are **Background: 0.971**, **Sclera: 0.674**, **Iris: 0.882**, **Pupil: 0.835**, with **Average: 0.841** [2005.03876].

The motivation is consistent across these works. The 2021 semi-supervised EyeSeg paper states that accurate and robust identification of **pupil, iris, and sclera** is central to modern appearance-based eye tracking because **pupil center and iris geometry** are used for gaze estimation, and accurate **ellipse fitting** improves robustness under eyelash or eyelid occlusion, reflections on cornea or glasses, off-axis camera placement, heavy makeup, and variation in illumination and gaze direction [2103.09369]. This suggests that “EyeSeg” is not merely a labeling exercise but a structural representation layer for later gaze-related inference.

## 2. Datasets, annotation regimes, and benchmark structure

Public benchmarks differ sharply in scale, sparsity, and intended use. OpenEDS2020 explicitly emphasizes temporal structure and sparse labels, whereas MagicEyes emphasizes scale, demographic diversity, and comprehensive annotation for multi-task XR pipelines [2005.03876, 2003.08806].

| Dataset | Segmentation structure | Reported benchmark details |
|---|---|---|
| OpenEDS2020 | **200 sequences** at **5 Hz** from **74 different subjects** | about **29,476 frames**, **2,605 semantic segmentation masks**, baseline **mIoU 0.841** [2005.03876] |
| OpenEDS-2019 | **8916** train, **2403** validation, **1440** test images | used by semi-supervised EyeSeg with **mean IoU** on validation [2103.09369] |
| MagicEyes | **587 subjects**; Device 1 has **62K** train and **18K** test segmentation labels | EyeNet reports **97.29%** averaged trace of the confusion matrix [2003.08806, 1908.09060] |

OpenEDS2020 is notable for its sparse-label protocol. Starting from roughly **600K images organized into 594 temporal sequences**, the authors randomly chose **11,476 images for hand annotation**, defined the sequence label ratio
\[
R^i = \frac{N_{label}^i}{N_{total}^i} \times 100\%,
\]
targeted about **5%**, subsampled from **100 Hz** to **5 Hz**, and then selected the **top 200 sequences sorted by decreasing label ratio**, yielding about **29,476 frames** and **2,605 semantic segmentation masks** [2005.03876]. In evaluation, **5 annotated samples per sequence** are hidden as test samples, resulting in roughly **5% labeled data for training and about 3% for testing**. The paper explicitly positions this as a benchmark for **temporal label propagation**, **spatio-temporal models for real-time video inference**, **few-shot learning**, **geometry-constrained semantic segmentation**, and **co-segmentation** [2005.03876].

Its annotation rules are unusually detailed. The **right eye was horizontally flipped** to align tear ducts with left-eye annotations; **tear ducts are included as part of the eye region**; in **partial occlusion**, annotators labeled only the visible parts of the pupil and iris; **eyelashes** are assigned to the underlying pupil or iris region; and during **blinks**, only a **thin sliver of the eye region** is labeled [2005.03876]. Label quality was checked by annotator agreement, with no labels produced by at least two annotators falling below **80% mIoU**.

MagicEyes, by contrast, is a large-scale mixed-reality dataset collected with real XR devices. The dataset paper reports **587 subjects**, **over 3 million images** total, **over 800,000 gaze target-labeled images**, and **80,000 images with human-labeled ground truth** in the abstract [2003.08806]. The EyeNet paper, which benchmarks segmentation on MagicEyes, states that the overall annotation corpus contains **87,000 annotated images** for tasks including segmentation, glint presence or absence, glint 2D position, and pupil 2D position [1908.09060]. Segmentation labels were produced with **polygon marking and ellipse fitting tools**, with manual polygons for **background**, **sclera**, **iris**, and **pupil** [1908.09060].

## 3. Semi-supervised EyeSeg

The 2021 EyeSeg paper formulates eye-part segmentation as learning
\[
f: \mathcal{X} \rightarrow \mathcal{Y},
\]
with labeled examples \(\mathcal{X}_l, \mathcal{Y}_l\), unlabeled examples \(\mathcal{X}_u\), and \(k < m\) [2103.09369]. Its baseline supervised model, denoted \(\mathcal{S}_L\), uses only labeled data and optimizes
\[
\mathcal{L}_{sup} = \mathcal{L}_{CEL}(\lambda_1 + \lambda_2 \mathcal{L}_{BAL}) + \lambda_3 \mathcal{L}_{SL}.
\]

The paper argues that standard consistency-based semi-supervision from classification is insufficient for segmentation because the output is spatially structured: when the input is rotated or translated, the label map must transform accordingly. It therefore proposes two frameworks. The first, \(\mathcal{SSL}_D\), uses **domain-specific augmentations** that do not change pixel locations, specifically **Gamma correction** and **CLAHE**. For each unlabeled image \(x_u\), it averages predictions over \(A\) augmentations,
\[
y_u = \frac{1}{A}\sum_{a=1}^{A} f(x_{u,a}; \theta),
\]
and imposes an \(L_2\) consistency penalty,
\[
\mathcal{L}_u = \| f(x) - y_u \|_2^2,
\]
with total loss
\[
\mathcal{L} = \mathcal{L}_{sup} + \lambda_u \mathcal{L}_u.
\]

The second framework, \(\mathcal{SSL}_{ss}\), extends this to **spatially varying transformations** by predicting in transformed space and inverse-mapping the output:
\[
y_u = \frac{1}{A}\sum_{a=1}^{A} \mathcal{T}^{-1}\left(f(\mathcal{T}(x_{u,a}); \theta)\right),
\]
and optimizing
\[
\mathcal{L} = \mathcal{L}_{sup} + \lambda_u \mathcal{L}_u + \lambda_{ss}\mathcal{L}_{ss}.
\]
The geometric transforms are rotation in \([-5^\circ, 5^\circ]\) and translation in \([-20, 20]\) pixels, applied with **50% probability for rotation** and **80% probability for translation** [2103.09369].

The implementation uses **RITnet** as backbone, input size \(240 \times 320\), **Adam**, learning rate \(0.001\), batch size **8** with **4 labeled** and **4 unlabeled** samples, and **250 epochs** [2103.09369]. On OpenEDS-2019, the headline low-label result is that with only **48 labeled images**, the two semi-supervised frameworks improve segmentation performance by **0.38%** and **0.65%** over the supervised baseline. The paper also reports that \(\mathcal{SSL}_D\) improves the baseline by up to **0.21%** in one low-label setting, and \(\mathcal{SSL}_{ss}\) improves further by up to **0.33%** over \(\mathcal{SSL}_D\). For class-wise IoU in the extreme **4 labeled images** regime, the gain from baseline to \(\mathcal{SSL}_{ss}\) is around **4.08%** for pupil and **4.48%** for iris [2103.09369]. This suggests that unlabeled eye images are most valuable when labeled subject diversity or annotation volume is severely limited.

## 4. Geometry-aware reformulations: EllSeg and CondSeg

A major line of work adjacent to EyeSeg redefines the segmentation target so that it is more directly aligned with geometric inference. EllSeg does this by replacing visible-part segmentation with segmentation of the **entire elliptical structures** of the **pupil** and **iris**, despite occlusion [2007.09600]. Instead of predicting only what is visible, the network predicts a **3-class semantic segmentation** over **background**, **iris**, and **pupil**, where the iris and pupil masks correspond to full ellipse regions. The motivation is explicit: conventional pipelines segment visible parts, extract edges, and fit ellipses afterward, but ellipse fitting is brittle when eyelids, eyelashes, reflections, or off-axis placement remove too much contour evidence [2007.09600].

EllSeg is evaluated with **DenseElNet**, **RITnet**, and **DeepVOG** backbones, and combines a segmentation loss
\[
\mathcal{L}_{SEG} = \mathcal{L}_{CEL} (\lambda_1 + \lambda_2 \mathcal{L}_{BAL} ) + \lambda_3 \mathcal{L}_{GDL} + \lambda_4 \mathcal{L}_{SL}
\]
with a center-of-mass loss \(\mathcal{L}_{COM}\) [2007.09600]. Its abstract reports at least **10%** and **24%** increases in **pupil** and **iris** center detection rate respectively within a **two-pixel error margin** compared to standard eye-parts segmentation. The paper also reports dataset-specific gains in detection rate for EllSeg over PartSeg, including **11%** pupil-center improvement on **NVGaze**, **12%** on **RIT-Eyes**, **29%** iris-center improvement on **NVGaze**, and **25%** on **RIT-Eyes** [2007.09600].

CondSeg further systematizes the geometric viewpoint by introducing two priors: an **ellipse prior**, in which projected pupil and iris boundaries are modeled by ellipses parameterized as
\[
(x_0, y_0, a, b, \theta),
\]
and a **condition prior**, in which the visible pupil or iris is the intersection of the full ellipse with the eye-region mask [2408.17231]. The key relation is
\[
R_p^v = R_p^f \cap R_e.
\]
CondSeg converts ellipse parameters into differentiable soft masks and trains against visible segmentation labels through **binary cross entropy**, thereby estimating full ellipses **without explicitly annotating full ellipses** [2408.17231].

On **OpenEDS-2019**, CondSeg reports **pupil 90.91** and **iris-region 94.37** IoU, with center errors **1.48** for pupil and **3.42** for iris; on **OpenEDS-2020**, it reports **pupil 86.80** and **iris-region 91.83** IoU, with center errors **1.61** and **5.91** [2408.17231]. The paper explicitly states that CondSeg is generally **competitive with direct regression** while requiring **no ellipse labels**. A plausible implication is that geometry-aware supervision can reduce the annotation gap between visible masks and the full structures needed by downstream gaze pipelines.

## 5. Segmentation as a component of multi-task, synthetic-to-real, and temporal gaze systems

In EyeNet, eye segmentation is both a standalone task and the representation-learning anchor for a broader off-axis XR pipeline [1908.09060]. The segmentation branch is trained first because it provides “the richest semantic information” and is “the most complicated supervised task to train accurately.” It is then jointly optimized with **pupil center localization** and **glint localization**, after which glint predictions are used for model-based supervision of the cornea branch [1908.09060]. The segmentation decoder is described as similar to **SegNet and U-Net**, and the branch is optimized with pixel-wise cross-entropy,
\[
\mathcal{L}(x, y) = -\sum^{4}_{k=1} I_{x,y}[k==c] \log{p_k(x,y)}.
\]
On MagicEyes, EyeNet reports a confusion-matrix averaged trace of **97.29%**, with diagonal entries **96.25** for pupil, **99.03** for iris, **96.71** for sclera, and **97.18** for background [1908.09060].

A different integration strategy appears in “Gaze Estimation with Eye Region Segmentation and Self-Supervised Multistream Learning,” where segmentation is not the final objective but a structural conditioning signal for gaze regression [2112.07878]. That pipeline trains a **U-Net type model** on **60,000 synthetic eye images** from **UnityEyes** to predict two masks—**iris** and **visible eyeball**—then uses the predicted masks on real images as inputs to a three-stream gaze network: raw eye image, visible eyeball mask, and iris mask. The raw-eye encoder is pretrained with SimCLR-style contrastive learning, and the full multistream model regresses **pitch** and **yaw** with **MSE loss** [2112.07878]. On **EYEDIAP**, the reported mean angular errors are **6.29** and **6.48** for the multistream network under **LOSO** and **5-fold**, improving to **6.15** and **6.34** with SSL pretraining [2112.07878].

OpenEDS2020 complements these model-centric papers by explicitly constructing a temporal benchmark in which only about **5% of frames are labeled**, while future methods are expected to use short-range continuity to propagate information to neighboring frames [2005.03876]. The baseline deliberately **does not exploit temporal information**; it uses a lightweight encoder-decoder loosely based on **SegNet**, with **7 downscaler blocks**, **7 upscaler blocks**, **separable convolution**, **LeakyReLU**, **multiplicative skip connections**, **40K parameters**, and **300 KB** model size, trained on **1,605 images** for **150 epochs** with **Adam**, initial learning rate **0.004**, and batch size **128** [2005.03876]. This suggests that the literature uses EyeSeg both as an explicit segmentation endpoint and as a scaffold for richer temporal or multi-task eye understanding.

## 6. Uncertainty-aware EyeSeg and the boundary of the concept

The 2025 EyeSeg framework adds an explicit uncertainty model on top of a deterministic segmentation backbone [2507.09649]. Its pipeline has three components: an **eye detector** that crops the eye patch, a deterministic segmentation network \(f_\phi\) producing latent features \(\boldsymbol{z}\), and an **uncertainty-aware projection head** \(\Lambda_\theta\) that predicts a **diagonal covariance matrix** in latent space. After eye detection and cropping, the label set is closed, and the paper defines a class-dependent prior centered at
\[
\boldsymbol{c}_{\boldsymbol{y}_{i,j}} := \boldsymbol{W}^T\boldsymbol{e}_{\boldsymbol{y}_{i,j}}.
\]
The standard segmentation loss is
\[
\mathcal{L}_{\text{seg}} = \mathbb{E}_{(\boldsymbol{x}, \boldsymbol{y}) \sim \mathcal{D}} \left[-\sum_{i,j} \boldsymbol{y}_{i,j}\log \boldsymbol{p}_{i,j}\right].
\]

Its main theoretical statement is **Theorem 1**, which says that for a pixel \((i,j)\), the trace of the optimal covariance equals the squared Euclidean distance between the latent code and the corresponding class center:
\[
\operatorname{tr}(\Lambda_{\boldsymbol{x}_{i,j}}^*) = ||\boldsymbol{z}_{i,j} - \boldsymbol{c}_{\boldsymbol{y}_{i,j}}||_2^2.
\]
The practical uncertainty score for a whole eye patch is
\[
s_{\text{unc}} = \sum_{i,j} \ln {\operatorname{det}(\Lambda_{\boldsymbol{x}_{i,j}})}.
\]
The model then uses this scalar to reject unreliable outputs or to weight and fuse multiple gaze estimates [2507.09649].

The reported empirical results are strong. On **OpenEDS**, **Ours(E)** achieves **98.6 iris MIoU** and **97.8 pupil MIoU**, with **average MIoU 94.8**. The same variant is reported as substantially cheaper than several baselines: **1.53G FLOPs** versus **39.41G FLOPs** for **DeepLabv3+**, **16.57G FLOPs** for **RITNet**, and **21.15G FLOPs** for **Pylids** [2507.09649]. For uncertainty-based filtering, EyeSeg achieves retained MIoU around **89.32–89.72** in-domain, **86.56–87.15** cross-domain, **89.23–89.60** under occlusion, and **88.04–88.36** under blur [2507.09649]. The paper explicitly states that these uncertainty scores outperform baselines such as **Ensemble**, **EvPSNet**, **Dudes**, and **BiTrans** on difficult cases.

A common simplification is to equate EyeSeg with eye tracking as a whole. The cited literature does not support that equivalence. Some pipelines make segmentation foundational, but others do not use explicit segmentation at all. “Towards End-to-end Video-based Eye-Tracking” states that it contains **no explicit eye segmentation network**, no mask prediction, and no region-level semantic segmentation objective; instead, it uses normalized **128×128 eye patches**, temporal recurrence, and screen-content-conditioned refinement [2007.13120]. The biosignal-based systems **earEOG** and **ElectraSight** are even further from image segmentation: earEOG uses **14 electrodes** around the ears in a headphone form factor and shows strong horizontal but weak vertical eye-tracking performance [2506.07193], while ElectraSight is a fully onboard smart-glasses system using hybrid contact and contactless EOG/QVar sensing with **81% accuracy for 10 classes** and **92% for 6 classes** [2412.14848]. This suggests that EyeSeg is best understood not as a synonym for all eye tracking, but as one technically specific and highly influential representation strategy within a broader eye-sensing landscape.

Source: https://www.emergentmind.com/topics/eyeseg