Seeing Eye: Multi-Modal Visual and Gaze Systems

• Seeing Eye is a systems term describing methods that integrate human visual perspective, gaze behavior, and control signals across assistive robotics, AR, and computational imaging.
• It spans diverse applications including first-person visual assistants, robotic guides, and eye-based sensor interfaces that enable real-time attention coordination and feedback.
• These systems employ advanced techniques like joint attention, structured memory, and discrete signal classification to improve interaction accuracy and AI interpretability.

In recent technical literature, “Seeing Eye” does not denote a single device class. It names a family of systems that either see with a human, see for a human, see through the human eye, or use eye behavior as a control and alignment signal. In one line of work, a “Seeing Eye” assistant is a first-person, perspective-sharing collaborator that coordinates joint attention, revisable common ground, and reflective feedback in augmented reality (Teng et al., 13 Mar 2026). In others, it is a quadruped guide robot responsive to human tugs (DeFazio et al., 2023), a home social robot for visually impaired users (Lin et al., 19 Nov 2025), a computational imaging system that reconstructs a scene from corneal reflections (Alzayer et al., 2023), a webcam or EOG-based eye interface (Park et al., 2020, Schärer et al., 2024, Zhang et al., 2 Jul 2025), or a method that uses gaze to align or interpret AI models (Lopez-Cardona et al., 2024, Kumar et al., 2023). The unifying theme is that visual perspective or oculomotor behavior becomes an operational channel rather than a passive measurement.

1. Terminological scope and conceptual distinctions

Across these works, the phrase has at least four technically distinct meanings. First, it can denote an assistive collaborator that shares the user’s egocentric visual stream and uses that shared perceptual space for coordination. Second, it can denote a robotic guide or home companion that substitutes, augments, or mediates visual access for a visually impaired person. Third, it can denote the eye itself as a sensing substrate, either optically through corneal reflections or electrophysiologically through EOG. Fourth, it can denote a source of implicit supervision for AI systems through gaze, fixation, or attention.

This multiplicity matters because several papers explicitly reject narrower interpretations. The Eye2Eye work states that a “Seeing Eye” assistant is “not just an object recognizer strapped to glasses,” but a collaborator that shares the user’s visual perspective, tracks and aligns attention in real time, builds and revises common ground, and exposes its internal state for correction (Teng et al., 13 Mar 2026). ElectraSight and VergeIO similarly show that “seeing” need not be camera-based at all: low-power, fully onboard, non-invasive eye interfaces can be built from hybrid EOG or vergence-sensitive EOG integrated into glasses (Schärer et al., 2024, Zhang et al., 2 Jul 2025). The corneal-reflection reconstruction work pushes the phrase in a literal direction: the human eye functions as a catadioptric sensor from which a 3D radiance field of the surrounding scene can be recovered (Alzayer et al., 2023).

A plausible implication is that “Seeing Eye” has become a systems term for coupling visual embodiment, attention, and inferential control. In that sense, the term spans HCI, assistive robotics, computational imaging, wearable sensing, and alignment research rather than a single application vertical.

2. Shared first-person perspective as collaborative infrastructure

The most explicit cognitive formulation appears in Eye2Eye, which identifies a communication gulf and an understanding gulf in current vision-based assistants (Teng et al., 13 Mar 2026). The communication gulf arises from channel mismatch: humans act through gaze, hand movement, posture, and deixis, while assistants typically require linear speech. The understanding gulf arises because an egocentric camera alone does not capture hesitation, gaze dwell, or evolving common ground. Eye2Eye addresses these with three components: joint attention coordination, revisable memory, and reflective situated feedback.

Joint attention is implemented through an always-on, event-driven perception pipeline using Apple Vision Pro gaze, first-person RGB frames, hand-object interaction, and STT. Heavy VLM reasoning is triggered only when hand keypoints overlap an object bounding box by at least $0.85$ or when gaze remains on the same object for at least $6$ seconds. GPT-4o is invoked for contextualized analysis of the current situation, while Gemini 2.5 Flash handles continuous video understanding and audio dialog; perception operates at $5$ FPS, and average end-to-end delay is kept around $4$–$5$ seconds by zero-shot inference and limiting concurrent triggers to $2$ (Teng et al., 13 Mar 2026).

The common-ground mechanism is object-centric. Memory is a set of object cards,

$\mathcal O = \{o_1, o_2, \dots\},$

where each card stores object label and description, inferred user intent and vector embeddings, AI responses and user reactions labeled success, failure, or pending, and relationships to other objects. Retrieval and update are formalized as

$$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$

$$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$

$o_t' = \mathcal{M}(s_t \mid o_t).$

This produces a symbolic-plus-vector memory resembling RAG, but with structured object units rather than a flat interaction log (Teng et al., 13 Mar 2026).

Reflective feedback maps situation type to modality. Object recognition uses short label plus short voice; error checking uses visual overlay plus detailed voice; knowledge recall uses text label plus detailed voice; action planning uses visual overlay for “where” and short voice for “what to do.” After feedback, the system monitors whether the user follows the suggestion, corrects the AI, or triggers clarification, and writes the outcome back to the relevant object card (Teng et al., 13 Mar 2026).

Empirically, the controlled study with $6$0 participants reports lower error rate per interaction turn for Eye2Eye than the w/o-JA-CG-SF baseline, with EMM $6$1 versus $6$2, $6$3, $6$4, implying about $6$5 reduction in error probability. Clarification cost falls from EMM $6$6 to $6$7, with about $6$8 fewer clarification turns. In the procedural task, interaction turns are reduced by $6$9. Collaboration measures also improve, including fluency, copresence, and performance trust, and the post-hoc ablation shows that full Eye2Eye outperforms w/o-JA, w/o-CG-SF, and raw GPT-4o on both accuracy and usefulness, indicating synergy rather than isolated component gains (Teng et al., 13 Mar 2026).

3. Assistive and guide embodiments

In assistive robotics, “Seeing Eye” often means a system that divides labor between human intent and robot-level safety. The quadruped navigation work formalizes that division through force-responsive locomotion control: the human decides the overall route through physical tugs on a rigid handle, while the robot maintains stable walking and locally avoids obstacles (DeFazio et al., 2023). The system combines a PPO-trained locomotion controller, a base-velocity estimator, a 1D-CNN force estimator over the last $5$0 time steps, AMCL localization, and a DWA local planner on a Unitree A1.

The locomotion policy is trained in Isaac Gym with $5$1 parallel simulated robots and reward terms for planar velocity tracking, yaw-rate tracking, vertical stability, low roll/pitch motion, low torque magnitude, low joint acceleration, foot air time, and smooth actions. During deployment, the policy consumes estimated external force $5$2 rather than privileged force labels. Human tugs are decoded mainly from the lateral component $5$3 using peak detection over a $5$4-second history, sampled at $5$5 Hz, and mapped to LEFT, RIGHT, or NONE at decision points (DeFazio et al., 2023).

Simulation results show a lower fall proportion and lower drift from trajectory for the learned controllers than for MPC under random pushes. The proposed “Learned Est” controller reports Proportion Fell $5$6 and Drift $5$7 m, versus $5$8 and $5$9 m for MPC. On hardware, force-estimator-based tug detection substantially outperforms raw accelerometer-based detection: for the expert participant, accuracy is $4$0 with FPR $4$1, versus $4$2 and $4$3 for accelerometer-based detection; for beginners, accuracy is $4$4 with FPR $4$5, versus $4$6 and $4$7 (DeFazio et al., 2023). The paper also notes what is absent: intelligent disobedience is future work rather than a current capability.

Eye Care You instantiates a different assistive interpretation. Built on ASUS Zenbo with Zenbo App Builder, DDE, PHP, and SQL, it targets home use through four voice-triggered functions: photo record, mood lift, greeting guest, and today highlight (Lin et al., 19 Nov 2025). The system uses the robot’s camera, microphones, speakers, mobility, and indoor map; a caregiver website provides status history, interior map, social-function descriptions, and an emotion index grounded in BSRS or “mood thermometer.”

Its “seeing eye” role is primarily domestic. Photo record lets visually impaired users capture images immediately in dangerous or emergency situations and upload them to Google cloud album for later inspection through the website. Mood lift uses BSRS-based questioning and content such as music, jokes, and talk shows. Greeting guest navigates the robot to the door and speaks “Welcome.” Today highlight reads weather forecast, horoscope, and reminders aloud (Lin et al., 19 Nov 2025). Unlike the quadruped system, this design is explicitly home-only and does not replace a guide dog for outdoor navigation. Unlike Eye2Eye, it does not report a formal user study with quantitative metrics; the paper states only that the system is “robust in the preliminary experiment using in a simulated home scenario” (Lin et al., 19 Nov 2025).

4. The eye as an optical sensing surface

The most literal interpretation of “Seeing Eye” appears in the corneal-reflection reconstruction work, which treats the human eye as a passive non-line-of-sight sensor (Alzayer et al., 2023). A stationary camera captures portrait images of a moving person; each corneal reflection is only about $4$8 pixels and contains a mixture of iris texture and scene reflection. From these reflections, the method reconstructs a 3D scene representation of what the person sees, including regions outside the camera’s direct line of sight.

The cornea is modeled as a section of an ellipsoid,

$4$9

with surface normal

$5$0

Weak perspective is used to approximate average corneal depth from the ellipse radius in the image, and the camera ray is reflected at the corneal surface to define the NeRF ray: $5$1 The scene radiance field $5$2 is optimized jointly with per-frame cornea poses and a 2D iris texture field $5$3, using reconstruction loss

$5$4

and a radial regularization prior

$5$5

The latter discourages static scene fragments from being absorbed into the iris texture field (Alzayer et al., 2023).

The end-to-end pipeline uses GroundingDINO for eye detection, ELLSeg for ellipse fitting, SAM for iris segmentation, and a standard nerfstudio-based NeRF implementation. Synthetic ablations show that texture decomposition improves both SSIM and LPIPS: in the classroom scene, SSIM rises from $5$6 to $5$7 and LPIPS falls from $5$8 to $5$9; in the kitchen scene, SSIM rises from $2$0 to $2$1 and LPIPS falls from $2$2 to $2$3. Qualitatively, pose optimization is critical: without it, reconstructions become largely unusable, and without radial regularization, parts of the scene can be baked into the iris texture (Alzayer et al., 2023).

The paper also foregrounds privacy. Because a seemingly ordinary portrait can contain recoverable environmental information in the eyes, the method implies that eye regions may leak details about rooms, bystanders, or screens. This is one of the clearest cases where the “Seeing Eye” metaphor is both technical and forensic (Alzayer et al., 2023).

5. Eye tracking and eye-gesture interfaces

A different research lineage makes the eye itself the primary input device. The EVE work addresses webcam-only, screen-based eye tracking through joint modeling of eye appearance and screen content (Park et al., 2020). EVE contains $2$4 participants, more than $2$5 frames, four $2$6p camera views, synchronized screen recordings, and Tobii Pro Spectrum ground truth. EyeNet estimates gaze direction and pupil size from $2$7 eye patches, while GazeRefineNet refines point of gaze from screen content and initial PoG heatmaps over a $2$8-second temporal window. Offset augmentation is used during training to simulate person-specific kappa-like offsets without per-user labels.

The key performance result is that the full GazeRefineNet with temporal GRU, screen content, and offset augmentation reduces error from $2$9, $\mathcal O = \{o_1, o_2, \dots\},$0 cm, and $\mathcal O = \{o_1, o_2, \dots\},$1 px to $\mathcal O = \{o_1, o_2, \dots\},$2, $\mathcal O = \{o_1, o_2, \dots\},$3 cm, and $\mathcal O = \{o_1, o_2, \dots\},$4 px, described as “up to a $\mathcal O = \{o_1, o_2, \dots\},$5 percent improvement in Point-of-Gaze estimates.” The work thereby approaches literature-reported performance associated with supervised personalization while remaining label-free at deployment (Park et al., 2020).

Wearable, non-camera eye interfaces pursue a different operating point. ElectraSight implements a fully onboard smart-glasses eye tracker using hybrid EOG with contact and contactless electrodes, QVar front-ends, an nRF5340-based VitalCore, and a GAP9 tinyML coprocessor (Schärer et al., 2024). The final prototype uses five differential channels, per-window standardization, Savitzky–Golay filtering, and a 1D-CNN quantized to $\mathcal O = \{o_1, o_2, \dots\},$6 bits. The deployed model occupies $\mathcal O = \{o_1, o_2, \dots\},$7 kB, performs inference in $\mathcal O = \{o_1, o_2, \dots\},$8, achieves $\mathcal O = \{o_1, o_2, \dots\},$9 accuracy for $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$0 classes and $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$1 for $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$2 classes, detects $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$3 of movements within $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$4 ms, and enables continuous operation for over $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$5 days on a $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$6 mAh battery. The paper is explicit that this is not full continuous gaze-angle estimation; it is discrete eye-movement classification suitable for hands-free control, health monitoring, and privacy-preserving smart-glasses interaction (Schärer et al., 2024).

VergeIO extends glasses-based EOG from cardinal eye gestures to depth-aware interaction via vergence (Zhang et al., 2 Jul 2025). Its optimized layout uses two positive electrodes at the temples, one reference electrode on the nose bridge, and one ground on the mastoid, producing two horizontal EOG channels sensitive to convergence and divergence across depth layers at $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$7, $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$8, and $$c_{mem} = \mathcal{M} \left( \underset{o_i \in \mathcal{O},\; \text{similarity}(o_t, o_i) > \lambda}{\arg\max} \; \text{similarity}(o_t, o_i) \right),$$9 cm. Compared with a JINS-style layout, the design improves mean SNR for vergence from $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$0 dB to $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$1 dB. In a study with $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$2 users and $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$3 gesture instances, six-gesture depth recognition reaches $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$4 within session, $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$5 cross-session, and $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$6 cross-user without calibration; a four-gesture subset reaches $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$7 in within-session, cross-session, and cross-user evaluations. Motion-artifact detection reports $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$8 accuracy with $$s_t = \mathcal M(i_t \mid c_{mem} + c_{state}, P_s),$$9 false negatives for vergence, and the eyebrow-raise preamble reduces false positives to $o_t' = \mathcal{M}(s_t \mid o_t).$0 across static, walking, chewing, and talking conditions (Zhang et al., 2 Jul 2025).

Taken together, these systems show that a “Seeing Eye” interface need not output a high-resolution gaze ray. It may instead output robust discrete events, depth transitions, or attention-refined PoG estimates, depending on the latency, power, and privacy constraints of the target device.

6. Gaze as alignment signal and interpretability probe

In language-model alignment, gaze appears as an auxiliary reward signal rather than as an input interface. GazeReward augments reward modeling for LLMs with synthetic eye-tracking features predicted from text by pretrained ET generators (Lopez-Cardona et al., 2024). Per-token gaze features $o_t' = \mathcal{M}(s_t \mid o_t).$1 are projected into the reward model embedding space and integrated with token embeddings either by concatenation,

$o_t' = \mathcal{M}(s_t \mid o_t).$2

or by elementwise addition,

$o_t' = \mathcal{M}(s_t \mid o_t).$3

The reward model then uses the standard Bradley–Terry pairwise loss

$o_t' = \mathcal{M}(s_t \mid o_t).$4

On OASST1, the best gaze-augmented models improve pairwise accuracy from $o_t' = \mathcal{M}(s_t \mid o_t).$5 to $o_t' = \mathcal{M}(s_t \mid o_t).$6 for Llama-3-8B-Instruct, from $o_t' = \mathcal{M}(s_t \mid o_t).$7 to $o_t' = \mathcal{M}(s_t \mid o_t).$8 for Llama-3-8B, and from $o_t' = \mathcal{M}(s_t \mid o_t).$9 to $6$00 for Mistral-7B. On HelpSteer2, the best results rise from $6$01 to $6$02, from $6$03 to $6$04, and from $6$05 to $6$06, respectively. On RewardBench, a Mistral-7B reward model trained only on OASST1 improves from $6$07 to $6$08 when augmented with gaze features (Lopez-Cardona et al., 2024). The paper is careful to state that gaze does not replace explicit preference labels; it enriches the reward model with a virtual estimate of how humans would read a response.

A complementary use of gaze appears in the video memorability work, where model attention is compared directly with human fixation density maps during a recognition-memory task (Kumar et al., 2023). The architecture uses frozen CLIP ResNet-50 conv5 features over uniformly sampled frames, spatio-temporal tokenization, a Transformer encoder, and a scalar memorability regressor. Last-layer [CLS]-to-token attention is reshaped into frame-level attention maps and compared against eye-tracking data from $6$09 participants using AUC-Judd, NSS, CC, KLD, and AUC-Percentile.

The resulting attention maps align strongly with human gaze despite the model never being trained on fixation data. On the eye-tracked subsets, AUC-Judd is $6$10 for both Memento10K and VideoMem; NSS is $6$11 and $6$12, respectively. The model also exhibits a strong early-frame bias, assigning greater importance to initial frames even when frame order is reversed, and panoptic analysis shows that both humans and model allocate a greater share of attention to things and less to stuff relative to their occurrence probability (Kumar et al., 2023). Here, the “seeing eye” is not a physical device but an interpretability question: whether model attention sees eye-to-eye with human attention.

7. Recurring limitations, controversies, and implications

Several constraints recur across these systems. Latency remains nontrivial in collaborative AR: Eye2Eye’s $6$13–$6$14-second inference delay can create micro-interruptions, particularly in subjective tasks (Teng et al., 13 Mar 2026). Quadruped guide navigation has been demonstrated mainly in indoor hallways with blindfolded researchers rather than visually impaired participants, and intelligent disobedience remains unimplemented (DeFazio et al., 2023). Eye Care You lacks a formal evaluation protocol and currently relies on manual classification of dangerous versus normal photo events (Lin et al., 19 Nov 2025). Corneal-reflection reconstruction depends on controlled portrait capture, good lighting, and accurate cornea pose; its privacy implications are significant (Alzayer et al., 2023). ElectraSight and VergeIO trade continuous gaze precision for lower power and greater unobtrusiveness, and both are evaluated in constrained settings relative to the variability of long-term daily use (Schärer et al., 2024, Zhang et al., 2 Jul 2025). GazeReward depends on synthetic ET models trained on English reading corpora rather than direct ET from LLM-response reading, and the memorability-attention study cautions that Transformer attention is not itself a definitive causal explanation (Lopez-Cardona et al., 2024, Kumar et al., 2023).

These limits clarify a common misconception: a “Seeing Eye” system is not simply any system with a camera near the face. In the literature surveyed here, successful systems typically combine at least one of the following: shared first-person context, explicit exposure of internal state, revisable memory, low-latency event triggering, physically grounded control channels, or eye-derived signals that are interpretable by design. This suggests that the concept is best understood not as a single modality, but as a systems architecture in which vision or oculomotor behavior participates directly in coordination, assistance, sensing, or alignment.