RoboEye: Active Robotic Vision Systems

• RoboEye is a design idiom that integrates perception, geometrically informed viewpoint selection, and action across domains from warehouse retrieval to ophthalmic interventions.
• It employs selective perception techniques—including gating, dwell filtering, and transformer-based re-ranking—to boost system accuracy and efficiency.
• Multiple implementations, from RGB-only object retrieval to gaze-mediated manipulation and biomechanical robotic eyes, illustrate its versatility and domain-specific challenges.

Searching arXiv for the cited RoboEye-related papers to ground the article in current records. RoboEye is not a single standardized system in contemporary robotics literature. The term is used most narrowly for a two-stage RGB-only warehouse object-identification framework, but it also appears as a convenient label for gaze-mediated assistive manipulation systems, eye-inspired active-vision hardware, and ophthalmic robot-vision pipelines. Across these usages, the common thread is the coupling of perception, viewpoint selection, and action through explicit geometric, kinematic, or task-level reasoning rather than passive imaging alone (Zhang et al., 18 Sep 2025, Wang et al., 2022, Osooli et al., 2023).

1. Nomenclature and scope

The literature assigns “RoboEye” to several technically distinct problem classes. In one usage, it denotes a retrieval framework for large-scale warehouse object identification. In others, it refers to systems that infer manipulation intent from human gaze, bioinspired robotic eye mechanisms, or clinical robot-vision subsystems for ophthalmic intervention. This plurality is important: many apparent inconsistencies across “RoboEye” papers arise because the name does not denote a single lineage or platform.

Usage of RoboEye	Representative work	Core function
Warehouse identification	"RoboEye: Enhancing 2D Robotic Object Identification with Selective 3D Geometric Keypoint Matching" (Zhang et al., 18 Sep 2025)	RGB-only two-stage object retrieval with selective 3D re-ranking
Gaze-guided assistive manipulation	"What You See is What You Grasp: User-Friendly Grasping Guided by Near-eye-tracking" (Wang et al., 2022)	Near-eye-tracking plus grasp prediction for pick-and-place
Bioinspired robotic eye hardware	"Design and Evaluation of a Bioinspired Tendon-Driven 3D-Printed Robotic Eye with Active Vision Capabilities" (Osooli et al., 2023)	Tendon-driven binocular eye mechanism with active vision
Active observer head	"DIJIT: A Robotic Head for an Active Observer" (Tabrizi et al., 8 Dec 2025)	Binocular head with vergence, version, and cyclotorsion
Ophthalmic robot vision	"Towards Robotic Eye Surgery: Marker-free, Online Hand-eye Calibration using Optical Coherence Tomography Images" (Zhou et al., 2018)	OCT-guided calibration between surgical robot and imaging

A common misconception is that RoboEye necessarily denotes a literal robotic eyeball. The warehouse RoboEye is instead a retrieval stack operating on RGB images and selective geometric verification; conversely, several embodied “robot eye” systems are not titled RoboEye at all but are presented as RoboEye-like platforms in active perception, gaze interaction, or robot-assisted eye surgery (Zhang et al., 18 Sep 2025, Kerr et al., 12 Jun 2025).

2. Gaze-mediated manipulation and assistive interaction

In assistive HRI, RoboEye-like systems use gaze as the primary or sole command channel for grasp selection and manipulation. The most technically complete formulation in this group is the near-eye-tracking grasping system of "What You See is What You Grasp," which combines a head-mounted near-eye-tracking device with a transformer-based grasp detector. The hardware uses four eye cameras with near-infrared illumination, a scene camera, a Franka Emika Panda 7-DoF arm, and a gripper-mounted Intel RealSense D435i. The grasp model outputs per-pixel grasp quality $Q(x,y)$, width $W(x,y)$, and angle $\theta(x,y)$, with angle recovered as $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$. End-to-end gaze-to-grasp decision latency is reported as $50.21$ ms, $59.21$ ms, and $91.21$ ms for the Tiny, Small, and Base transformer variants, and Cornell accuracy reaches $97.72\%$ for RGB and $98.86\%$ for RGB-D (Wang et al., 2022).

GazeGrasp implements a lower-cost variant of the same interaction logic for collaborative manipulation. It mounts an ESP32-CAM on glasses, uses MediaPipe for gaze estimation, YOLOv8 for object localization from a UR10 end-effector camera, and executes pick-and-place with a Robotiq 2F-85 gripper. Its calibration maps iris coordinates $(u,v)$ to screen coordinates through coupled third-degree polynomial regression, smooths the gaze with a discrete Kalman filter, and introduces deterministic “magnetic snapping” to the center of any bounding box containing the gaze cursor. The snapping rule is

$W(x,y)$0

In 13 participants, this reduced mean alignment time from $W(x,y)$1 s to $W(x,y)$2 s, a $W(x,y)$3 improvement, with repeated-measures ANOVA $W(x,y)$4, $W(x,y)$5 (Tokmurziyev et al., 13 Jan 2025).

RaycastGrasp shifts the interface from screen-based gaze to egocentric mixed reality. It uses an HTC Focus Vision headset, gaze raycasting into passthrough space, a 2-second dwell trigger, and YOLOv8 label matching between headset and robot views, thereby avoiding full metric head–eye–robot calibration in the baseline workflow. The reported single-pass intention and object recognition accuracy is greater than $W(x,y)$6, and gaze tracking spatial precision is $W(x,y)$7 m when fixation remains stable for at least 2 seconds (Lin et al., 25 Oct 2025).

A more explicitly cognitive-level formulation appears in the non-invasive eye-driven reaching-and-grasping interface for human augmentation. That system combines SMI ETG 2W glasses, an Intel RealSense D435 mounted on the glasses, OptiTrack head tracking, a UR10, and a BioServo Carbonhand glove. Intention is inferred when gaze fixations enter the right-most third of a detected object’s bounding box; a hand-derived action grammar implemented as a finite-state machine then expands that intent into reach, grasp, place, or pour actions. The interface is worn, calibrated, and ready in about 5 minutes, followed by about 5 minutes of familiarization, and in 5 healthy participants it achieved an average first-attempt success rate of $W(x,y)$8 across 6 tasks (Shafti et al., 2021).

Taken together, these systems define RoboEye in assistive robotics as a pipeline from gaze estimation to task execution, with selection stabilized by calibration, filtering, or dwell logic, and with downstream autonomy shouldering low-level motion generation. The principal variation lies in where gaze is measured—near-eye cameras, mixed-reality headsets, or wearable eye trackers—and in how much of the reach–grasp sequence is delegated to autonomous planning rather than direct human input.

3. RoboEye as a warehouse identification framework

The exact-title RoboEye system is a two-stage object-identification framework for large-scale e-commerce warehouses. Its motivation is the failure of 2D-only appearance features under long-tail category growth, intra-class variability, clutter, packaging changes, occlusion, and large viewpoint change. On ARMBench, even scaling the 2D backbone to BEiT-3 Large leaves Recall@1 at $W(x,y)$9 in the global gallery, motivating selective geometric verification rather than unconditional feature enlargement (Zhang et al., 18 Sep 2025).

Stage 1 uses BEiT-3 as a large vision model to encode single-view or fused multi-view queries and references, trained with a contrastive objective following RoboLLM. Similarity is computed by dot product or cosine to generate a top-$\theta(x,y)$0 candidate list, with default $\theta(x,y)$1. A lightweight 3D-feature-awareness gate then predicts whether geometric re-ranking is likely to help. In its practical instantiation,

$\theta(x,y)$2

and Stage 2 is invoked only if $\theta(x,y)$3 (Zhang et al., 18 Sep 2025).

The gate is trained by MRR-driven supervision, called M3AT. Each query is labeled positive if 3D re-ranking improves mean reciprocal rank over 2D-only retrieval, and the gate is optimized with weighted cross-entropy using approximately $\theta(x,y)$4 positive-to-negative class weights. This makes the gate explicitly predictive of utility rather than merely of geometric salience (Zhang et al., 18 Sep 2025).

Stage 2 is the robot 3D retrieval transformer. It derives geometry-aware dense features from RGB using VGGT’s aggregator, avoiding explicit depth or point clouds. SIFT detects $\theta(x,y)$5 keypoints on the query image; each keypoint descriptor is sampled from the dense 3D-aware feature map as

$\theta(x,y)$6

A transformer matcher then estimates matched coordinates and confidences $\theta(x,y)$7 on each reference, and the candidate score is

$\theta(x,y)$8

This replaces global cosine similarity with correspondence confidence aggregation (Zhang et al., 18 Sep 2025).

The framework also inserts Houlsby-style knowledge adapters into matcher transformer blocks while freezing the VGGT aggregator. Each adapter uses bottleneck width $\theta(x,y)$9:

$$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$0

This is a domain-adaptation mechanism for warehouse imagery rather than a full retraining of the 3D extractor (Zhang et al., 18 Sep 2025).

Empirically, the strongest gains appear in the multi-view global gallery. RoboEye improves Recall@1 from $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$1 to $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$2, a gain of $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$3 over RoboLLM. In the single-view global gallery, Recall@1 rises from $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$4 to $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$5. Ablations show that naive 3D cosine re-ranking is ineffective: 2D-FE only yields $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$6, 3D-FE only with cosine yields $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$7, naive 2D$$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$83D cosine re-ranking yields $$\theta(x,y)=0.5\arctan2(\sin(2\theta),\cos(2\theta))$$9, and the full system with matcher, adapters, and gating yields $50.21$0. Runtime is likewise shaped by selectivity: with candidate pool size $50.21$1 on a single NVIDIA 5090 GPU with 32 GB, 2D-only inference is $50.21$2 s, unconditional 3D re-ranking is $50.21$3 s, and gated 3D verification is $50.21$4 s while raising Recall@1 to $50.21$5 (Zhang et al., 18 Sep 2025).

This usage of RoboEye is therefore not anthropomorphic and not gaze-based. It is best understood as a selective visual reasoning stack in which a large 2D retriever, a learned utility gate, and a correspondence-based RGB-only 3D verifier are composed to preserve near-2D latency while improving retrieval robustness.

4. Embodied robotic eyes and head–eye mechanisms

Another strand of RoboEye research concerns literal robotic eye hardware. The tendon-driven 3D-printed robotic eye is a binocular system designed around two rotational degrees of freedom per eye, implemented by agonist–antagonist tendon pairs actuated by four Dynamixel XL-320 servos in total. Each ABS eyeball comprises a sclera shell and cornea, with the cornea mated to the sclera at a $50.21$6-degree angle relative to the optical axis. The eyeball diameter is $50.21$7 mm to accommodate a Microsoft LifeCam HD-3000, and the inter-ocular spacing is $50.21$8 cm. The design emphasizes realistic appearance and efficient actuation rather than torsional completeness; roll is absent in the current build, though the authors note it could be added by one extra servo per eye. Control is vision-based PID on face-center error, implemented from MATLAB through an OpenCM 9.04-C controller (Osooli et al., 2023).

A more explicitly anthropomorphic head–eye configuration is built around a 3-DOF Agile Eye spherical parallel robot with a 2-DOF eye mechanism mounted on the end-effector. The head workspace is conservatively limited to approximately $50.21$9 in roll, pitch, and yaw to avoid singularities, and inverse kinematics are given in closed form by

$59.21$0

The system uses Mediapipe Face Mesh or a learned linear regression model over landmarks to estimate human head orientation, and smooths motor commands with a $59.21$1-$59.21$2-$59.21$3 trajectory profile $59.21$4 (Radmehr et al., 2021).

DIJIT extends this embodied approach to a substantially richer binocular head. It has 9 mechanical degrees of freedom—3 in the neck and 3 in each eye—and 4 optical degrees of freedom via focus, exposure time, sensor gain, and white balance. The baseline is $59.21$5 mm, the cameras are iDS U3-3881LE-AF global-shutter units at $59.21$6 and $59.21$7 fps, and liquid lenses provide active accommodation. Eye pan and tilt ranges are approximately $59.21$8 and $59.21$9, peak saccade speed reaches $91.21$0 deg/s, and a calibration-based homography method maps desired fixation locations directly to motor commands. In 191 monocular saccades, mean angular error is $91.21$1, with $91.21$2 within $91.21$3 and $91.21$4 within $91.21$5; one corrective saccade reduces the mean to $91.21$6 (Tabrizi et al., 8 Dec 2025).

These systems define RoboEye in a literal, biomechanical sense: a robotic sensor head whose morphology and kinematics are themselves experimental variables. Their emphasis is on DOF allocation, tendon or parallel-robot actuation, servo bandwidth, vergence or version, and calibration of the mapping between desired orientation and motor state.

5. Active gaze policies, observer heads, and socially legible eyes

Recent work also treats RoboEye as an active observer whose gaze policy is learned or whose attention is made externally legible. EyeRobot is exemplary in this respect. It uses a 2-DoF pan/tilt mechanical eyeball with direct-drive brushless motors, a global-shutter RGB camera at $91.21$7 fps and $91.21$8 resolution, a $91.21$9 fisheye lens, and a BC-RL loop in which the hand policy is behavior-cloned from demonstrations while the eye policy is optimized by PPO to improve hand prediction accuracy. The eye reward is the negative Fréchet distance between predicted and demonstrated end-effector trajectories, and eye actions are smoothed by an exponential moving average with coefficient $97.72\%$0. On panoramic manipulation tasks spanning about $97.72\%$1–$97.72\%$2 around the arm, EyeRobot achieved $97.72\%$3 success on both slow and fast E-Stop servoing conditions and outperformed non-foveated ablations, reducing E-Stop error from $97.72\%$4 cm to $97.72\%$5 cm and settling time from $97.72\%$6 s to $97.72\%$7 s (Kerr et al., 12 Jun 2025).

EyeVLA addresses the same problem from a vision-language-action perspective. It models a robotic eyeball with pan, tilt, and optical zoom as a token-generating agent built on Qwen2.5-VL-7B-Instruct. Action outputs are discretized into 43 tokens using a hierarchical decimal encoding with basis $97.72\%$8, yielding average action-token length $97.72\%$9 on 500 real samples and covering more than $98.86\%$0 of actions with two digit levels. After supervised fine-tuning on pseudo-labels and reinforcement learning with Group Relative Policy Optimization, the best RL3 model reaches $98.86\%$1 MAE of $98.86\%$2, $98.86\%$3 MAE of $98.86\%$4, Zoom MAE of $98.86\%$5, and a Completion Rate of $98.86\%$6 over 50 real-world lab scenes, compared with $98.86\%$7 for the ML baseline (Yang et al., 19 Nov 2025).

Mirror Eyes approaches RoboEye from explainable HRI rather than task optimization. It uses a moving robot head with a single front display rendering two eyes, and overlays a horizontally flipped crop of the attended camera region onto each pupil. In a within-subject study with 33 participants performing monitored pick-and-place tasks, the Mirror Eyes condition raised subjective information processing awareness from $98.86\%$8 to $98.86\%$9, improved early error interruption times from $(u,v)$0 s to $(u,v)$1 s for Step 1 errors and from $(u,v)$2 s to $(u,v)$3 s for Step 2 errors, and increased overall UEQ-S from $(u,v)$4 to $(u,v)$5 (Krüger et al., 23 Jun 2025).

A more minimal social-gaze control framework appears in the robotic eye-contact literature. There, attention capture is treated as a sequence of escalating actions—head turning, head shaking, and the spoken reference term “excuse me”—conditioned on four viewing situations: CFOV, NPFOV, FPFOV, and OFOV. The proposed method, which adds blink-based “ensuring attention capture” after success, achieved an overall success ratio of $(u,v)$6 across 48 participants, compared with $(u,v)$7 for head turning alone (Hoque, 2020).

This branch of RoboEye research replaces static camera placement with task-conditioned viewing behavior or interpretable gaze display. Its central claim is not merely that robots can look, but that they should decide where to look next, at what scale, and with what externally intelligible cue structure.

6. Ophthalmic RoboEye systems

In ophthalmic robotics, RoboEye-like systems are defined by micrometer-scale calibration, constrained instrument kinematics, and depth-aware imaging rather than by anthropomorphic gaze. Marker-free online hand–eye calibration with microscope-integrated OCT is one foundational example. That framework estimates OCT distortion parameters, segments the needle tip from 3D-OCT volumes, and solves

$(u,v)$8

using SVD, Horn quaternions, or a quaternion solver with Kalman-filtered tip positions. In ex vivo pig-eye experiments, mean calibration error was about $(u,v)$9m for the marker-free method versus about $W(x,y)$00m for the marker-based reference, with average processing time $W(x,y)$01 s per needle-tip volume (Zhou et al., 2018).

Autonomous orbital manipulation extends this clinical meaning of RoboEye from calibration to eye reorientation. Using Vector-Field Inequalities, the controller constrains the squared distance between two remote-center-of-motion points to remain within a band around its initial value,

$W(x,y)$02

with $W(x,y)$03 mm and $W(x,y)$04. The result is globe rotation that improves manipulability and visible fundus area while preserving tele-operated control of the dominant tool (Koyama et al., 2023).

EyeLS provides a different perception primitive: shadow-guided depth inference for initial intraocular approach. With a static light probe, the system uses coincident visual projection and shadow alignment to establish a unique tip state for target contact. In simulation over 1926 trials, 1922 were successful, one failed because the shadow overlapped with the needle, and three became stuck. Mean depth error was $W(x,y)$05 mm for floating targets and $W(x,y)$06 mm for retinal targets, while tip-to-retina distance remained positive, preventing damage (Yang et al., 2023).

Force-controlled vitreoretinal manipulation introduces yet another meaning of RoboEye, centered on scleral safety. In the SHER 2.0/2.1 bimanual adaptive teleoperation framework, adaptive force control limits tool–sclera force using FBG sensors and a safe threshold of $W(x,y)$07 mN. In the sitting condition, bimanual adaptive teleoperation reduced dominant-hand mean sclera force from $W(x,y)$08 mN in cooperative mode to $W(x,y)$09 mN and reduced time above $W(x,y)$10 mN from $W(x,y)$11 to $W(x,y)$12 (Esfandiari et al., 2024).

The most integrated autonomous clinical pipeline is the microscope+iOCT framework for subretinal injection. It combines a microscope RGB stream at 30 Hz, an iOCT B-scan stream at 11 Hz, three CNNs running above 40 Hz on an RTX 3090, adaptive Jacobian visual servo in RGB, and MPC-style safety-constrained motion generation. Galvo steering aligns the B-scan with the needle axis via a linear calibration

$W(x,y)$13

and the system stops $W(x,y)$14m above the ILM before insertion. Across 30 ex vivo pig-eye trials, mean insertion accuracy was $W(x,y)$15m, mean navigation error to the retinal surface goal was $W(x,y)$16m, mean RCM error was $W(x,y)$17m, and total autonomous subretinal injection duration was $W(x,y)$18 s (Kim et al., 2023).

In this clinical literature, RoboEye therefore denotes an imaging–robotics coupling in which OCT, microscopy, calibration, and constrained control produce a continuously updated spatial relationship between tool, tissue, and robot.

7. Shared design patterns, misconceptions, and recurring limitations

Taken together, these papers suggest that RoboEye is better understood as a design idiom than as a fixed architecture. Across domains, three recurring operators appear. The first is selective perception: gating in warehouse retrieval, dwell or fixation filtering in assistive interfaces, action-tokenized gaze in active observers, and scan-line steering in ophthalmic surgery. The second is explicit geometric mediation: hand–eye transforms, gaze-ray or ray–plane intersection, RCM constraints, or coordinate mappings from image pixels to robot poses. The third is safety or relevance gating: repeated-measures selection logic in gaze interfaces, 3D-feature-awareness in retrieval, VFI constraints in orbital manipulation, or MPC and force thresholds in eye surgery (Zhang et al., 18 Sep 2025, Tokmurziyev et al., 13 Jan 2025, Kim et al., 2023).

Several misconceptions recur. One is that RoboEye always requires depth hardware. The warehouse RoboEye explicitly operates using only RGB images, deriving geometry-aware features through a transformer rather than from explicit 3D sensors (Zhang et al., 18 Sep 2025). Another is that RoboEye always implies direct human gaze input. EyeRobot and EyeVLA instead learn robotic gaze policies for autonomous active perception, and Mirror Eyes uses eye imagery primarily to externalize robot attention rather than to receive user commands (Kerr et al., 12 Jun 2025, Yang et al., 19 Nov 2025, Krüger et al., 23 Jun 2025). Conversely, in assistive manipulation, gaze often is the interface, but the literature still differs on whether selection is fixation-based, dwell-based, or grammar-constrained (Wang et al., 2022, Shafti et al., 2021).

The reported limitations are likewise domain-specific but structurally similar. Assistive gaze-control systems often omit detailed latency, calibration-residual, collision-avoidance, or controller-stack reporting, and frequently rely on 2D target centers or dwell-only gestures (Tokmurziyev et al., 13 Jan 2025). Warehouse identification depends critically on Stage 1 candidate coverage and can still fail on textureless or heavily occluded items, especially when SIFT keypoints are sparse (Zhang et al., 18 Sep 2025). Active-observer systems remain constrained by camera motion blur, limited zoom-out behavior, no-parallax replay environments, or incomplete binocular coordination (Kerr et al., 12 Jun 2025, Yang et al., 19 Nov 2025). Ophthalmic systems typically assume rigid microscope–robot geometry, stable eye conditions, or sufficiently visible tools, and several are validated only in ex vivo or phantom settings (Zhou et al., 2018, Kim et al., 2023).

This heterogeneity means that an encyclopedia treatment of RoboEye cannot reduce the term to a single robot, algorithm, or application. The literature instead uses it to name systems that make vision actionable by coupling viewpoint, geometry, and task execution—whether the problem is identifying a SKU in a warehouse, selecting a cup by gaze, rendering an eye-like mechanism, exposing robot intent to a collaborator, or aligning a needle to the retina at micrometer scale.