- The paper introduces GazeLNN, an auto-regressive scanpath predictor that leverages liquid neural networks for real-time fixation-guided perception with dramatic computational efficiency.
- The architecture combines MobileNetV3 with the CfC variant, achieving significant reductions in FLOPs and improved scanpath metrics over recurrent and Bayesian models.
- The integration of GazeLNN with RL-based active perception enhances scene coverage and salient feature capture, validated through both simulation and real-world experiments.
Efficient Scanpath Prediction and Its Integration into Active Perception for Autonomous Navigation
Background and Motivation
Human visual attention is fundamentally structured by temporally ordered scanpaths, which allow selective processing of salient features within dynamic scenes through iterative saccades and fixations. Robotic perception systems have traditionally relied on dense processing or static saliency mapping, failing to capture the sequential and adaptive dynamics inherent in human gaze. Modern deep scanpath prediction architectures have improved accuracy but at the expense of significant computational burden, precluding deployment on resource-limited autonomous platforms.
This work introduces GazeLNN, an auto-regressive scanpath predictor leveraging the expressivity and computational efficiency of Liquid Neural Networks (LNN), specifically the Closed-Form Continuous-time (CfC) variant, integrated with a MobileNetV3 feature backbone. The architecture is designed for real-time, onboard inference suitable for embodied autonomous agents. GazeLNN is demonstrated to achieve superior scanpath prediction accuracy with a drastic reduction in computational cost and runtime. Furthermore, the scanpath output from GazeLNN is integrated into an RL-based active camera policy to facilitate fixation-driven perception, leading to quantifiable improvements in spatial scene coverage and salient feature acquisition.
Scanpath Prediction Architecture
The GazeLNN model predicts fixation scanpaths in an auto-regressive fashion by sequentially updating the fixation heatmap conditioned on the current visual stimulus and prior fixation history. The visual features are extracted via MobileNetV3, selected for its high parameter-to-performance efficiency, and concatenated with Gaussian-encoded fixation heatmaps augmented by CoordConv layers for enhanced spatial structure.
The recurrent module is an LNN (CfC variant), initialized with zero hidden state. At each step, the combined feature representation is processed by a shared fully connected backbone followed by parallel layers that control the time-dependent gating mechanism, facilitating adaptive integration of input features based on elapsed fixation duration, as per the continuous-time update rule for CfC. The predicted heatmap is projected, upsampled, normalized, and its maxima used to extract the fixation location for the next step.
Training is performed using the KL-DTW loss function, which aligns spatial and temporal components of the predicted scanpaths with ground truth, on the OSIE dataset. Model evaluation employs the MIT Low Resolution dataset for cross-dataset generalization.
Comparative Evaluation and Computational Efficiency
GazeLNN outperforms state-of-the-art recurrent and Bayesian ConvLSTM baselines across all scanpath metrics:
- ScanMatch: 0.47, a 34% improvement versus competing models.
- Levenshtein Distance: 2.18% reduction.
- Hausdorff Distance: 5.09% reduction.
- Fréchet Distance: 7.92% reduction.
- FastDTW: 11.85% reduction.
- Time Delay Embedding: 37.81% reduction.
Inference latency averages 6.84 ms per frame for an 8-fixation scanpath (NVIDIA RTX 3500 Ada), while computational complexity is restricted to 0.61 GFLOPs and 15.24M parameters, representing a 99.4% reduction in FLOPs and 91.8% reduction in parameters compared to the prior second-best tSPM-Net model. This ultra-efficient profile enables real-time onboard execution, critical for practical robotic deployment.
Ablation studies demonstrate that backbone architecture choice minimally affects scanpath metric performance, establishing MobileNetV3 as optimal for embedded systems. Similarly, the CfC variant consistently outperforms ConvLSTM and Bayesian variants in both accuracy and efficiency.
Integration with Active Perception via Reinforcement Learning
To operationalize fixation-guided perception during navigation, GazeLNN is embedded within a multitask RL policy trained in the Aerial Gym simulator. The RL policy receives robot state, depth image, fixation heatmap, local occupancy grid, and camera orientation, outputting vehicle velocities and camera pose commands.
A custom reward function incentivizes goal-directed progress, motion smoothness, collision avoidance, and attention-guided camera reorientation. During simulation, fixation heatmaps are generated by sampling salient mesh feature points, perturbed to introduce spatial noise and convolved to match GazeLNN's predictive output.
The policy is trained with APPO, with visual inputs downsampled to minimize computational overhead. The fixation heatmap encoder is based on a compact 2D ResNet architecture; depth encoding utilizes a frozen Deep Collision Encoder for spatial feature compression.
Real-world Experimental Validation
Onboard deployment is conducted on a custom quadrotor with a pan-tilt RGB-D camera, actuated via servo motors and controlled with precise feedback. The system runs GazeLNN in real-time on NVIDIA Jetson Orin NX, directing the camera to salient regions as predicted by scanpaths.
The fixation-guided active camera policy is directly compared to a static forward-facing baseline in an unstructured indoor environment:
- Scene Awareness: Active camera accumulates 55,524 voxels; static camera only 37,067.
- Task-Relevant Observation: Active policy observes 6,770 salient voxels, an eightfold increase over static baseline (873).
- Attention Retention: Maximum voxel hit count increases from 537 to 756.
Active perception delivers broader spatial coverage, improved mapping of peripheral and contextually relevant features, and more robust data for state estimation and obstacle avoidance. Qualitative scanpath visualizations confirm the alignment with human-like attention sequences.
Theoretical and Practical Implications
GazeLNN operationalizes a parsimonious yet expressive scanpath prediction architecture, demonstrating that continuous-time LNNs can efficiently model stochastic temporal dynamics of human gaze at a fraction of prior computational cost. The tight coupling of fixation-guided attention with RL-based active perception establishes a scalable framework for embodied agents to leverage human visual strategies for adaptive information acquisition.
These results challenge the necessity of over-parameterized networks (e.g., transformers) for scanpath prediction, with implications for the design of lightweight, explainable attention models for mobile robotics and real-time active vision systems. The RL integration further exemplifies a practical avenue for human-informed policy shaping, potentially advancing autonomy in UAVs, inspection robots, and other real-world agents.
Conclusion
GazeLNN sets a new benchmark in scanpath prediction for low-cost, high-speed deployment. Its integration with reinforcement learning enables autonomous agents to mimic human-like attention allocation, enhancing spatial mapping and salient feature coverage during navigation. Future research will explore deeper correlations between scanpath dynamics, camera motion, and RL policy convergence, as well as extensions to more complex active perception contexts and heterogeneous sensory platforms.
(2606.20491)