Late-Fusion RGB-Event Navigation Policy

• The paper presents a multimodal policy that independently encodes RGB and event data then fuses them via a Transformer to reduce action prediction error.
• It utilizes dual MobileNetV3 backbones—one frozen for RGB and one trainable for events—to exploit complementary sensing under varied lighting and motion conditions.
• Experiments on indoor person-following demonstrate that ENP-Fusion achieves up to 47.7% lower MAE than RGB-only models, particularly in low-light and complex scenarios.

Searching arXiv for the cited paper and closely related context papers. Late-Fusion RGB-Event Navigation Policy denotes a multimodal behavioral cloning policy for indoor mobile robot navigation that independently encodes RGB imagery and event-camera observations and fuses the resulting modality tokens with a Transformer encoder before regressing continuous differential-drive actions. In eNavi, this policy is instantiated as ENP-Fusion for indoor person-following with a TurtleBot 2 under both normal-light and low-light conditions, using synchronized RGB frames, raw event streams, and odometry-derived control labels to learn the action pair $a=(v,\omega)$, where $v$ is forward linear velocity and $\omega$ is yaw rate (Ramesh et al., 15 Mar 2026). The formulation is motivated by the complementary sensing properties of dense RGB appearance and asynchronous event measurements, particularly when conventional RGB sensing degrades under fast motion or dim illumination.

1. Definition and operating setting

A late-fusion RGB-Event navigation policy is a control architecture in which each sensing modality is encoded independently to a compact feature representation before cross-modal interaction occurs. In eNavi, late fusion is explicitly contrasted with early fusion, which concatenates inputs, and mid fusion, which merges intermediate feature maps; the late-fusion design allows each backbone to specialize to heterogeneous statistics and then uses attention to reweight modality contributions per sample (Ramesh et al., 15 Mar 2026).

The operating task is indoor person-following with a differential-drive TurtleBot 2 using a Kobuki base. The robot must follow a walking person across cluttered indoor spaces under both normal and low-light illumination. Sensing is provided by a co-axially aligned RGB camera and an event camera mounted via a beam splitter to minimize parallax. The event camera is a Prophesee EVK4 with an IMX636 sensor at $1280\times720$, while the RGB camera is a Teledyne Blackfly S at $1280\times1024$ and up to 170 fps; both use 25 mm f/1.4 Kowa lenses (Ramesh et al., 15 Mar 2026).

The control space is continuous differential-drive motion:

$a=(v,\omega),$

with $v$ defined in ROS base_link as forward speed in m/s along the $x$-axis and $\omega$ as yaw rate in rad/s about the $z$-axis. The paper gives the standard differential-drive kinematic context,

$v$0

but emphasizes that eNavi uses odometry twist as ground truth rather than computing labels from wheel encoders through these formulas (Ramesh et al., 15 Mar 2026).

The underlying rationale for using event data is stated directly: event cameras provide high dynamic range and microsecond temporal resolution, preserving motion cues during fast motions and in low light, when RGB frames blur or lose contrast. This is presented as crucial for maintaining closed-loop control robustness when the target moves quickly or lighting is dim (Ramesh et al., 15 Mar 2026).

2. Dataset, synchronization, and action labeling

The eNavi dataset is a real-world indoor person-following corpus collected across 3 indoor maps, 3 person-following paths (P1, P2, P3), 2 human subjects with varied gait and clothing, and both normal-light and low-light conditions. Its reported scale is approximately 2 hours of driving and more than 175 episodes. Control during collection is teleoperated person-following via joystick using FPV video, and the human target faces away from the robot for privacy and consistency (Ramesh et al., 15 Mar 2026).

Raw data are recorded as ROS 2 bags containing RGB frames, the event stream, and robot state, including odometry and commanded velocities. Post-processing converts these streams into synchronized HDF5 tuples

$v$1

where $v$2 is an RGB frame, $v$3 an event representation, and $v$4 the action label $v$5 (Ramesh et al., 15 Mar 2026).

A key component is hardware-level synchronization. An STM32F407 microcontroller generates a 30 Hz 50% duty-cycle trigger sent to both cameras. This trigger defines RGB exposure times and injects markers into the event stream for temporal alignment. Spatial calibration is performed with a blinking checkerboard to estimate a homography $v$6 mapping RGB pixels into the event camera frame, after which RGB frames are warped into the event frame before fusion (Ramesh et al., 15 Mar 2026).

Temporally, RGB frame timestamps serve as anchors:

$v$7

The raw event stream is

$v$8

with polarity $v$9. For each RGB frame $\omega$0 at time $\omega$1, the associated event set contains all events since the previous frame:

$\omega$2

This produces an event window aligned to the frame interval rather than an independently clocked asynchronous control loop (Ramesh et al., 15 Mar 2026).

Ground-truth actions are aligned over the same interval. If

$\omega$3

then the label is

$\omega$4

The labels are taken from /odom twist and averaged over $\omega$5, rather than using joystick commands. The stated purpose is to ensure that labels reflect executed motion and reduce noise from teleoperation latency and jitter (Ramesh et al., 15 Mar 2026).

Unless otherwise specified, the train/validation/test split is 80/10/10 with held-out trajectories for test. One environment is used for in-distribution training and testing, and the other two are reserved for generalization studies (Ramesh et al., 15 Mar 2026).

3. Event representation and late-fusion architecture

The event representation used by ENP-Fusion is a 2-channel polarity histogram over the RGB inter-frame interval $\omega$6. For spatial size $\omega$7, the two channels count positive and negative polarities:

$\omega$8

$\omega$9

This yields a tensor $1280\times720$0. In the released models, both modalities are resized to $1280\times720$1, so RGB input is $1280\times720$2 and event input is $1280\times720$3 (Ramesh et al., 15 Mar 2026).

At the encoder level, ENP-Fusion uses two parallel MobileNetV3-Small backbones, one for RGB and one for events. The RGB encoder is pre-trained and frozen to preserve features and mitigate overfitting in low light, whereas the event encoder is trainable end-to-end so that it can adapt to the statistics of polarity histograms (Ramesh et al., 15 Mar 2026). Each encoder emits a single feature vector, or token, per frame.

Fusion is performed with a Transformer encoder block operating on the 2-token sequence

$1280\times720$4

where $1280\times720$5 and $1280\times720$6 are the modality tokens. The attention mechanism is standard scaled dot-product attention:

$1280\times720$7

With only two tokens, the module is described as learning cross-modal dependencies and reweighting contributions per sample. The paper does not specify the number of Transformer layers, attention heads, or hidden size, and it notes that explicit positional encodings are optional for this two-token case (Ramesh et al., 15 Mar 2026).

The fused representation is then passed to a compact MLP that predicts the 2D action

$1280\times720$8

Activation functions, dropout, and layer sizes are not specified in the paper (Ramesh et al., 15 Mar 2026).

The stated architectural advantage of late fusion is that it avoids forcing heterogeneous modalities—asynchronous sparse events and dense RGB images—to co-adapt at the pixel level or in early feature maps. This suggests that the main design objective is not only multimodal aggregation, but modality specialization followed by adaptive cue selection under changing illumination and motion regimes.

4. Behavioral cloning objective and training regimes

Training is framed as imitation learning with a behavioral cloning objective over dataset

$1280\times720$9

where $1280\times1024$0 and $1280\times1024$1. The loss is an $1280\times1024$2 action regression objective:

$1280\times1024$3

The paper states that $1280\times1024$4 loss, interpreted as MAE, is chosen for robustness to outliers and high-frequency label noise (Ramesh et al., 15 Mar 2026).

Training is implemented in PyTorch with AdamW, learning rate $1280\times1024$5, weight decay $1280\times1024$6, batch size 64, and up to 50 epochs with early stopping of patience 8 based on validation MAE. Inputs are resized to $1280\times1024$7; RGB is normalized according to MobileNetV3 conventions, and event tensors are provided as raw per-polarity counts. The paper does not report explicit image or event augmentations (Ramesh et al., 15 Mar 2026).

The study evaluates 12 training configurations formed by the Cartesian product of three architectures, two path-complexity settings, and two lighting regimes. The three architectures are ENP-RGB, ENP-Event, and ENP-Fusion. The path settings are Single-Path, using P1 only, and Multi-Path, using P1+P2+P3. The lighting settings are Normal-Light training only and Mixed-Light training with both normal and low-light data (Ramesh et al., 15 Mar 2026).

Observed optimization behavior is also reported. Incorporating events, through either ENP-Event or ENP-Fusion, converges faster and to lower validation MAE than RGB-only training. RGB-only models tend to train for the full 50 epochs, whereas event and fusion models often trigger early stopping between 20 and 35 epochs (Ramesh et al., 15 Mar 2026). A plausible implication is that the event representation provides a stronger supervisory signal for action regression in this task setting, especially when low-light data are included.

5. Quantitative performance and ablations

The evaluation protocol uses offline action prediction MAE on the linear and angular components, together with a Total MAE averaged across components. In addition to the learned policy variants, the baselines include a classical YOLOv8n detector with a PID controller (Ramesh et al., 15 Mar 2026).

Validation results reported in Table 1 are as follows.

Training configuration	ENP-RGB	ENP-Event	ENP-Fusion
Single-Path, Normal-Light	0.0548	0.0393	0.0388
Multi-Path, Normal-Light	0.0667	0.0378	0.0402
Single-Path, Mixed-Light	0.0584	0.0388	0.0358
Multi-Path, Mixed-Light	0.0707	0.0416	0.0370

The most prominent validation result is Multi-Path, Mixed-Light, where ENP-Fusion achieves 0.0370 MAE versus 0.0707 for ENP-RGB, described in the paper as approximately 47.7% lower (Ramesh et al., 15 Mar 2026). Across these validation settings, fusion is either best or near-best, and event-bearing models consistently outperform RGB-only models.

Test results under Normal-Light training further distinguish simple and complex regimes. For Single-Path, Normal-Light training and testing in Normal versus Low light, the reported Total MAEs are:

Model	Normal light	Low light
ENP-RGB	0.0210	0.0213
ENP-Event	0.0266	0.0235
ENP-Fusion	0.0221	0.0222
YOLOv8n+PID	0.0819	0.0713

For Multi-Path, Normal-Light training and testing in Normal versus Low light, the reported Total MAEs are:

Model	Normal light	Low light
ENP-RGB	0.0463	0.0514
ENP-Event	0.0305	0.0534
ENP-Fusion	0.0335	0.0467
YOLOv8n+PID	0.0998	0.1033

These results support three explicit ablation conclusions from the paper. First, fusion consistently provides the best or near-best MAE, particularly in complex trajectories and mixed or low light. Second, event-only often outperforms RGB-only in validation and simple settings, but in complex low-light tests it can lose global context that RGB provides, making fusion preferable. Third, Mixed-Light training yields the best overall robustness, although even Normal-Light training alone shows that fusion generalizes better to low light than RGB-only on complex trajectories (Ramesh et al., 15 Mar 2026).

Trajectory analysis in Fig. 4 reports that ENP-Fusion trained on Multi-Path, Mixed-Light tracks fast segments of approximately 0.33 m/s and turning maneuvers in both normal and low light. A representative low-light trajectory plot shows linear MAE of approximately 0.125 and angular MAE of approximately 0.078 (Ramesh et al., 15 Mar 2026). The paper also notes slightly noisier $1280\times1024$8 predictions in low light, while major directional changes remain captured.

6. Deployment characteristics, limitations, and future directions

The reported onboard compute platform is an NVIDIA Jetson Orin Nano used for sensor drivers and logging, while training and evaluation were performed offline. Synchronization is anchored by the 30 Hz hardware trigger, so event windows are aggregated over 33.3 ms intervals and the synchronized policy naturally operates at the RGB frame rate of 30 Hz (Ramesh et al., 15 Mar 2026).

The principal deployment constraint identified in the paper is computational overhead. Constructing synchronized event tensors and running dual encoders with a Transformer on an edge device is described as challenging, and single-modality variants such as ENP-Event or ENP-RGB are noted as more feasible when strict real-time budgets apply on the Orin Nano (Ramesh et al., 15 Mar 2026). The implementation guidance supplied by the release includes pre-warping RGB into the event frame using homography $1280\times1024$9, pre-allocating event histograms, and using ring buffers to limit per-frame allocations. The document further notes that reducing resolution below $a=(v,\omega),$0 or replacing the Transformer with concatenation plus MLP can reduce latency at some performance cost (Ramesh et al., 15 Mar 2026).

Several limitations and failure modes are reported. The fixed-frequency alignment and event accumulation incur nontrivial compute overhead, limiting on-board fusion rates. Dataset scale is moderate at approximately 2 hours and 175 episodes, and the paper states that more diverse environments, more subjects, and longer trajectories would further strengthen generalization. In multi-path low-light tests, ENP-Event shows increased angular error, reflecting the possibility that events alone lack sufficient global context. Low-light operation also introduces somewhat noisier angular predictions more generally (Ramesh et al., 15 Mar 2026).

Future directions suggested by the paper are concrete. These include asynchronous policies that ingest RGB at frame rate and events at native higher rates without strict synchronization, stronger fusion mechanisms and better low-light RGB preprocessing, and online learning approaches such as DAgger together with sequence-level policies such as diffusion-style action chunking to smooth trajectories and improve closed-loop robustness (Ramesh et al., 15 Mar 2026). This suggests that the present late-fusion policy is best viewed as a synchronized multimodal baseline and systems framework rather than a final formulation of event-based navigation control.

7. Position within the literature and reproducibility resources

Within the broader literature, the paper places late-fusion RGB-Event navigation policies at the intersection of RGB-only imitation learning and event-based perception. It states that end-to-end visuomotor control from RGB is standard for navigation but degrades in low light and during fast motion due to blur and limited dynamic range. It also notes that event-based perception has extensive literature in visual odometry, SLAM, and optical flow, but that few works learn end-to-end control from real event data (Ramesh et al., 15 Mar 2026).

The specific contribution claimed for eNavi comprises three elements: a real-world indoor person-following dataset with synchronized RGB, raw events, and executed action labels $a=(v,\omega),$1; a multimodal late-fusion policy with dual MobileNetV3 encoders and a Transformer attention block trained via behavioral cloning; and a thorough real-data evaluation across 12 training variations showing that fusion reduces action error and improves robustness, especially for complex trajectories and low-light conditions (Ramesh et al., 15 Mar 2026).

Reproducibility is supported through the project release, which includes raw ROS 2 bags, processed HDF5 files containing synchronized tuples $a=(v,\omega),$2, the multimodal synchronization pipeline, and trained models. The reproduction checklist given in the paper specifies the 30 Hz hardware synchronization, homography estimation using a blinking checkerboard, polarity-histogram event construction, resizing to $a=(v,\omega),$3, odometry-twist action averaging, MobileNetV3-Small backbones with the RGB branch frozen, the $a=(v,\omega),$4 behavioral cloning objective, AdamW with the stated hyperparameters, and evaluation by per-component and total MAE over the 12 training configurations (Ramesh et al., 15 Mar 2026).

In that sense, the late-fusion RGB-Event navigation policy represented by ENP-Fusion functions both as a control model and as an experimental template for real-world multimodal event-based navigation. Its central empirical claim is narrowly defined but consequential: under the synchronized indoor person-following setting studied in eNavi, combining RGB and event observations through late fusion yields lower action prediction error, faster convergence, and improved low-light robustness than RGB-only baselines, while avoiding some of the brittleness of event-only control in more complex trajectories (Ramesh et al., 15 Mar 2026).