RPEP: RGB Pre-training for Event Hand Pose

• The paper introduces a novel pre-training framework that leverages labeled RGB images and unlabeled event streams to address the scarcity of event labels in 3D hand pose estimation.
• It constructs pseudo-event–RGB pairs via an iterative accumulation process that decomposes non-rigid hand motion into incremental intermediate states.
• The framework achieves significant improvements by aligning features with a shared MLP head and enforcing a motion reversal constraint to bridge the RGB-to-event modality gap.

Searching arXiv for the RPEP paper and closely related event-based hand pose estimation work. RPEP, short for Leveraging RGB Images for Pre-training of Event-based Hand Pose Estimation, is a pre-training framework for event-based 3D hand pose estimation that uses labeled RGB hand images as supervision, unlabeled event streams as motion priors, and pseudo RGB-event pairs to bridge the modality gap (Liu et al., 21 Sep 2025). It is presented as the first pre-training approach of this kind for the task. The method is motivated by a structural asymmetry in data availability: event cameras offer very high temporal resolution, low latency, and robustness to challenging lighting and motion blur, but event-based hand pose estimation remains constrained by the scarcity of labeled event datasets, with EvRealHands identified as the only large-scale real dataset mentioned in the source paper (Liu et al., 21 Sep 2025).

1. Problem setting and motivation

Event-based hand pose estimation is attractive because event cameras measure brightness changes with properties that are well matched to fast, articulated motion. The paper specifically emphasizes very high temporal resolution, low latency, and robustness to challenging lighting and motion blur as the central advantages of event sensing (Liu et al., 21 Sep 2025). These properties are particularly relevant for hand tracking, where fine articulation and rapid local motion are common.

The principal bottleneck is annotation scarcity. The source paper states that there are very few labeled event datasets, and that even EvRealHands relies on a complex synchronized sensor rig, which limits scale and diversity (Liu et al., 21 Sep 2025). RPEP addresses this by avoiding dependence on large labeled event corpora. Instead, it repurposes existing labeled RGB datasets and combines them with unpaired event data so that RGB supervision becomes transferable to event-based estimators.

A common misconception in this area is that RGB-to-event transfer can be handled by generic pseudo-event generation designed for rigid or stationary scenes. RPEP argues that this assumption is inadequate for hands. The paper’s central claim is not merely that event labels are scarce, but that the modality gap is amplified by the non-rigid and temporally evolving nature of hand motion (Liu et al., 21 Sep 2025). This suggests that successful transfer requires a hand-specific event construction mechanism rather than a direct reuse of rigid-scene methods.

2. Pseudo-event–RGB pair construction

RPEP constructs pseudo-event–RGB pairs so that a labeled RGB image can supervise an event-based pose estimator (Liu et al., 21 Sep 2025). In such a pair, the RGB image provides the ground-truth hand pose, while a generated pseudo-event frame approximates what an event camera would have observed for the corresponding hand motion. This is the mechanism by which RGB annotations are made useful for event-based pre-training.

The input formulation distinguishes the two modalities explicitly. The method uses a labeled RGB image

$$\mathbf{x}_{\text{rgb}} \in \mathbb{R}^{H \times W \times 3}$$

and an unlabeled event frame

$$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$

with the event input represented as an event histogram (Liu et al., 21 Sep 2025). Two feature extractors are used: $\mathcal{F}_{\text{rgb}}$ extracts appearance features from RGB, and $\mathcal{F}_{\text{ev}}$ extracts both an appearance feature $\mathbf{f}_{\text{ev}}$ and a motion prior $\mathbf{z}_{\text{ev}}$. The RGB branch only requires appearance features $\mathbf{f}_{\text{rgb}}$, since the RGB image is stationary (Liu et al., 21 Sep 2025).

Knowledge transfer is then mediated through feature alignment. The method shares the MLP head $\mathcal{P}$ between RGB and event branches, uses adversarial learning to align $\mathbf{f}_{\text{rgb}}$ and $\mathbf{f}_{\text{ev}}$, aligns $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$0 with pseudo-event features $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$1, and aligns motion priors $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$2 with pseudo-event motion priors $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$3 (Liu et al., 21 Sep 2025). The stated purpose is to learn a common latent space spanning RGB, real events, and pseudo-events.

This formulation differs from direct domain adaptation in a narrower sense. Rather than only mapping RGB features into the event domain, RPEP synthesizes a pseudo-event representation tied to a labeled RGB pose and then aligns both appearance and motion structure. A plausible implication is that the transfer problem is treated simultaneously at the data level and the representation level.

3. Why prior pseudo-event generation is insufficient for hands

The paper explicitly contrasts RPEP with prior pseudo-event generation methods such as EKLT and RPG-EV, which typically generate pseudo-events from a single RGB frame and implicitly assume that the object between two frames mostly undergoes rigid motion such as translation or rotation (Liu et al., 21 Sep 2025). This assumption is tractable for stationary objects or simple frame-to-frame changes, but the paper argues that it fails for hands.

The failure mode is anatomical and temporal. For hands, fingers bend, joints articulate, and motion is non-rigid and temporally evolving (Liu et al., 21 Sep 2025). Prior constructions tend to generate events mainly along static edges, whereas real hand events are denser and concentrated in articulation regions such as fingers and palm. The result is a domain gap between pseudo-events and real event data.

This point is central to understanding RPEP. The paper’s argument is not only that previous pseudo-events are sparse, but that they are sparse in the wrong places. In event-based hand pose estimation, the salient evidence often lies in articulation-induced local intensity changes rather than in globally rigid displacement. This suggests that fidelity to the event formation process must be evaluated with respect to articulated motion, not merely edge correspondence.

The paper also addresses a related misconception: that final frame-to-frame differences are sufficient to approximate event accumulation. RPEP rejects this view for hands, arguing that real event cameras respond to continuous intensity changes over time, not simply to static initial-versus-final discrepancies (Liu et al., 21 Sep 2025).

4. Iterative motion decomposition and pseudo-event generation

RPEP’s principal technical contribution is an iterative accumulation process for pseudo-event generation that is intended to be closer to how real event cameras work (Liu et al., 21 Sep 2025). Rather than generating events once from a static frame, the method starts from an RGB image, predicts motion, warps the image to create intermediate articulated hand states, generates pseudo-events at each step, and accumulates them over time.

The construction process is divided into $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$4 iterations, with the paper reporting the empirical choice

$$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$5

At each iteration $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$6, a decoder $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$7 predicts an optical flow map

$$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$8

and the flow is used both to warp the RGB image to the next intermediate articulated state and to generate a sub-pseudo event frame $$\mathbf{x}_{\text{ev}} \in \mathbb{R}^{H \times W \times 2},$$9 (Liu et al., 21 Sep 2025).

The final pseudo-event frame is accumulated across iterations: $\mathcal{F}_{\text{rgb}}$0 The paper interprets this as event generation driven by intensity changes across warped intermediate RGB states (Liu et al., 21 Sep 2025). In operational terms, the procedure computes image gradients, combines them with flow, generates event contributions at each iteration, and sums them over time.

The significance of this design lies in step-by-step motion decomposition. By decomposing motion into smaller steps, each step yields a slightly different intermediate hand pose, allowing the pseudo-events to capture gradual finger bending, palm shifts, local articulation, and time-varying appearance changes (Liu et al., 21 Sep 2025). This decomposition is identified by the paper as the most important part of the method. It is the mechanism by which RPEP attempts to close the gap between rigid-scene pseudo-events and the denser, articulation-centered event structure of real hands.

5. Motion reversal constraint and motion-prior regularization

RPEP introduces a motion reversal constraint to ensure that estimated motion priors are semantically meaningful (Liu et al., 21 Sep 2025). From a constructed pseudo-event stream, the method creates a reversed-motion version by flipping both the motion direction and the trajectory.

If the original pseudo-event stream is $\mathcal{F}_{\text{rgb}}$1, the reversed version $\mathcal{F}_{\text{rgb}}$2 is generated as

$\mathcal{F}_{\text{rgb}}$3

The paper explains that $\mathcal{F}_{\text{rgb}}$4 reverses motion direction, while using frames from the subsequent iteration $\mathcal{F}_{\text{rgb}}$5 inverts the trace (Liu et al., 21 Sep 2025).

The extracted motion features from the original and reversed streams are required to differ. To impose this, the method introduces a divergence loss $\mathcal{F}_{\text{rgb}}$6 that minimizes their cosine similarity (Liu et al., 21 Sep 2025). The stated effect is to regularize the motion-prior encoder so that it learns true physical motion cues rather than arbitrary appearance artifacts.

This component serves an important disambiguating role. In a multimodal transfer pipeline, appearance-correlated shortcuts can masquerade as motion understanding. The motion reversal constraint directly penalizes such degeneracy by forcing directional sensitivity in the learned motion representation. A plausible implication is that this improves not only pseudo-event realism but also the downstream utility of motion features during fine-tuning.

6. Training protocol, datasets, and evaluation

RPEP is trained in two stages. The paper reports a pre-train phase using labeled RGB data from InterHand2.6M and unlabeled event data from EvRealHands, followed by fine-tuning on a small number of labeled EvRealHands samples (Liu et al., 21 Sep 2025). The target task throughout is event-based 3D hand pose estimation.

The evaluation is conducted on EvRealHands, with splits by Normal, Strong light, and Flash, and each lighting condition further divided into Scripted and Unscripted subsets (Liu et al., 21 Sep 2025). The reported evaluation metrics are 3D-MPJPE and PA-MPJPE.

The paper compares RPEP against a broad set of baselines: SimCLR, Vid2E, CycleGAN, ADDA, RPG-EV, a baseline without pre-training, and synthetic-event pre-training on EventHands (Liu et al., 21 Sep 2025). This comparison is methodologically important because it situates RPEP against self-supervised pre-training, cross-modal transfer, adversarial adaptation, prior pseudo-event generation, and synthetic pre-training.

The training setup clarifies the scope of the contribution. RPEP is not presented as a fully supervised event-only method, nor as a synthetic-data-first pipeline. Instead, it is a pre-training framework that exploits the asymmetry between abundant labeled RGB hand images and scarce labeled event data. This suggests that its primary value lies in annotation-efficient transfer rather than in changing the downstream estimator definition itself.

7. Empirical results, implications, and limitations

The paper reports that RPEP achieves the best performance across all evaluated scenarios on EvRealHands (Liu et al., 21 Sep 2025). Quantitatively, it reports 24% relative improvement in normal scenes, 20% in strong light, and 14% in flash light. It also states that RPEP outperforms RPG-EV by an approximately 8 mm lower error.

An additional result concerns low-label adaptation. The method is reported to perform strongly when only a small number of labeled event samples are available for fine-tuning, which the paper identifies as one of its major practical benefits (Liu et al., 21 Sep 2025). This directly supports the paper’s motivating claim that expensive event annotation can be partially substituted by RGB supervision plus unlabeled event motion priors.

The paper also reports a negative result of broader relevance: pre-training on the synthetic EventHands dataset performs poorly—worse than even training from scratch—indicating that synthetic-to-real event transfer has a large domain gap (Liu et al., 21 Sep 2025). This serves as an objective counterpoint to the assumption that synthetic event pre-training is automatically beneficial. In the context of RPEP, it strengthens the argument for using real labeled RGB data and real unlabeled event data rather than relying primarily on synthetic events.

The practical implications stated in the paper include reduced dependence on scarce event annotations, leveraging widely available RGB datasets, improved robustness in challenging lighting and fast motion, and strong performance even with few labeled event samples (Liu et al., 21 Sep 2025). The paper identifies application areas including AR/VR hand tracking, robotics, egocentric interaction understanding, and low-latency hand motion capture.

The source paper does not dwell on limitations in depth, but it identifies several implicit constraints. The quality of pseudo-events still depends on how well motion can be inferred from RGB and event priors; the method assumes access to some unlabeled event streams during pre-training; and it is designed specifically for hand motion, so extension to more complex articulated objects may require adaptation (Liu et al., 21 Sep 2025). These are not framed as failures of the method, but as boundary conditions on its applicability.

In summary, RPEP is defined by two coupled ideas: iterative motion decomposition for non-rigid hand articulation and a motion reversal constraint for motion-semantic regularization (Liu et al., 21 Sep 2025). Together with pseudo-event–RGB pair construction and cross-modal alignment, these components are used to close the RGB-to-event annotation gap in event-based 3D hand pose estimation.