PFM-VEPAR: Prompting Foundation Models for RGB-Event Camera based Pedestrian Attribute Recognition

Abstract: Event-based pedestrian attribute recognition (PAR) leverages motion cues to enhance RGB cameras in low-light and motion-blur scenarios, enabling more accurate inference of attributes like age and emotion. However, existing two-stream multimodal fusion methods introduce significant computational overhead and neglect the valuable guidance from contextual samples. To address these limitations, this paper proposes an Event Prompter. Discarding the computationally expensive auxiliary backbone, this module directly applies extremely lightweight and efficient Discrete Cosine Transform (DCT) and Inverse DCT (IDCT) operations to the event data. This design extracts frequency-domain event features at a minimal computational cost, thereby effectively augmenting the RGB branch. Furthermore, an external memory bank designed to provide rich prior knowledge, combined with modern Hopfield networks, enables associative memory-augmented representation learning. This mechanism effectively mines and leverages global relational knowledge across different samples. Finally, a cross-attention mechanism fuses the RGB and event modalities, followed by feed-forward networks for attribute prediction. Extensive experiments on multiple benchmark datasets fully validate the effectiveness of the proposed RGB-Event PAR framework. The source code of this paper will be released on https://github.com/Event-AHU/OpenPAR

• The paper introduces a lightweight fusion model replacing dual-stream backbones with a single ViT enhanced by DCT-based frequency event prompting, achieving 90.05% mA on EventPAR.
• It leverages a Modern Hopfield Network for associative memory to retrieve prototypical patterns, integrating cross-modal contextual cues and reducing computational overhead.
• Component analysis shows that using 1 RGB frame and 5 event frames with dual prompt layers yields significant performance gains and faster inference times.

Prompting Foundation Models for RGB-Event Camera-Based Pedestrian Attribute Recognition: An Expert Analysis of PFM-VEPAR

Technical Motivation and Prior Art

Pedestrian Attribute Recognition (PAR) is central to visual surveillance, person re-identification, and activity analysis, especially under non-ideal imaging conditions such as low-light, rapid motion, and occlusion. Conventional approaches overwhelmingly rely on RGB imagery and deep feature extractors, but even advanced models fail when appearance features are weak or distorted. Event cameras, offering high temporal resolution and robust motion cues, are increasingly paired with RGB sensors. However, existing RGB-Event fusion models mostly employ dual-backbone architectures that are computationally expensive and insufficiently context-aware, neglecting global relational knowledge and contextual guidance from sample prior.

Figure 1: A comparative overview of prior RGB-Event PAR architectures versus the lightweight prompting paradigm proposed in PFM-VEPAR.

Algorithmic Innovation: Frequency-Aware Prompting and Associative Memory

PFM-VEPAR introduces a paradigm shift by discarding redundant dual-stream backbones. Instead, a lightweight Event Prompter module extracts frequency-domain features directly from event streams using DCT/IDCT transformations. These are injected as high-dimensional semantic prompts into a single ViT backbone processing RGB tokens. This approach efficiently modulates RGB feature formation with dynamic event information, achieving cross-modal fusion without architectural complexity.

Figure 2: PFM-VEPAR architecture, combining event prompting, associative memory, cross-attention, and classification head.

Associative memory augmentation further distinguishes the design. Internally, a Modern Hopfield Network (MHN) refines features by retrieving prototypical patterns, echoing Transformer self-attention. Externally, a pre-built memory bank containing clustered RGB- and event-modal prototypes allows content-addressable retrieval, regularizing the representation and enhancing robustness. A cross-modal consistency gate adaptively fuses RGB and event memories based on semantic similarity $\alpha$, optimizing inter-modal complementarity.

Figure 3: Modern Hopfield Network (left) and Hopfield layers (right) underpinning associative memory retrieval in PFM-VEPAR.

Experimental Evaluation: Quantitative Results and Component Analysis

EventPAR Dataset

On EventPAR, PFM-VEPAR achieves mean Accuracy (mA) of 90.05%, surpassing prior benchmarks VTB (88.41%) and HAP (88.28%). Recall peaks at 90.18%. The model demonstrates a further significant computational advantage: inference completes in only 92 seconds, markedly faster than key competitors (OTN-RWKV: 361s, HAP: 148s). Parameter count and GPU memory usage are substantially reduced relative to dual-backbone competitors. These results underscore the model's efficacy at both the primary recognition metric and operational speed.

DukeMTMC-VID-Attribute Dataset

On simulated RGB-Event DukeMTMC-VID-Attribute, PFM-VEPAR attains Accuracy of 67.58%, Precision of 80.21%, and F1-score of 79.16%, outstripping state-of-the-art methods even in cross-domain application. This reflects strong generalization ability, confirming that design gains are not dataset-specific.

Component Analysis and Ablation

Detailed ablation studies establish the indispensable roles of the Event Prompter and associative memory. Event Prompter alone boosts mA from 88.28% to 88.37%; memory augmentation propels it to 89.47%. Synergistic integration of both modules yields the 90.05% peak. Optimal configuration utilizes 1 RGB frame and 5 event frames, two prompt injection layers (6,8), DCT/IDCT-based frequency transformation, and a dual-modal memory bank with 100 prototypes. Notably, increasing prompt injection layers or memory size degrades performance, affirming "less is more" for context-driven multimodal learning.

Figure 4: Component analysis showing cumulative improvements from event prompting and memory augmentation.

Visualization and Qualitative Insights

Model predictions on diverse real-world samples display robust core attribute recognition (e.g., Male, Adult, BlackHair, LongSleeve, Walking) under severe blur, low illumination, and partial occlusion. Minor uncertainties persist for fine-grained or small-scale attributes (e.g., Glasses, Backpack), attributable to event camera insensitivity to static objects and global cross-attention dilution of local cues.

Figure 5: Attribute prediction visualizations highlighting model robustness and limitations regarding fine-grained details.

Theoretical and Practical Implications

PFM-VEPAR demonstrates that event-based frequency prompt fusion can reliably supplement and modulate large foundation vision models, achieving SOTA under operational constraints. The combination of frequency-aware prompt injection and associative memory provides not only computational efficiency but also robust adaptation to long-tail distributions and contextual ambiguity. The explicit theoretical linkage between MHN and Transformer-style attention elevates memory retrieval as an energy-based optimization process, extendable to diverse multimodal settings.

Practically, the architecture is immediately adaptable to pedestrian search, video surveillance, activity recognition, and other event-assist visual tasks. The lightweight, modular prompting strategy and static memory bank are conducive to on-device inference and deployment in constrained environments.

Future Directions

The limitations in fine-grained attribute recognition signal demand for explicit part-based attention and local memory augmentation. The architecture's multi-stage pipeline, while systematically ablated, is sensitive to hyperparameters; converging towards end-to-end trainability and principled simplification is a priority. Extending frequency-domain prompting and associative memory to other sensing modalities (e.g., IR, depth) and resource-limited settings will further realize the potential established by PFM-VEPAR.

Conclusion

PFM-VEPAR sets a new standard for RGB-Event PAR by integrating frequency-domain event prompting and MHN-based associative memory augmentation within a single backbone, achieving state-of-the-art recognition accuracy and efficiency. Its lightweight, context-aware multimodal fusion framework is both theoretically sound and practically impactful. Future work should address fine-grained attribute robustness via localized attention and unify pipeline stages to expand generalizability across broader vision tasks.