CEMFormer: Cross-View Episodic Memory Transformer

• CEMFormer is a unified transformer framework that fuses multi-camera video streams with recurrent episodic memory for driver intention prediction.
• It integrates a spatial-temporal encoder and novel context-consistency loss to enforce logical coherence and enhance prediction accuracy on benchmark datasets.
• The architecture achieves state-of-the-art performance with fewer parameters, improving F1 scores and real-time inference speed.

CEMFormer (Cross-View Episodic Memory Transformer) is a unified, recurrent transformer-based framework for driver intention prediction that integrates multi-camera video streams—including both in-cabin and forward-facing (external) views—with a compact episodic memory mechanism and a novel context-consistency loss. CEMFormer is designed to anticipate maneuvers such as lane changes or turns by jointly processing visual input and historical scene information, and by explicitly enforcing logical consistency with the driving context. This architecture achieves state-of-the-art performance on benchmark datasets for intention prediction while maintaining computational efficiency and leveraging fewer parameters than previous multi-modal approaches (Ma et al., 2023).

1. Model Architecture

CEMFormer employs a spatial-temporal encoder based on the Vision Transformer (ViT-B/16) backbone, extended to support: (a) multi-view (cross-camera) fusion, (b) recurrent episodic memory integration, and (c) joint representation learning across time and views.

Input Representation

Each time step $t$ receives $M$ synchronized RGB frames, $x_{t,1}, \ldots, x_{t,M}$, where typically $M = 2$ (in-cabin and external). Each frame, with dimensions $C \times H \times W$ ($C = 3$, $H = W = 224$), is partitioned into $N_m = (H \cdot W)/P^2$ non-overlapping patches ($P = 16$), flattened and projected to $D$-dimensional tokens ($M$0 for ViT-B/16), incorporating view-specific learned positional embeddings:

$M$1

where $M$2 and $M$3.

All view tokens are concatenated:

$M$4

Recurrent Episodic Memory

To incorporate historical context, $M$5 learnable memory tokens, $M$6, are prepended:

$M$7

where $M$8 is chosen empirically.

A stack of $M$9 transformer encoder layers (e.g., $x_{t,1}, \ldots, x_{t,M}$0) processes $x_{t,1}, \ldots, x_{t,M}$1. Each layer uses standard multi-head self-attention (MHSA) and MLP blocks:

$x_{t,1}, \ldots, x_{t,M}$2

with $x_{t,1}, \ldots, x_{t,M}$3 and $x_{t,1}, \ldots, x_{t,M}$4.

After the final layer, the first $x_{t,1}, \ldots, x_{t,M}$5 positions are sliced as updated memory for the next time step:

$x_{t,1}, \ldots, x_{t,M}$6

This mechanism realizes a trainable, recirculating memory bank, enabling compression and propagation of salient historical context.

Cross-View Fusion

Self-attention is performed jointly over memory tokens and all input view tokens, enabling unified context modeling across modalities (in-cabin, external) and time.

2. Context-Consistency Loss

CEMFormer incorporates a loss function that augments standard cross-entropy with a context-consistency term designed to enforce compatibility between predicted intentions and the current traffic context.

• Cross-Entropy Loss ($x_{t,1}, \ldots, x_{t,M}$7) at each time step:

$x_{t,1}, \ldots, x_{t,M}$8

where $x_{t,1}, \ldots, x_{t,M}$9 is a one-hot ground-truth label and $M = 2$0 is the softmax output over intention classes.

• Context-Consistency Loss ($M = 2$1): To penalize predictions that contradict known driving constraints, the following auxiliary loss is introduced:

$M = 2$2

where $M = 2$3 is the set of incompatible (intention, context) pairs, $M = 2$4 represents the current driving context (e.g., leftmost, rightmost, near intersection), and $M = 2$5 is the predicted probability for intention $M = 2$6.

• Joint Objective:

A temporally exponentially weighted sum is used to encourage early and confident prediction:

$M = 2$7

3. Training Protocol and Dataset

• Dataset: All evaluations use the Brain4Cars benchmark comprising 594 video clips with dual-view (in-cabin and forward-facing) data and 5 intention classes: go straight, left lane change, left turn, right lane change, right turn.
• Preprocessing:
  • Frames are resized to $M = 2$8.
  • Frames are split into $M = 2$9 patches per view.
  • Clips are partitioned into $C \times H \times W$0 equal-duration segments; a single frame per segment is sampled (TSN-inspired).
  • Random horizontal flips and crops act as augmentation.
• Optimization:
  • Encoder initialized from DINO-pretrained ViT-B/16 on ImageNet.
  • Only the self-attention layers of the spatial-temporal encoder are fine-tuned.
  • AdamW optimizer, weight decay 0.05, cosine learning rate schedule (initial $C \times H \times W$1).
  • Batch size 10, trained over 200 epochs on NVIDIA RTX 3090 Ti.
• Validation:
  • 5-fold cross-validation, mean and standard deviation of accuracy and F1 reported.

4. Experimental Results

CEMFormer demonstrates superior performance across all evaluated settings, as summarized below:

Configuration	Accuracy (%)	F1 (%)	Gain vs. SOTA	Parameter Count
In-cabin only	84.5	82.7	+1.4/1.0	—
External only	64.8	66.3	+4–6 acc	—
Multi-view fusion	85.4	87.1	+2.8 F1	~87M (~60% less)

• CEMFormer outperforms Gebert et al. (83.1%/81.7% in-cabin) and prior multi-view methods by 2.8 F1 on Brain4Cars.
• Parameter efficiency: approximately 87 million parameters, which is about 60% fewer than leading competing architectures.

Ablation Studies:

• Episodic Memory module raises F1 by ≈4.4%.
• Context-Consistency loss confers a +5.4% F1 improvement.
• Combined EM and CC yield 85.37% accuracy / 87.09% F1.
• Optimal number of memory tokens: $C \times H \times W$2. Lower or higher $C \times H \times W$3 (e.g., 2 or 8) show diminished and more variable gains.
• Inference throughput: 22 FPS (single-view) and ≈15.6 FPS (dual-view) on RTX 3090 Ti.

5. Qualitative Analysis and Attention Interpretation

Attention analysis of the final encoder layer provides critical insight into CEMFormer's mechanisms:

• Without episodic memory, attention patterns are frame-local and temporally variable, resulting in inconsistent saliency and degraded intention prediction.
• With episodic memory, the model exhibits temporally stable and spatially coherent focus—e.g., on temporally relevant driver cues such as head pose and steering wheel from early time steps, refining attention as more context accumulates.
• Absence of context-consistency loss permits attention drift to contextually irrelevant regions, such as backgrounds.
• Integration of both context-consistency loss and dual-view input encourages attention towards logically consistent, task-relevant regions, including driver gaze and appropriate lane markers, respecting traffic constraints like lane boundaries.

6. Architectural and Methodological Significance

CEMFormer delivers a compact, real-time-capable driver intention prediction system that enhances prior transformer-based and fusion techniques. Its key contributions are:

• Unified memory-augmented spatial-temporal transformer for cross-view and temporal fusion.
• Trainable episodic memory bank enabling efficient history compression and propagation.
• Augmentation of standard intention classification losses with auxiliary context-based supervision, demonstrably improving prediction fidelity and logical coherence with real-world constraints.

By jointly leveraging multi-modal visual input and explicit temporal reasoning, with direct enforcement of logical consistency, CEMFormer sets a new state-of-the-art in compact, accurate driver intention modeling on publicly available benchmarks (Ma et al., 2023).