OpenLongTail for Multi-View Driving Data

• OpenLongTail is an open-source generative data engine that transforms monocular dashcam videos into pose-grounded, synchronized multi-view assets for autonomous driving.
• It employs a four-stage pipeline—trajectory recovery, depth estimation, generative view synthesis, and cross-view memory—to convert heterogeneous inputs into policy-ready surround-view data.
• Evaluations demonstrate improved visual fidelity, geometric consistency, and driving robustness, nearly matching real multi-view data in closed-loop tests.

OpenLongTail is an open-source generative data engine for scaling autonomous driving policies under long-tail events by converting heterogeneous, often monocular driving videos into pose-grounded, synchronized multi-view assets under a fixed target camera rig for policy learning (Liu et al., 10 Jul 2026). It addresses a modality gap in which rare safety-critical events are widely recorded in the wild, especially by monocular dashcams, but modern end-to-end driving policies typically require multi-view, synchronized, pose-annotated inputs. The system combines metric-scale ego-trajectory recovery, pose-informed extrapolative view synthesis, Plücker-ray conditioning, and cross-view memory so that generated surround-view rollouts can be used as training data and then evaluated in closed loop (Liu et al., 10 Jul 2026).

1. Terminology and conceptual scope

The phrase “long tail” has domain-specific meanings. In web search, it can denote low-ranked results for otherwise ordinary queries rather than rare queries themselves (Schaer et al., 2016). In session-based recommendation, it refers to low-frequency items in a heavy-tailed item-popularity distribution (Liu et al., 2020). In open long-tailed recognition, it denotes learning under naturally long-tailed and open-ended data, with both seen classes and open classes appearing at test time (Liu et al., 2022). OpenLongTail uses the term in a different but related sense: rare, safety-critical driving events that are underrepresented in curated multi-sensor corpora yet frequently observed in heterogeneous real-world video sources (Liu et al., 10 Jul 2026).

Within this driving setting, the long tail includes work zones with cones and temporary lanes, human-guided intersections, animals on the road, unusual vehicle types, and extreme weather (Liu et al., 10 Jul 2026). These events are not primarily scarce in the world; rather, they are scarce in the calibrated, synchronized, multi-view format required by VLA and end-to-end driving policies. OpenLongTail therefore defines the central problem as one of conversion rather than mere collection: transforming monocular or partial-view recordings into multi-view assets that are sufficiently view-aligned, temporally coherent, and pose-grounded to support policy learning (Liu et al., 10 Jul 2026).

2. Problem formulation and system pipeline

OpenLongTail takes as input a monocular or front-view driving video $\mathbf{x}^{\mathrm{mono}}_{1:T}$, optionally partial multi-view data and rough poses, together with a specification of a target camera rig $\mathcal{R}$, and produces a multi-view rollout $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$ under that rig (Liu et al., 10 Jul 2026). The paper factorizes the problem as

$$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$

with the first term representing camera trajectory recovery and the second geometry-grounded generative synthesis (Liu et al., 10 Jul 2026).

The pipeline has four stages. First, OpenLongTail uses MapAnything to estimate a metric trajectory $$\hat{\mathcal{T}}_{1:T} = \{\hat{\mathbf{T}}_t\}_{t=1}^T$$, with $\hat{\mathbf{T}}_t \in SE(3)$, and then applies Kalman filtering and Rauch–Tung–Striebel smoothing to obtain a stable trajectory $\tilde{\mathcal{T}}_{1:T}$ (Liu et al., 10 Jul 2026). Second, it constructs geometry from $\tilde{\mathcal{T}}_{1:T}$ and the target rig, including per-pixel front-view depth from a frozen DepthCrafter model, Plücker rays for all target pixels, analytic depth warps from front to side and rear views, and a cross-view dependency graph for autoregressive multi-view generation (Liu et al., 10 Jul 2026). Third, it performs pose-informed extrapolative view synthesis with a Wan 2.1-VACE diffusion backbone conditioned on front-view latents, Plücker ray geometry, temporal depth warping, and a learned cross-view memory bank, generating five missing non-front views: cross-left, cross-right, rear-left, rear-right, and rear-tele (Liu et al., 10 Jul 2026). Fourth, the resulting logs are integrated into a VLA policy such as Alpamayo-R1 for supervised fine-tuning and closed-loop evaluation in AlpaSim (Liu et al., 10 Jul 2026).

A defining characteristic is that the system is extrapolative rather than interpolative. Because the input is often only a front camera, side and rear target cameras may observe regions never visible in the same frame, or visible only at different times as the ego vehicle moves. OpenLongTail therefore does not assume sufficient multi-view overlap and instead synthesizes views under a fixed target rig by combining recovered pose, depth-based temporal warping, and generative completion (Liu et al., 10 Jul 2026).

3. Geometry, view synthesis, and cross-view consistency

OpenLongTail’s geometric core is a pose-informed view-synthesis formulation. For a front camera $0$ and target camera $t$, if a front-view pixel $\mathcal{R}$0 at time $\mathcal{R}$1 has depth $\mathcal{R}$2, then same-frame warping is defined through unprojection and reprojection via the relative pose $\mathcal{R}$3 (Liu et al., 10 Jul 2026):

$\mathcal{R}$4

$\mathcal{R}$5

For rear cameras with little or no overlap, the system uses temporal offset warping, drawing from a past front frame $\mathcal{R}$6 whose field of view covers what the rear camera sees at time $\mathcal{R}$7 (Liu et al., 10 Jul 2026). This explicit use of time offsets is central to synthesizing rear views from monocular front-view video.

Cross-view and temporal coherence are further constrained through Plücker ray geometry. A ray is represented as $\mathcal{R}$8, where $\mathcal{R}$9 is direction and $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$0 is moment; OpenLongTail computes, for each token location $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$1 in view $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$2 at time $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$3,

$\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$4

These rays are expressed in a common ego-anchor frame using the stabilized trajectory $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$5, and then embedded by a geometry encoder together with camera-ID, stream-role, and trajectory-context embeddings (Liu et al., 10 Jul 2026). The resulting geometry features are broadcast-added into the main DiT hidden states, the VACE control branch, and the memory bank embeddings, so the generative model shares a consistent 3D reference frame throughout the network (Liu et al., 10 Jul 2026).

The cross-view memory architecture is organized by a directed dependency graph. Cross-left and cross-right depend on front; rear-left depends on front and cross-left; rear-right depends on front and cross-right; rear-tele depends on front and both rear views (Liu et al., 10 Jul 2026). For each target view, the model builds dense memory and semantic memory from already available conditioning views, and at selected DiT layers target tokens attend over both memories with noise-dependent gates $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$6 and $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$7 (Liu et al., 10 Jul 2026). The dense branch provides pixel-level alignment, while the semantic branch provides higher-level consistency. This design is intended to prevent trivial front-copy solutions and to improve seam consistency across synthesized cameras.

4. Backbone, training data, and implementation

The generative backbone is Wan 2.1-VACE-1.3B, fine-tuned with a frozen Wan VAE and a flow-matching training objective rather than classical DDPM noise prediction (Liu et al., 10 Jul 2026). A noisy target-view latent is formed as

$\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$8

with target velocity field $\hat{\mathbf{X}}^{\mathcal{R}}_{1:T}$9, and training minimizes

$$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$0

where all per-view weights are $$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$1 except the rear-tele view, which uses $$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$2 to account for difficulty (Liu et al., 10 Jul 2026). Trainable components include LoRA adapters on self-attention projections $$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$3, the geometry encoder, and the memory-bank modules and gating, while the depth-warp module is analytic (Liu et al., 10 Jul 2026).

Training and in-distribution evaluation rely mainly on NVIDIA PhysicalAI Autonomous Vehicles, PandaSet, and nuScenes, with the front view used as input and side or rear views used as supervision (Liu et al., 10 Jul 2026). The reported scale is approximately 200K clips, 41 frames each, around 50K scenes (Liu et al., 10 Jul 2026). External evaluation uses Waymo WOD-E2E and the Nexar dashcam dataset, the latter being purely monocular and characterized by unknown, varying intrinsics, rolling shutter, and compression artifacts (Liu et al., 10 Jul 2026). The system is trained on latent sequences of approximately $$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$4 for 41 frames, with LoRA rank 32, AdamW, learning rate $$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$5 for LoRA and new modules, cosine scheduling with warmup 500 steps and minimum learning rate $$P\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathcal{R} \right) = P\!\left( \mathbf{p}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T} \right) \cdot P_{\theta}\!\left( \hat{\mathbf{X}}^{\mathcal{R}}_{1:T} \mid \mathbf{x}^{\mathrm{mono}}_{1:T}, \mathbf{p}_{1:T}, \mathcal{R} \right),$$6, gradient clip 0.5, global batch size 32 clips, and 20K training steps on 32 H200 GPUs for roughly 96 hours (Liu et al., 10 Jul 2026).

For downstream policy training, the paper uses Alpamayo-R1-10B in a two-stage SFT recipe. The camera interface includes Front, Front-tele as a crop of front, Cross-left and Cross-right as either ground-truth multi-view or OpenLongTail synthesis, and ego-motion recovered via the OpenLongTail trajectory module (Liu et al., 10 Jul 2026). The project promises release of code, model checkpoints, generated long-tail multi-view data, and conversion scripts and configuration for PAV, Waymo, and Nexar so that others can integrate the system into their own driving stacks and VLA pipelines (Liu et al., 10 Jul 2026).

5. Evaluation protocol and empirical performance

OpenLongTail is evaluated along three axes: generative quality, pose recovery quality, and closed-loop driving robustness (Liu et al., 10 Jul 2026). Generative quality includes per-view frame metrics PSNR, SSIM, and LPIPS; distributional metrics FID and FVD; and cross-view geometric consistency through GeoKPM, which computes the fraction of LoFTR keypoint matches satisfying epipolar constraints across synchronized generated views (Liu et al., 10 Jul 2026). Pose evaluation reports metric-scale and Sim(3)-aligned ATE, RPE-r, RPE-t, RotErr, TrErr, Jerk, Acc.Var, and LDJ (Liu et al., 10 Jul 2026). Closed-loop evaluation is performed in AlpaSim on 53 long-tail events, with AS as a composite driving score and CR as collision rate (Liu et al., 10 Jul 2026).

On unseen scenes, the paper reports higher PSNR and lower LPIPS across all target cameras than camera-controlled baselines such as TrajectoryCrafter, Gen3C, ReCamMaster, and Vista4D, together with a large gain in GeoKPM (Liu et al., 10 Jul 2026). In the unseen-scenario table, cross-left PSNR is 13.28 versus 11–12 for the baselines, LPIPS is 0.597 versus 0.649–0.734, and GeoKPM mean is 82.41 versus 8.61–18.86 (Liu et al., 10 Jul 2026). These numbers support the claim that the system improves not only visual fidelity but also adherence to multi-view geometry.

For ego-trajectory recovery, evaluated on 218 clips from 109 scenes, OpenLongTail matches MapAnything on metric-scale ATE at 2.212 while improving RPE-r to 0.129 from 0.155 and reducing Jerk to 283.9 from 4737.4 and Acc.Var to 5.96 from 53.22 (Liu et al., 10 Jul 2026). In Sim(3)-aligned evaluation it reports ATE 0.162, best RotErr among metric-scale methods, and the lowest Jerk and Acc.Var by a large margin (Liu et al., 10 Jul 2026). The stated interpretation is that smoothing preserves metric accuracy while making the trajectory substantially more stable for rendering and policy conditioning.

The closed-loop results are the paper’s central systems claim. Base Alpamayo-R1 achieves AS approximately 0.534 and CR approximately 58.8% on the long-tail AlpaSim suite (Liu et al., 10 Jul 2026). Fine-tuning with ground-truth multi-view long-tail data yields AS 0.764 and CR 0.0%, while fine-tuning with OpenLongTail synthesized data yields AS 0.748 and CR 0.0% (Liu et al., 10 Jul 2026). The generated multi-view assets therefore nearly match real multi-view data in improving robustness under long-tail events. When external Waymo E2E synthesized assets are added, the paper reports gains in uncommon vehicles, cyclists, and work zones, together with some regression in complex intersections attributed to distribution mismatch because Waymo lacks the human-guided and cone-guided patterns abundant in the NVIDIA data (Liu et al., 10 Jul 2026).

6. Broader context, limitations, and research significance

In the wider literature, long-tail problems are addressed through recognition, recommendation, continual learning, and mining rather than through multi-view data generation alone. “Open Long-Tailed Recognition++” defines a setting in which learning must handle imbalance, few-shot classes, open classes, and active exploration in one framework (Liu et al., 2022). Long-tailed class incremental learning studies ordered and shuffled incremental task streams under long-tailed distributions and emphasizes classifier bias correction through two-stage learning and learnable weight scaling (Liu et al., 2022). HeroLT extends benchmarking across text, vision, and graph modalities with heterogeneous tasks and long-tailedness metrics such as imbalance factor, Gini coefficient, and Pareto-LT ratio (Wang et al., 2023). VLMine addresses the data-selection side of the problem by mining rare examples from unlabeled corpora with vision-LLMs and Pareto mining (Ye et al., 2024). This suggests that OpenLongTail occupies the data-generation layer of a broader long-tail stack: it converts rare heterogeneous observations into training assets, whereas the surrounding literature often focuses on how classifiers, detectors, or miners should exploit long-tail data once such assets exist.

The system’s limitations are explicitly practical and safety-critical. Inference is compute-heavy because diffusion-based multi-view generation over 41 frames and five extrapolated views is expensive, making the framework more suitable for offline dataset generation than real-time deployment (Liu et al., 10 Jul 2026). Residual artifacts and temporal flicker can appear in highly dynamic scenes, especially in heavily extrapolated rear-tele views (Liu et al., 10 Jul 2026). Large discrepancies in camera hardware, including varying intrinsics and rolling-shutter artifacts, can still induce geometry mismatches or blurring despite the reported robustness on Nexar (Liu et al., 10 Jul 2026). The method also relies on MapAnything for metric reconstruction, so pose errors propagate into rays and warps, and the framework remains video-only rather than integrating LiDAR, radar, or BEV priors (Liu et al., 10 Jul 2026). Ethical and safety considerations follow directly: synthetic views must not hide or hallucinate critical agents in ways that distort risk, and validation with real data and closed-loop simulators remains necessary (Liu et al., 10 Jul 2026).

Future directions identified in the paper include integrating more sensors, improving efficiency through distillation or lighter autoregressive models, conditioning more explicitly on camera parameters, using distribution-aware source selection and weighting, and extending the approach beyond driving to domains such as mobile robotics and drones (Liu et al., 10 Jul 2026). Taken together, these directions position OpenLongTail not as a generic synonym for open long-tailed learning, but as a specific generative infrastructure for transforming scattered monocular long-tail observations into multi-view, policy-ready assets for robust autonomous driving under rare events (Liu et al., 10 Jul 2026).