FR3D: 3D Forecasting in Vision & Fluid Dynamics

• FR3D is a term used for distinct systems, representing a disentangled 3D world model for monocular video and a conformal-mapping approach for fluid flow reconstruction.
• In computer vision, FR3D leverages a persistent latent space and dual masked transformer forecasters to separately predict scene geometry and ego-camera motion for improved forecasting.
• In fluid dynamics, FR3D employs a convolutional autoencoder aided by conformal mapping to reconstruct full pressure and velocity fields from sparse measurements, enabling detailed aerodynamic force estimation.

FR3D is a name used in recent technical literature for distinct three-dimensional reconstruction systems rather than for a single unified method. In computer vision and autonomous-scene modeling, FR3D denotes a disentangled 3D world model for future dynamic 3D reconstruction from monocular video, forecasting depth or point maps and ego-camera trajectory in a persistent 3D latent space (Morbitzer et al., 16 Jun 2026). In fluid dynamics, FR3D denotes a conformal-mapping-aided convolutional autoencoder for reconstructing unsteady three-dimensional flow fields and estimating lift and drag around extruded bluff bodies from sparse measurements (Özbay et al., 2023). The shared acronym does not imply a shared implementation or application domain.

1. Nomenclature and research usage

The term FR3D appears in at least two unrelated research contexts in the cited literature.

FR3D usage	Domain	Core description
FR3D (Morbitzer et al., 16 Jun 2026)	Computer vision, world modeling	A disentangled 3D world model for future dynamic 3D reconstruction from monocular video
FR3D (Özbay et al., 2023)	Fluid dynamics	A conformal-mapping-aided convolutional autoencoder for 3D flow reconstruction and force estimation

In the first usage, FR3D addresses future dynamic 3D reconstruction: given a short monocular context, typically $N=4$ frames, it predicts the future 3D scene together with the ego-camera trajectory for up to about $2$ seconds, without access to future images. In the second usage, FR3D reconstructs full pressure and velocity fields around extruded bluff bodies with widely varying cross-sections from sparse measurements, and then derives quantities such as the $Q$-criterion and aerodynamic forces. This suggests that any citation of “FR3D” benefits from explicit domain qualification (Morbitzer et al., 16 Jun 2026, Özbay et al., 2023).

2. FR3D as a disentangled 3D world model

In the vision formulation, FR3D is motivated by a limitation of purely 2D world models: camera motion and scene dynamics are entangled in image space, which degrades geometric consistency over time and can produce morphing or vanishing objects and depth-inconsistent parallax. FR3D instead operates in a persistent 3D latent space and explicitly separates ego-motion from world motion, treating the inferred ego-motion as a latent proxy for action. The stated objective is to resolve the self-motion versus environment-motion ambiguity and preserve physical consistency over long horizons (Morbitzer et al., 16 Jun 2026).

A defining design choice is that FR3D does not use a volumetric grid, tri-planes, NeRF/SDF, or 3D Gaussians. It forecasts directly in the latent space of CUT3R, a feed-forward, stateful 3D reconstruction model. The CUT3R latent comprises a compact persistent state $s_t \in \mathbb{R}^{768 \times 768}$, a spatial token grid $F_t \in \mathbb{R}^{D \times H_F \times W_F}$ derived from the current image $I_t$, and a pose token $z_t$ that encodes the ego-camera pose at time $t$. Given $I_t$, CUT3R’s encoder $E$ produces per-frame tokens $2$0, and two cross-attending decoders combine $2$1 with the previous state $2$2 to produce state-enriched tokens and an updated scene state.

FR3D adds two masked transformer forecasters on top of these state-enriched tokens. The pose forecaster $2$3 predicts future pose tokens, while the spatial forecaster $2$4 predicts future scene tokens. The two streams exchange information via four bidirectional cross-attention layers spaced through depth. This coupling is central to the method’s factorization: pose and scene are forecast separately, but not independently, so that predicted depth and predicted camera motion remain geometrically compatible. The readout stage does not directly render RGB; instead, FR3D uses CUT3R’s frozen heads to read out intrinsics, ego poses, and multi-view consistent depth or point maps. The paper presents this as a way to avoid differentiable ray-marching or radiance rendering while inheriting CUT3R’s generalization (Morbitzer et al., 16 Jun 2026).

The geometric formulation is explicit. For a world point $2$5, camera intrinsics $2$6, and world-to-camera pose $2$7, projection is written as

$2$8

with ego-motion updates

$2$9

Dynamic scene evolution is not represented as an explicit 3D motion field; rather, it is implicit in the evolution of the spatial tokens. The paper frames the model as a factorized latent world model in which observation likelihood is mediated by the teacher’s decoder, scene transitions are implemented by $Q$0, and ego-motion transitions by $Q$1 (Morbitzer et al., 16 Jun 2026).

3. Training, rollout, and empirical behavior in monocular forecasting

FR3D is trained by teacher-student token distillation. The teacher is the frozen feed-forward reconstruction model CUT3R, pre-trained on $Q$2 datasets. For each training sequence, the teacher’s state-enriched tokens are precomputed at every timestep, and FR3D is trained to regress the next-step teacher tokens from a context of $Q$3 frames. The pose-token and spatial-token objectives use a Smooth L1 loss with $Q$4, and the total loss is

$Q$5

with $Q$6. FR3D itself uses no photometric supervision and no geometry reprojection loss; the paper attributes efficiency and zero-shot generalization to the fact that it learns dynamics directly in the teacher’s latent token space (Morbitzer et al., 16 Jun 2026).

The training schedule uses a sliding-window curriculum that progressively increases the number of self-predicted tokens in the context while keeping the context length $Q$7. Architecturally, the spatial forecaster has $Q$8 layers, $Q$9 heads, and hidden dimension $s_t \in \mathbb{R}^{768 \times 768}$0; the pose forecaster has $s_t \in \mathbb{R}^{768 \times 768}$1 layers, $s_t \in \mathbb{R}^{768 \times 768}$2 heads, and hidden dimension $s_t \in \mathbb{R}^{768 \times 768}$3; and there are four bidirectional cross-attention blocks. Optimization uses AdamW with $s_t \in \mathbb{R}^{768 \times 768}$4, learning rate $s_t \in \mathbb{R}^{768 \times 768}$5 in pretraining and $s_t \in \mathbb{R}^{768 \times 768}$6 in finetuning, cosine decay, effective batch size $s_t \in \mathbb{R}^{768 \times 768}$7 on $s_t \in \mathbb{R}^{768 \times 768}$8A100 GPUs, and rollout horizon during training up to $s_t \in \mathbb{R}^{768 \times 768}$9 steps.

The training distribution is Waymo Open Dataset. Zero-shot evaluation is reported on KITTI at $F_t \in \mathbb{R}^{D \times H_F \times W_F}$0 and nuScenes at $F_t \in \mathbb{R}^{D \times H_F \times W_F}$1. Depth is evaluated with AbsRel and $F_t \in \mathbb{R}^{D \times H_F \times W_F}$2 at depth capped at $F_t \in \mathbb{R}^{D \times H_F \times W_F}$3 m; pose is evaluated with ATE, $F_t \in \mathbb{R}^{D \times H_F \times W_F}$4, and $F_t \in \mathbb{R}^{D \times H_F \times W_F}$5 with Sim(3) alignment.

Benchmark and horizon	FR3D	Comparator
KITTI depth at $F_t \in \mathbb{R}^{D \times H_F \times W_F}$6 s	$F_t \in \mathbb{R}^{D \times H_F \times W_F}$7 (AbsRel / $F_t \in \mathbb{R}^{D \times H_F \times W_F}$8)	DINO-Foresight: $F_t \in \mathbb{R}^{D \times H_F \times W_F}$9; Copy Last: $I_t$0
nuScenes depth at $I_t$1 s	$I_t$2	DINO-Foresight: $I_t$3; Copy Last: $I_t$4
KITTI pose at $I_t$5 s	$I_t$6 for $I_t$7	Representative FR3D value reported
nuScenes pose at $I_t$8 s	$I_t$9	Representative FR3D value reported

On the teacher distribution, Waymo depth at $z_t$0 s and $z_t$1 s is reported as $z_t$2 and $z_t$3, compared with DINO-Foresight at $z_t$4 and $z_t$5, and CUT3R-Prompt at $z_t$6 and $z_t$7. The qualitative account emphasizes smoother and more physically plausible longer rollouts, such as turning trajectories and traffic moving in opposite directions, with reduced morphing and better object persistence than 2D feature forecasters.

Ablations attribute the gains to the pose-scene factorization and the cross-attention exchange between the two streams. Enabling cross-attention significantly boosts both depth and pose forecasting, especially at $z_t$8 s. Autoregressive training improves long-horizon stability by reducing compounding error under self-conditioning. Fixed-camera subsequences show that relative depth change is much smaller in static regions than in dynamic ones after per-frame scale alignment, with static regions at about $z_t$9–$t$0 and dynamic ones at about $t$1–$t$2, which the paper interprets as evidence that scene dynamics are not spuriously attributed to camera motion. The reported limitations are scale drift across rollout, underrepresentation of lateral motion in training, lack of explicit uncertainty, sensitivity to rapid or erratic egomotion, and the difficulty of highly nonrigid dynamics without explicit 3D motion fields. Practical efficiency is also reported: peak VRAM is about $t$3 GB and total latency about $t$4 s for a $t$5-second forecast given $t$6 context frames, compared with about $t$7 s and about $t$8 GB for Vista (Morbitzer et al., 16 Jun 2026).

4. FR3D as conformal-mapping-aided flow reconstruction

In fluid mechanics, FR3D addresses the flow reconstruction task for unsteady three-dimensional flows around extruded bodies with different cross-sections. The inputs are sparse measurements, either point sensors or planes, and the outputs are full fields for pressure $t$9 and velocity components $I_t$0, $I_t$1, and $I_t$2, sampled on a uniform $I_t$3 grid in a geometry-invariant mapped space. From these reconstructions, the framework computes derived quantities such as the $I_t$4-criterion and instantaneous lift and drag forces (Özbay et al., 2023).

The key geometric device is a conformal mapping from each doubly-connected physical fluid domain to a canonical annulus in a complex plane. The inner circle of the annulus maps to the body boundary and the outer circle to the far-field boundary. Sampling then proceeds in cylindrical coordinates $I_t$5 on the mapped domain, with $I_t$6 points in each dimension. Because the inner ring corresponds exactly to the body surface for every geometry, body-surface alignment is built into the representation. The paper presents this mapping as the mechanism that enables generalization to unseen shapes: disparate geometries are standardized into a common coordinate system before learning.

The architecture has three submodels: an encoder $I_t$7 that reduces a dense field to a latent embedding, a decoder $I_t$8 that reconstructs the dense field from the latent code, and a latent embedder $I_t$9 that maps sensor inputs to a latent code compatible with the decoder. The encoder and decoder use 3D convolutional blocks with residual connections, BatchNorm, LeakyReLU, average pooling in the encoder, and transpose convolutions for upsampling in the decoder. Separate models are trained for $E$0, $E$1, $E$2, and $E$3. In the sparse setup, the parameter counts are reported as $E$4 for the encoder, $E$5 for the decoder, and $E$6 for the latent embedder, for a total of $E$7 parameters.

Training uses only mean squared error; physics-informed penalties are not used. The procedure has two stages. First, the autoencoder $E$8 is trained on dense fields. Second, the trained decoder is frozen and $E$9 is trained to map sensor vectors to latent codes so that decoded outputs match the ground-truth dense fields. Optimization uses Adam with initial learning rate $2$00, and early stopping is based on validation loss. Each variable converged in about $2$01 hours on dual NVIDIA A100 GPUs.

The dataset contains $2$02 randomly generated 2D cross-sections obtained via Bézier curves with $2$03 control points, including convex, concave, sharp-cornered, and airfoil-like sections. Each cross-section is placed in a $2$04 square domain in $2$05–$2$06, extruded in $2$07 by $2$08, meshed with about $2$09 elements, and simulated with PyFR using an artificial-compressibility solver for incompressible Navier–Stokes at Reynolds number $2$10. The split is $2$11 training geometries and $2$12 withheld geometries for validation and test, enforcing generalization to unseen shapes. The sparse sensing configuration uses $2$13 pressure sensors on the body surface plus $2$14 downstream velocity sensors; the plane setup uses the same pressure sensors plus two perpendicular downstream velocity planes with $2$15 sensors each.

The derived quantities are explicit. The $2$16-criterion is computed from the reconstructed velocity field using

$2$17

where $2$18 is the symmetric strain tensor and $2$19 the antisymmetric rotation tensor. Lift and drag are obtained from the surface traction

$2$20

with pressure forces approximated on body-surface patches and skin friction estimated by one-sided wall-normal differences. Because the mapped inner ring lies exactly on the true body surface, the force calculation does not require interpolation from a Cartesian grid to the surface (Özbay et al., 2023).

5. Generalization, quantitative performance, and limitations in fluid mechanics

Validation is performed on the $2$21 unseen geometries. In the sparse-sensor setup, averaged over a subdomain $2$22 in $2$23–$2$24 and the full span in $2$25, the reported errors are: pressure MAPE $2$26, min–max MAPE $2$27, MSE $2$28, min–max MSE $2$29; $2$30 MAPE $2$31, min–max MAPE $2$32, MSE $2$33, min–max MSE $2$34; $2$35 MAPE $2$36, min–max MAPE $2$37, MSE $2$38, min–max MSE $2$39; and $2$40 MAPE $2$41, min–max MAPE $2$42, MSE $2$43, min–max MSE $2$44. In the plane-sensor setup, the reported MAPEs are $2$45 for pressure, $2$46 for $2$47, $2$48 for $2$49, and $2$50 for $2$51 (Özbay et al., 2023).

Quantity	Sparse setup	Plane setup
Pressure $2$52	MAPE $2$53	MAPE $2$54
Velocity $2$55	MAPE $2$56	MAPE $2$57
Velocity $2$58	MAPE $2$59	MAPE $2$60
Velocity $2$61	MAPE $2$62	MAPE $2$63

The force estimates are more accurate than the worst fieldwise MAPE values might suggest. For sparse sensors, the mean absolute percentage errors are $2$64 for $2$65 and $2$66 for $2$67; for plane sensors, they are $2$68 for $2$69 and $2$70 for $2$71. Time histories of $2$72 and $2$73 reportedly track the ground truth from PyFR closely, with particularly strong performance on high-aspect-ratio airfoil-like shapes and larger errors on rare concave shapes. Qualitatively, iso-surfaces at $2$74 recover vortex cores, hairpin and finger-like streamwise structures, and spanwise-bent vortical features across different unseen geometries.

Several limitations are explicit. The model is trained and validated only at $2$75; performance at other Reynolds numbers is unknown. It is defined for extruded, spanwise-uniform bodies with periodic spanwise boundary conditions; highly three-dimensional non-extruded geometries are out of scope. The quality of the annulus map affects sampling alignment and near-wall gradient estimates, especially for complex concavities. No measurement noise is injected, so robustness to experimental noise is left for future work. Some fine-scale structures are smoothed, and residual noise appears in predicted surfaces. An adversarial GAN variant was tested but degraded $2$76 and $2$77 accuracy and worsened force estimation, with $2$78 and $2$79 MAPE rising to about $2$80 and about $2$81, so the MSE-only configuration was retained (Özbay et al., 2023).

6. Conceptual relationship and terminological ambiguity

The two FR3D systems are methodologically unrelated in implementation, supervision, and application. One forecasts future scene geometry and ego motion from monocular video by distilling a stateful 3D reconstruction teacher into a persistent latent world model; the other reconstructs pressure and velocity fields from sparse flow measurements by learning on annulus-mapped canonical grids. Their outputs are correspondingly different: depth, point maps, and $2$82 pose in one case; fluid fields, $2$83-criterion, and aerodynamic forces in the other (Morbitzer et al., 16 Jun 2026, Özbay et al., 2023).

A plausible commonality is representational rather than domain-specific. Both methods replace raw observation space with a structured 3D intermediate representation: a persistent latent state in the world-model formulation, and a geometry-invariant annular coordinate system plus latent embedding in the flow-reconstruction formulation. In both cases, this intermediate representation is used to stabilize inference from incomplete observations and to support generalization beyond the exact training instances. Beyond that broad similarity, the term “FR3D” remains an overloaded acronym whose meaning is determined by disciplinary context.