MCFormer: Multi-Cost Volume PIV Transformer
- The paper introduces MCFormer, a multi-frame transformer network that integrates multiple cost volumes to improve velocity estimation in sparse PIV imagery.
- It constructs three local cost volumes and one global cost volume to encode motion before, during, and after the target interval, enhancing temporal consistency and robustness.
- Extensive synthetic CFD benchmarks show MCFormer achieves the lowest normalized endpoint error compared to traditional two-frame optical flow methods.
Multi Cost Volume PIV (MCFormer) is a PIV-specific, multi-frame, multi-cost-volume transformer network built on the transformer-based FlowFormer optical flow architecture. It estimates the velocity field between the middle two frames of four consecutive PIV images, while using temporally enriched features and multiple cost volumes to address the sparse nature of particle imagery and the limitations of generic two-frame optical flow models on PIV data. It was introduced together with a large-scale synthetic benchmark derived from diverse CFD simulations, and on that benchmark it achieves the lowest overall normalized endpoint error (NEPE), whereas FlowFormer attains the best overall absolute endpoint error (EPE) (Lin et al., 7 Jul 2025).
1. PIV-specific problem setting
Particle Image Velocimetry estimates a continuous fluid velocity field from images of tracer particles. In this setting, the velocity vector is defined over the full flow domain, but the observed images contain sparse, bright particles on a dark background, and the particles move with the local fluid velocity. Classical cross-correlation PIV is designed for this regime, yet it assumes near-uniform motion in each interrogation window and struggles with strong shear, vortices, large displacements, low particle density, and high-resolution or time-resolved data (Lin et al., 7 Jul 2025).
The paper positions MCFormer against a specific mismatch between PIV and generic optical flow. Deep optical-flow-style CNNs such as FlowNet, PWC-Net, LiteFlowNet, RAFT, and FlowFormer are trained on dense, textured scenes such as Sintel and KITTI, where most pixels contain semantic intensity structure. By contrast, in PIV most pixels are background, motion information is concentrated only where particles appear, and large uniform background regions have zero appearance signal but a nonzero fluid velocity that must be inferred from particles elsewhere. The paper further notes that optical flow assumptions such as brightness constancy and smoothness do not map directly to isolated Gaussian blobs on black backgrounds, and that two-frame methods cannot exploit longer-term temporal coherence even though particle trajectories in fluid flows are temporally informative (Lin et al., 7 Jul 2025).
MCFormer is therefore framed as both an architectural and an evaluation response. Architecturally, it is specifically designed to exploit multi-frame temporal information and to work with sparse PIV imagery. Evaluatively, it is accompanied by a benchmark intended to expose where generic optical flow fails on PIV and to enable standardized comparison across optical flow and PIV algorithms. A plausible implication is that, in this literature, method design and benchmark design are inseparable because the same sparsity and temporal-coherence issues that motivate MCFormer also make conventional optical-flow rankings unreliable for PIV.
2. Benchmark, synthetic data generation, and metrics
A central contribution of the work is a comprehensive synthetic PIV benchmark generated from diverse CFD simulations. The flow-field sources are five flow types: homogeneous buoyancy-driven turbulence (“Mixing”), forced MHD turbulence (“MHD”), forced isotropic turbulence (“Isotropic”), turbulent channel flow (“Channel”), and a laminar boundary layer (“Blasius”). The four turbulent cases are drawn from JHTDB, whereas the boundary-layer case comes from a numerical solution of the Blasius boundary layer equation over a flat plate. This combination spans turbulent and laminar regimes, highly random and highly structured flows, and different Reynolds numbers and velocity distributions (Lin et al., 7 Jul 2025).
For the four JHTDB turbulent cases, the benchmark creates multiple effective velocity magnitudes without unphysical velocity scaling by extracting regions of size $1/4$ and $1/8$ of the original simulation domain, upsampling them back to the original resolution, and multiplying velocities in those regions by and , using $1$st-order interpolation for and $4$th-order interpolation for . This yields three velocity regimes per turbulent flow: , , and $1/8$0. Blasius uses a fixed $1/8$1 amplification. The benchmark therefore covers slow, moderate, and fast flows, including very high-speed turbulent regimes such as Isotropic $1/8$2 (Lin et al., 7 Jul 2025).
Particles are assumed to follow the fluid exactly: $1/8$3 where $1/8$4 is the ground-truth velocity at $1/8$5. Each particle is rendered as a Gaussian blob with intensity
$1/8$6
The dataset uses an iterative particle image generator: start with particles at $1/8$7, render the first image, advect particles to $1/8$8, remove particles exiting the domain, render the second image, add new particles to maintain the target density, and repeat. Three particle densities are used: dense $1/8$9 ppp, moderate 0 ppp, and sparse 1 ppp. The benchmark contains 2 conditions, each with 3 sequential pairs, for a total of 4 image pairs, split into 5 train (6) and 7 test (8) (Lin et al., 7 Jul 2025).
The evaluation uses endpoint error and normalized endpoint error: 9
0
The paper emphasizes NEPE because absolute errors in fast regions can be large simply because the underlying flows are fast, whereas NEPE measures relative error with respect to the true velocity magnitude. This metric choice is important for interpreting MCFormer’s results, because the method is strongest in overall normalized accuracy even when it is not always the best in absolute EPE (Lin et al., 7 Jul 2025).
3. Architectural basis and multi-frame feature extraction
MCFormer is built on the transformer-based FlowFormer optical flow architecture, but it is heavily modified to use four input frames, introduce multi-frame transformer feature extraction blocks, construct multiple distinct cost volumes spanning different temporal relationships, and fuse these cost volumes into a cost memory that drives flow estimation. The input is four consecutive PIV images, 1, and the output is the velocity field from 2 (Lin et al., 7 Jul 2025).
Its high-level pipeline consists of shared low-level convolutions, two Multi-Frame (MF) Blocks operating on overlapping temporal windows, construction of three local cost volumes plus one general cost volume, an adapted Cost Memory Encoder, and a Cost Memory Decoder that iteratively predicts the target flow. The two MF streams are 3 and 4, so the target interval 5 is represented from both a preceding and a following temporal window (Lin et al., 7 Jul 2025).
The MF Block is the temporal modeling core. For a triplet 6, shared initial convolutions produce
7
Using the middle frame as reference, the block applies self-attention on 8 and cross-attention from the reference to its neighbors. The resulting features are
9
For MCFormer, the first MF Block takes $1$0 with reference $1$1, and the second takes $1$2 with reference $1$3. The paper further states that large-kernel and dilated convolutions before or inside attention compensate for sparse particle signals by integrating over a larger receptive field while keeping computations tractable (Lin et al., 7 Jul 2025).
These design choices distinguish MCFormer from a simple four-frame extension of a two-frame flow model. The architecture does not merely add temporal recurrence; it uses overlapping triplets and attention-based temporal enrichment before cost-volume construction. This suggests that the method treats temporal context as something to be encoded into the feature space prior to matching, rather than as a post hoc smoothing mechanism.
4. Multi-cost-volume formulation and cost memory
The defining mechanism of MCFormer is the use of four cost volumes constructed from temporally enriched features. The paper describes three local cost volumes and one general cost volume. Although it does not write an explicit cost formula for MCFormer, it notes, following common practice in RAFT, FlowFormer, and GMFlow, that a typical cost-volume entry can be written as
$1$4
for feature maps $1$5 and $1$6 and offsets $1$7 in a search window (Lin et al., 7 Jul 2025).
The local volumes are organized around three consecutive flow segments. $1$8 correlates $1$9 with 0 and approximates the transition 1. 2 correlates 3 with 4 and directly targets the desired flow 5. 6 correlates 7 with 8 and approximates the transition 9. The paper’s rationale is that these three local volumes encode motion before, during, and after the target interval, thereby exposing temporal consistency and approximate acceleration or curvature of particle paths, while also providing cues when particles appear late or leave the field of view (Lin et al., 7 Jul 2025).
The general cost volume, $4$0, captures a broader relationship between the two temporal windows as wholes. Conceptually, it correlates the feature set from Stream 1, $4$1, with the feature set from Stream 2, $4$2. The paper describes $4$3 qualitatively as capturing the global correspondence and temporal shift between the contexts represented by the two streams. It notes that this can be implemented by concatenating features over time and forming a large 4D cost volume, or by summing multiple pairwise correlations into a unified volume (Lin et al., 7 Jul 2025).
All four volumes are fed simultaneously into an adapted Cost Memory Encoder derived from FlowFormer. FlowFormer introduced a transformer architecture in which a 4D cost volume is tokenized and processed by a cost-volume encoder-decoder; the encoder derives a cost memory with alternate-group transformer layers in a latent space, and the decoder recurrently decodes flow from the cost memory with dynamic positional cost queries (Huang et al., 2023). MCFormer retains this cost-memory logic but adapts it to a multi-cost-volume setting: the encoder fuses $4$4, $4$5, $4$6, and $4$7 into a single compact cost memory, and the decoder iteratively refines flow between $4$8 and $4$9 through dynamic, position-dependent queries (Lin et al., 7 Jul 2025).
The paper attributes several advantages to this formulation: greater temporal redundancy, better modeling of trajectories rather than only pairwise displacements, and improved robustness to sparsity and missing particles because different volumes may remain informative in different regions. A plausible implication is that MCFormer treats matching ambiguity in PIV as a multi-hypothesis problem distributed across time, rather than as a purely spatial ambiguity inside a single two-frame cost volume.
5. Training protocol, comparative performance, and empirical behavior
The training protocol is kept uniform across all models in the benchmark for fair comparison. MCFormer is trained with Adam, an initial learning rate of 0, batch size 1, and a maximum of 2 epochs; early stopping is applied if validation EPE does not improve for 3 epochs. The training loss is EPE,
4
and no explicit smoothness or regularization term is mentioned. The paper does not describe pretraining on generic datasets for MCFormer; it states that, given the PIV specificity, MCFormer is effectively trained from scratch on the PIV benchmark, whereas other baselines may use official pretraining before fine-tuning on the benchmark (Lin et al., 7 Jul 2025).
The baseline suite includes FlowNetC, FlowNetS, LiteFlowNet, LiteFlowNet3s, RAFT, SEA-RAFT, GMA, GMFlow, GMFlowNet, FlowFormer, and StreamFlow. This breadth is integral to the paper’s argument that performance on generic optical-flow benchmarks is not predictive of performance on PIV, because several strong optical flow models degrade substantially under the domain shift from natural imagery to particle fields (Lin et al., 7 Jul 2025).
| Model | Overall EPE | Overall NEPE |
|---|---|---|
| FlowFormer | 0.715 | 0.454 |
| MCFormer | 1.016 | 0.295 |
Overall, FlowFormer attains the best absolute error with EPE 5, while MCFormer attains the best normalized accuracy with NEPE 6. Per dataset, MCFormer is best on both EPE and NEPE for MHD (7 / 8), Mixing (9 / 0), and Channel (1 / 2). FlowFormer leads on Isotropic with 3 EPE and 4 NEPE, whereas MCFormer records 5 EPE and 6 NEPE. On the boundary-layer case, StreamFlow achieves the best EPE and NEPE (7 / 8), and MCFormer is close but not best (Lin et al., 7 Jul 2025).
The error analyses reported in the paper show further structure. EPE increases with velocity for most models, but MCFormer dominates in MHD, Mixing, and Channel across 9, 0, and 1. On Isotropic, MCFormer is best at 2 and 3 but degrades at 4, which drives its weaker aggregate EPE on that flow type. NEPE often decreases with speed in some datasets, and MCFormer maintains the lowest NEPE in most flow-speed combinations. With respect to particle density, many models improve from sparse 5 ppp to moderate 6 ppp but may degrade at dense 7 ppp because of particle overlap and more complex texture; MCFormer shows consistently low EPE across all densities (Lin et al., 7 Jul 2025).
The paper does not present experiments on real experimental PIV images; evaluation is confined to synthetic CFD-derived sequences. It also does not include detailed ablation charts such as removing individual cost volumes or reducing the number of frames. However, its narrative comparison to multi-frame StreamFlow and two-frame FlowFormer functions as a partial ablation. The authors state that StreamFlow is multi-frame but does not perform consistently well on PIV, whereas MCFormer’s combination of MF Blocks and multi-cost-volume integration yields much stronger performance, especially on sparse and moderate densities. This suggests that the manner in which temporal information is incorporated matters more than simply increasing the number of frames (Lin et al., 7 Jul 2025).
6. Position within multi-cost-volume research and future directions
MCFormer inherits its cost-memory foundation from FlowFormer, which introduced a transformer architecture for optical flow that tokenizes a 4D cost volume, derives a latent cost memory with alternate-group transformer layers, and recurrently decodes flow with dynamic positional cost queries. FlowFormer also introduced Masked Cost Volume AutoEncoding for pretraining and achieved top optical-flow accuracy on Sintel and KITTI-2015, but it remains a two-frame architecture centered on a single source-target image pair (Huang et al., 2023). MCFormer extends that framework into a four-frame, four-volume configuration tailored to sparse PIV imagery.
Its multi-cost-volume design also belongs to a broader class of architectures that treat matching as a fusion problem across heterogeneous cost representations. HCVFlow, for example, combines two aggregated global 3D cost volumes with a local 4D cost volume to achieve memory-efficient optical flow at high resolution, explicitly describing the result as a multi-cost-volume representation with controlled complexity (Zhao et al., 2024). IVF-AStereo, in asymmetric stereo, uses a correlation volume and a concatenation volume, then performs iterative, two-phase fusion because each volume exhibits a distinct information distortion and complementary strengths (Gao et al., 13 Aug 2025). These works differ in domain and implementation, but they situate MCFormer within a wider methodological shift from single dense pairwise matching toward structured fusion of multiple cost spaces.
For PIV specifically, MCFormer’s practical significance lies in three claims made by the paper. First, it is explicitly designed for sparse particle images through large receptive fields, dilated convolutions, attention-based temporal modeling, and multi-frame cost volumes. Second, it uses four frames to estimate a single interval flow, thereby exploiting temporal consistency without requiring long recurrent chains. Third, it produces dense, high-resolution flow fields directly, in contrast to classical cross-correlation PIV pipelines based on interrogation windows, multiple passes, window deformation, and downstream validation or cleanup steps (Lin et al., 7 Jul 2025).
The paper also states several limitations and future directions. The benchmark is synthetic, so domain gap to real PIV remains nontrivial because of noise, illumination variation, camera distortion, and out-of-plane motion. Performance degrades in extremely complex, high-speed isotropic turbulence, where FlowFormer may still be preferable. Computational cost remains substantial because multi-frame transformers with multiple cost volumes are memory-intensive. The authors identify future directions in 3D PIV and tomographic PIV, better noise and artifact modeling, physics-informed and hybrid methods, domain adaptation from synthetic to real experiments, and more efficient multi-frame cost-volume architectures for large experimental datasets (Lin et al., 7 Jul 2025).
In that sense, MCFormer is both a model and a benchmark-era statement. It formalizes the view that PIV is not adequately served by direct transplantation of natural-scene optical flow, and that sparse particle imagery benefits from temporal context and multiple cost volumes organized around distinct temporal relationships. The benchmark’s outcome—that FlowFormer is best in overall EPE while MCFormer is best in overall NEPE—also guards against a common oversimplification: no single metric or flow regime exhausts PIV performance, and the relative-error perspective is central when flow magnitudes vary widely across CFD-derived conditions (Lin et al., 7 Jul 2025).