SMoEStereo: Adaptive Stereo Matching

• The paper introduces SMoEStereo, a stereo matching framework that integrates MoE-LoRA and MoE-Adapter for adaptive low-rank and convolutional refinement.
• It employs a lightweight decision network to selectively activate expert modules per layer, enhancing cross-domain robustness and controlling computational overhead.
• Empirical evaluations demonstrate improved accuracy and efficiency over fully tuned models, with significant memory savings and faster inference times.

SMoEStereo is a stereo matching framework that adapts Vision Foundation Models (VFMs) to dense disparity estimation through a selective Mixture-of-Experts design centered on two parameter-efficient adaptation mechanisms, MoE-LoRA and MoE-Adapter, together with a lightweight decision network that can disable adaptation modules on a per-layer basis (Wang et al., 7 Jul 2025). It is presented for robust stereo matching “in the wild,” with the stated objective of improving cross-domain performance under domain shifts and imbalanced disparity distributions while controlling the computational overhead that would arise from na\"ive VFM adaptation. The framework combines a frozen ViT-style backbone, scene-specific expert routing, a shallow CNN feature compression stage, correlation-volume construction, and recurrent cost aggregation using a multi-level GRU as in RAFT-Stereo, producing a final dense disparity estimate $\hat D$ (Wang et al., 7 Jul 2025).

1. Problem formulation and design rationale

The framework is motivated by two difficulties identified for learning-based stereo matching networks: lack of robustness and weak cross-domain performance under domain shifts and imbalanced disparity distributions among diverse datasets (Wang et al., 7 Jul 2025). The proposed response is not full end-to-end fine-tuning of a large VFM, but a parameter-efficient adaptation strategy that preserves a frozen backbone and injects scene-dependent capacity only where needed.

Within this formulation, VFMs are treated as a source of robust, large-scale pre-trained representations, but their direct incorporation into stereo matching is described as challenging if one seeks to do so cost-effectively and to fully realize their robustness. SMoEStereo addresses this by combining Low-Rank Adaptation and Mixture-of-Experts. The resulting architecture is explicitly selective: expert blocks are present inside Transformer layers, but a learned decision mechanism can suppress them when the input does not warrant the extra adaptation cost (Wang et al., 7 Jul 2025).

This suggests that the framework is not merely a PEFT variant transplanted into stereo, but an attempt to make adaptation conditional on scene complexity. A plausible implication is that robustness is pursued through heterogeneity of adaptation capacity rather than through uniform scaling of model size.

2. End-to-end architecture

The input is a rectified stereo pair $(I_L, I_R)$ (Wang et al., 7 Jul 2025). Both views are processed by a frozen Vision Foundation Model, for example a ViT backbone pre-trained on large-scale data. Inside each Transformer block, SMoEStereo interleaves two MoE modules: MoE-LoRA on the self-attention projections and MoE-Adapter on the MLP path. A small decision network may disable them per layer (Wang et al., 7 Jul 2025).

After the Transformer stages, a shallow CNN composed of a small stack of residual CNN layers compresses token features back to a regular 2D feature map with strong locality. Stereo matching then proceeds through a standard dense correspondence pipeline. For each pixel in the left view, the method computes inner-product cost with a window of right-view features over a disparity range, producing a 4D cost volume. This volume is then processed by recurrent cost aggregation using a multi-level GRU, described as being “as in RAFT-Stereo,” which iteratively refines a disparity field until the final dense disparity $\hat D$ is obtained (Wang et al., 7 Jul 2025).

The data flow is summarized in the source description as follows:

$W$3

Architecturally, the framework therefore couples frozen global representations from a VFM with explicit geometric processing stages typical of stereo systems: locality restoration through shallow convolution, correlation-volume construction, and iterative disparity refinement (Wang et al., 7 Jul 2025).

3. Adaptive modules inside the frozen VFM

MoE-LoRA

Standard LoRA is introduced as a low-rank update $\Delta W = A B^\top$ injected into a projection matrix $W$ while freezing $W$ itself. SMoEStereo replaces a single fixed-rank update with an $M$-way Mixture-of-Experts:

$$x_{\rm out} = W_{qkv}\,x_{\rm in} \;+\;\sum_{i=1}^M \bigl[R_L(x_{\rm in})\bigr]_i\;\bigl(E^i_L(x_{\rm in})\bigr)$$

where each expert is

$$E^i_L(x_{\rm in}) = \Delta W_i \,x_{\rm in} = \bigl(W^{\rm up}_i\,W^{\rm down}_i\bigr)\,x_{\rm in}, \quad W^{\rm down}_i\in\mathbb R^{r_i\times d},\; W^{\rm up}_i\in\mathbb R^{d\times r_i}.$$

Each expert therefore has its own rank $r_i$ rather than sharing a single global rank. The router $(I_L, I_R)$0 is described as a tiny linear head plus softmax that selects the Top-k experts for each token:

$(I_L, I_R)$1

At test time, the configuration is $(I_L, I_R)$2, meaning that the single best expert is chosen (Wang et al., 7 Jul 2025).

The source description characterizes this as allowing the framework to “dial” a small rank on “easy” patches and a larger rank on “hard” ones. In the terms of the method, MoE-LoRA is therefore the component responsible for scene-adaptive low-rank refinement of self-attention projections.

MoE-Adapter

The second adaptive component addresses a different limitation: “Plain ViT MLPs lack locality” (Wang et al., 7 Jul 2025). To reintroduce locality within the frozen Transformer, SMoEStereo inserts a bank of $(I_L, I_R)$3 small CNN adapters with different kernel sizes $(I_L, I_R)$4 into the MLP path. The modified MLP computation is

$(I_L, I_R)$5

with each adapter expert defined as

$(I_L, I_R)$6

The router $(I_L, I_R)$7 follows the same style as $(I_L, I_R)$8 and chooses which receptive field to apply per token or per patch (Wang et al., 7 Jul 2025).

Taken together, the two modules divide adaptation labor across complementary functions. The description states that low-rank adaptation supports long-range cue refinement, while convolutional adapters inject geometric inductive bias. This suggests a deliberate separation between global representational adjustment and local geometric feature extraction.

4. Selective routing, gating, and optimization

Activating every MoE module in every layer is described as costly and partly redundant. To mitigate this, SMoEStereo introduces a selective decision network that learns, for each Transformer layer $(I_L, I_R)$9, two binary masks $\hat D$0 to gate MoE-LoRA and MoE-Adapter respectively (Wang et al., 7 Jul 2025).

The input to the decision network is the layer’s $\hat D$1 token $\hat D$2. This token is passed through a small 2-layer MLP to produce scores $\hat D$3, and the Gumbel-Softmax trick with temperature $\hat D$4 is then applied to obtain a soft binary decision vector. During inference, the model uses arg-max to obtain a hard $\hat D$5 decision. The gating rule is expressed as

$\hat D$6

If $\hat D$7, the MoE-LoRA block is skipped and only the frozen $\hat D$8 is used; the same applies to $\hat D$9 for the adapter path (Wang et al., 7 Jul 2025). The decision network is thus not an auxiliary classifier but a learned computational budget controller embedded in the forward pass.

Training uses three loss terms. The stereo loss follows RAFT-Stereo and applies an $\Delta W = A B^\top$0 penalty over a sequence of disparity iterates $\Delta W = A B^\top$1:

$\Delta W = A B^\top$2

A balance loss encourages all experts to be used:

$\Delta W = A B^\top$3

and is applied to both routers $\Delta W = A B^\top$4 and $\Delta W = A B^\top$5 (Wang et al., 7 Jul 2025). A usage loss encourages the average kept ratio to match a budget $\Delta W = A B^\top$6:

$\Delta W = A B^\top$7

The total objective is

$\Delta W = A B^\top$8

The stated training protocol consists of pre-training on synthetic SceneFlow without adaptation to real domains, followed by cross-domain evaluation by freezing the model and testing on KITTI, Middlebury, and ETH3D. For joint generalization in the RVC setting, the model is fine-tuned on mixed KITTI + Middlebury + ETH3D train splits and tested on all three with a single model (Wang et al., 7 Jul 2025).

5. Efficiency profile and parameter economy

The efficiency discussion is a central part of the framework’s definition. Full VFM fine-tuning is described as requiring updates to all approximately $\Delta W = A B^\top$9 VFM parameters, incurring huge training cost even though it introduces no extra inference cost. A na\"ive MoE alternative, in which all experts remain active at all times, adds roughly 24 additional blocks and slows training by approximately $W$0 and inference by approximately $W$1 (Wang et al., 7 Jul 2025).

SMoEStereo’s decision network is reported to prune approximately $W$2 of MoE blocks on average at $W$3, thereby recovering much of this overhead (Wang et al., 7 Jul 2025). The source gives the following practical comparison from Table 3:

Quantity	SMoEStereo	Fully-tuned / other PEFT baselines
Memory	1.9–2.0 GB	4–6 GB for fully-tuned ViT-Large baselines
Inference time	0.18–0.20 s/frame	0.47–0.52 s/frame
Activated VFM parameters at test	only 2–4 M (LoRA+Adapter)	~6.9 M in other PEFT methods

These figures locate the method within parameter-efficient adaptation rather than full model updating. The description further states that the selective MoE approach achieves state-of-the-art cross-domain and joint-domain robustness with only a few million extra parameters (Wang et al., 7 Jul 2025). A plausible implication is that the gating mechanism is not merely a convenience for deployment, but a structural requirement for making expertized adaptation competitive under practical memory and latency constraints.

6. Empirical behavior, ablations, and stated limitations

The reported experimental results are organized around cross-domain zero-shot generalization, weather robustness, and joint generalization (Wang et al., 7 Jul 2025).

For cross-domain zero-shot evaluation, training is performed on SceneFlow and testing on KITTI 2012/15 using Bad 3px, Middlebury using Bad 2px, and ETH3D using Bad 1px. The reported performance is approximately $W$4, stated to beat previous domain-general methods including CFNet, UCFNet, and CREStereo++ (Wang et al., 7 Jul 2025). For weather robustness on DrivingStereo, the metric is D1 error under Sunny, Cloudy, Rainy, and Foggy conditions, with an average of $W$5 versus approximately $W$6 for the prior best. For joint generalization in RVC 2022, evaluation on Middlebury, KITTI 2015, and ETH3D with a single model yields overall rank 1 among approximately 20 submissions (Wang et al., 7 Jul 2025).

The ablation studies clarify the contribution of individual components. MoE-LoRA only yields an approximately $W$7–$W$8 drop from full SMoE, while MoE-Adapter only gives a similar single-module improvement. Using both MoEs without the decision network gives the best accuracy but with $W$9 runtime. Random gating at the same keep ratio causes an approximately $W$0–$W$1 accuracy drop, which is presented as evidence that learned gating is essential. Comparisons between single-rank LoRA with $W$2 and the heterogeneous SMoE configuration indicate that SMoE is consistently better across all test sets (Wang et al., 7 Jul 2025).

The stated limitations and future directions are specific. Current routing is performed per token with a lightweight gating head; more sophisticated routing, such as spatially-varying or multi-task-aware routing, is identified as a possible direction for further improvement. Other open avenues include integrating other VFMs, such as depth-oriented ViTs, and incorporating self-supervised multi-modal cues. The description also states that extending the framework to stereo plus monocular depth fusion or to multi-view stereo is straightforward and promising (Wang et al., 7 Jul 2025).

In interpretive terms, the framework’s central claim is that robustness can be improved by combining selective low-rank adaptation with convolutional inductive bias inside a frozen VFM, while using a learned budget mechanism to avoid unconditional activation of all experts. This suggests a broader research pattern in which stereo matching is treated as a test case for scene-adaptive PEFT rather than as a domain requiring complete retraining of large vision backbones.