CVF: Variational Fusion in Segmentation

• Cross‐Branch Variational Fusion (CVF) is a distribution-aware mechanism that integrates global ViT features with local CNN details for improved coronary artery segmentation.
• It employs latent Gaussian coding and the reparameterization trick to generate adaptive, position-specific attention weights for each modality.
• Empirical results demonstrate significant Dice score improvements, particularly in challenging vessel regions, over naive fusion approaches.

Searching arXiv for the cited papers to ground the article in current records. Tool invocation: arxiv_search(query="(Dong et al., 17 Jul 2025)") Cross-Branch Variational Fusion (CVF) is a fusion module introduced within a parallel ViT–CNN framework for coronary artery segmentation. Its defining objective is to merge “global” features from a vision transformer branch with “local” features from a CNN branch in an adaptive, distribution-aware manner. In the underlying formulation, the two branches are not fused by direct deterministic aggregation; instead, each branch is mapped to a latent Gaussian code, attention-score distributions are estimated over those codes, and a softmax over sampled attention scores yields modality-specific weights for the final fused representation (Dong et al., 17 Jul 2025).

1. Architectural Role in the Parallel ViT–CNN Framework

CVF is positioned immediately after two encoders that run in parallel on the same 3D input patch $X \in \mathbb{R}^{C\times D\times H\times W}$. A ViT-based encoder $E_V$ from a pre-trained volumetric foundation model produces a global feature map $F_v = E_V(X)$, while a 3D U-Net-style CNN encoder $E_C$ produces a local feature map $F_c = E_C(X)$. CVF takes $\{F_v, F_c\}$ as inputs and outputs a single fused tensor $F_{\mathrm{fuse}}$ of the same spatial size, which is then sent to the decoder (Dong et al., 17 Jul 2025).

Within the broader segmentation framework, the ViT encoder is described as capturing global structural features, enhanced by activation of the final two ViT blocks and integration of an attention-guided enhancement module, whereas the CNN encoder extracts local details. CVF is therefore the mechanism that mediates between these complementary feature types. Its role is not merely to concatenate or sum branch outputs, but to construct a probabilistic fusion stage whose weights are inferred from latent distributions. This design directly targets the difficulties identified for coronary artery segmentation: the small size of the vessels, their complex morphology, and their low contrast with surrounding tissues (Dong et al., 17 Jul 2025).

A plausible implication is that CVF is intended to resolve a central tension in hybrid transformer–CNN segmentation systems: global structure is valuable for continuity and morphology, while local detail is indispensable for thin boundaries and small vessels. The module formalizes that tradeoff through branch-conditioned latent variables rather than fixed heuristics.

2. Latent Gaussian Modeling of the Two Branches

The probabilistic core of CVF begins by treating the ViT and CNN feature maps as observations for two latent Gaussian codes, $Z_v$ and $Z_c$. Two small MLPs, $E_\mu$ and $E_V$0, with tied structure but separate parameters, extract per-position means and diagonal standard deviations from each branch:

$E_V$1

$E_V$2

These are interpreted as approximate posteriors

$E_V$3

To enable backpropagation through stochastic sampling, the module uses the reparameterization trick:

$E_V$4

$E_V$5

This formulation makes the fusion stage explicitly distributional rather than point-estimate-based. Because the means and standard deviations are computed per position, the latent representation is spatially resolved. This suggests that uncertainty or ambiguity in one branch need not propagate uniformly across the feature map; instead, it can be encoded locally and then mediated during fusion (Dong et al., 17 Jul 2025).

3. Variational Attention and the Fused Representation

CVF does not fuse $E_V$6 and $E_V$7 by direct addition. Instead, it introduces another pair of MLPs to construct attention-score distributions over the two latent codes. From $E_V$8 and $E_V$9, the module computes

$F_v = E_V(X)$0

$F_v = E_V(X)$1

It then samples

$F_v = E_V(X)$2

$F_v = E_V(X)$3

The sampled score vectors $F_v = E_V(X)$4 are stacked along a new modality dimension, and a softmax across that dimension yields nonnegative fusion weights $F_v = E_V(X)$5 and $F_v = E_V(X)$6. The fused feature is then

$F_v = E_V(X)$7

where $F_v = E_V(X)$8 is a small linear projection implemented as a $F_v = E_V(X)$9 convolution for channel realignment (Dong et al., 17 Jul 2025).

The immediate significance of this design is that the fusion weights are themselves stochastic functions of the latent codes. In contrast to static branch weighting, CVF uses sampled attention scores to decide how much each branch should contribute at a given position. The source description characterizes this as allowing the network to “decide on the fly” how much global versus local information to use at each spatial location. That interpretation is consistent with the per-position Gaussian parameterization and the modality-wise softmax over sampled attention scores.

4. Optimization, Losses, and Gradient Flow

CVF is trained end to end with the rest of the network under a segmentation objective and an evidential regularization term from the evidential-learning uncertainty refinement (EUR) module. The total loss is specified as

$E_C$0

with

$E_C$1

Here, $E_C$2 is the Dirichlet-based evidential regularizer described for the EUR module. A critical clarification is that no additional KL term on $E_C$3 and $E_C$4 is used; the fusion block learns by contributing to the gradients of $E_C$5 rather than by explicit variational regularization on the branch latents themselves (Dong et al., 17 Jul 2025).

The forward procedure can be summarized as follows: parameterize branch latents from $E_C$6 and $E_C$7; sample $E_C$8 and $E_C$9 via reparameterization; parameterize attention-score distributions from those latent codes; sample attention scores; apply softmax across the stacked modality dimension to produce $F_c = E_C(X)$0 and $F_c = E_C(X)$1; and compute $F_c = E_C(X)$2 through weighted summation followed by projection. Backward propagation is automatic: gradients from the fused tensor flow to the attention weights and latent codes, then through the mean and variance networks, and finally to the ViT and CNN feature maps. This makes the entire fusion stage fully differentiable despite its stochastic sampling operations (Dong et al., 17 Jul 2025).

A common misunderstanding is to assume that any module described as “variational” must be trained with an explicit KL divergence on its latent variables. In this case, that is incorrect: the only KL-designated term belongs to the evidential regularizer, not to the Gaussian branch latents.

5. Empirical Contribution and Ablation Evidence

The reported ablation study isolates CVF’s contribution relative to simpler fusion strategies. In Table 6, replacing a naïve sum-fusion of ViT/CNN features with CVF alone, corresponding to Net 2 $F_c = E_C(X)$3 Net 3, yields a $F_c = E_C(X)$4 gain in Dice. When CVF is combined with the enhanced ViT and the uncertainty-refinement module, the full model achieves a $F_c = E_C(X)$5 absolute Dice improvement over the baseline Net 1 (Dong et al., 17 Jul 2025).

These results are presented as evidence that learning latent distributions and variational attention weights provides a significant, quantifiable boost to segmentation quality. The source further attributes the benefit especially to thin, tortuous vessel regions that challenge standard CNN/ViT hybrids. It also states that the adaptive fusion translates into better anatomical continuity, fewer false positives and false negatives, and state-of-the-art coronary artery segmentation performance (Dong et al., 17 Jul 2025).

The broader evaluation context is also relevant. The framework is reported to have been tested on one in-house and two public datasets, with extensive evaluations indicating strong generalization across multiple datasets. That claim pertains to the complete framework rather than CVF in isolation, but it situates the fusion module within a system whose performance was evaluated beyond a single benchmark setting.

6. Relation to Variational Fusion Research and Conceptual Boundaries

CVF belongs to a broader family of probabilistic fusion methods that model fused representations through random variables rather than deterministic operators. A closely related precedent is the Variational Probabilistic Fusion Network (VPFNet) for RGB-T semantic segmentation, which regards fusion features as random variables and obtains robust segmentation by averaging segmentation results under multiple samples of fusion features. In that work, random sample generation is realized by a Variational Feature Fusion Module designed based on variation attention, and prior information of illumination and category is introduced to control the module (Lin et al., 2023).

The relation between CVF and VPFNet is structural rather than task-identical. VPFNet addresses RGB and thermal images for semantic segmentation under poor lighting conditions, whereas CVF fuses transformer and CNN branches inside a coronary artery segmentation framework. The former is modality fusion in the sense of RGB-T sensing; the latter is branch fusion between global and local feature extractors. This distinction is essential, because “cross-branch” in CVF does not denote cross-sensor fusion. It denotes adaptive fusion across parallel architectural branches (Lin et al., 2023).

Another conceptual boundary concerns the meaning of “variational” in CVF. The probabilistic machinery is used to model latent distributions and derive attention weights, but the training objective does not impose a separate latent KL penalty on $F_c = E_C(X)$6 or $F_c = E_C(X)$7. Accordingly, CVF is best understood as a stochastic attention-based fusion block embedded in an end-to-end segmentation network, rather than as a standalone variational inference model.

Taken together, the available evidence characterizes CVF as a branch-level fusion mechanism that combines latent Gaussian modeling, reparameterized sampling, and modality-wise softmax attention to integrate ViT-derived global context with CNN-derived local detail. Its main documented contribution is to replace naïve branch aggregation with a distribution-aware fusion process that improves Dice performance and strengthens segmentation behavior in difficult vascular regions (Dong et al., 17 Jul 2025).