Determine the source of late-fusion performance advantages

Determine whether the reported advantage of late-fusion architectures in automated pain assessment arises from the fusion principle itself or from the more specialised and extensively engineered modality-specific encoders used by those architectures.

Background

Automated pain-assessment systems commonly process heterogeneous physiological signals such as electromyography, electrodermal activity, and electrocardiography. Although early-fusion architectures combine modalities at an earlier stage, late-fusion architectures typically use separate modality-specific processing paths before combining learned representations or decisions. The paper notes that late- or intermediate-fusion methods often perform better than early-fusion methods, but emphasizes that this comparison is confounded by the greater specialization and engineering of their modality-specific encoders.

The unresolved issue is whether late fusion is intrinsically superior or whether a suitably selected shared early-fusion architecture could match its performance while retaining greater computational efficiency. This question is distinct from the paper’s formal research questions because the paper does not resolve the causal source of the observed late-fusion advantage.

References

However, it remains unclear whether their advantage arises from the fusion principle itself or from their more specialised and extensively engineered modality-specific encoders.

— Efficient Architecture Search under Leave-One-Subject-Out Evaluation  (2609.21457 - Hihn et al., 18 Sep 2026) in Section 1, Introduction

We conjecture this is because of how attention mechanisms are already powerful modality fusion algorithms~\citep{attn_strong,mm_transformer_survey}. Adding to that is the fact that the CLIP and BERT encoders for both modalities are state-of-the-art.

— Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback  (2609.39456 - Park et al., 30 Sep 2026) in Section 4.1, Accuracy Results