M3S: Multi-Source Signal Supervision

• M3S is a mechanism that maps noisy, partial, multi-granular causal supervision onto BridgeVLM's internal causal interface, producing Causal Tokens for decoding.
• It integrates auxiliary objectives—edge supervision, description alignment via symmetric InfoNCE, and DAG regularization—to explicitly train latent causal computation.
• Empirical results demonstrate significant improvements in intervention and counterfactual reasoning performance versus prompt-based causal supervision methods.

M3S, short for Multi-Source Signal Supervision, is a unified training interface introduced within BridgeVLM for multi-image causal reasoning in vision–LLMs. Rather than injecting causal knowledge as prompts or rationales, M3S converts available causal supervision into direct constraints on the internal causal interface of the model: the induced routing DAG over latent variables and the resulting Causal Tokens attended by the decoder. In BridgeVLM, M3S therefore marks a shift from supervising causal reasoning through text alone to supervising the model’s internal causal state, with the stated aim of making causal supervision operational where causal reasoning actually happens (Yu et al., 10 Jun 2026).

1. Definition and conceptual role

M3S is defined as a mechanism that takes causal supervision that may be noisy, partial, and at heterogeneous granularities and maps it onto BridgeVLM’s internal representations. The two principal targets of this supervision are the induced routing DAG over latent variables and the Causal Tokens—that is, the node and graph tokens consumed by the decoder. In the BridgeVLM pipeline, multi-image visual tokens are first mapped to latent variable features, then to an induced routing DAG with edge probabilities, then through RAMP into Node Tokens, after which node tokens and graph tokens are combined into Causal Tokens; the decoder finally attends to $$[\text{visual tokens}; \text{Causal Tokens}; \text{text}]$$ and is trained with language modeling loss (Yu et al., 10 Jun 2026).

This design is presented as fundamentally different from prompt-based causal supervision. In that earlier pattern, causal knowledge is represented only as text, the LVLM must voluntarily ground that text in hidden states, and supervision cannot selectively target particular nodes, edges, or latent states. M3S instead adds auxiliary losses directly on internal structure and semantics. This suggests that, in BridgeVLM, causal supervision is treated not as explanatory side information but as an explicit training signal for latent causal computation.

2. Internal causal interface and placement in BridgeVLM

BridgeVLM’s causal interface consists of latent variable features, a routing DAG, node tokens, graph tokens, and their concatenation into Causal Tokens. The variable features are

$\mathbf{Q} \in \mathbb{R}^{D \times H},$

and edge probabilities are induced as

$$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$

The induced DAG is then used by RAMP (Route-Aware Message Propagation). Starting from

$\mathbf{H}^{(0)} = \mathbf{Q},$

the adjacency is normalized as

$\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$

For each layer $\ell$,

$$\mathbf{M}^{(\downarrow)} = \bar{\mathbf{P}}\,\mathbf{H}^{(\ell-1)}\mathbf{W}^{(\ell)}_{\downarrow}, \qquad \mathbf{M}^{(\uparrow)} = \bar{\mathbf{P}}^{\top}\mathbf{H}^{(\ell-1)}\mathbf{W}^{(\ell)}_{\uparrow},$$

$$\mathbf{H'}^{(\ell)} = \mathrm{LN}\big(\mathbf{H}^{(\ell-1)} + \mathbf{M}^{(\downarrow)} + \mathbf{M}^{(\uparrow)}\big),$$

$$\mathbf{H}^{(\ell)} = \mathrm{LN}\big(\mathbf{H'}^{(\ell)} + \mathrm{FFN}^{(\ell)}(\mathbf{H'}^{(\ell)})\big).$$

After $L_p$ layers,

$\mathbf{Q} \in \mathbb{R}^{D \times H},$0

Graph tokens aggregate global context and feed it back to nodes: $\mathbf{Q} \in \mathbb{R}^{D \times H},$1 The resulting Causal Tokens are

$\mathbf{Q} \in \mathbb{R}^{D \times H},$2

and the decoder input is

$\mathbf{Q} \in \mathbb{R}^{D \times H},$3

M3S sits on top of this entire pipeline. It constrains $\mathbf{Q} \in \mathbb{R}^{D \times H},$4, $\mathbf{Q} \in \mathbb{R}^{D \times H},$5, $\mathbf{Q} \in \mathbb{R}^{D \times H},$6, and $\mathbf{Q} \in \mathbb{R}^{D \times H},$7, thereby shaping both the structure of message passing and the semantics exposed to the decoder. A plausible implication is that M3S treats causal reasoning as a latent computational substrate rather than as a textual accompaniment to generation.

3. Supervision sources, granularity, and alignment mechanism

M3S is described as a collection of auxiliary objectives that can be selectively turned on depending on what supervision is available for a given example. It unifies two axes of variation. The first is local vs. global supervision: local supervision includes node- or edge-level descriptions, node identities, and local adjacency, whereas global supervision includes full adjacency matrices, acyclicity constraints, and global descriptions. The second is structural vs. semantic supervision: structural supervision concerns “who causes whom,” while semantic supervision concerns what each variable means and what the whole system means (Yu et al., 10 Jun 2026).

The supervision signals it can consume include:

• Causal graph edges / adjacency as a possibly partial binary matrix

$\mathbf{Q} \in \mathbb{R}^{D \times H},$8

• Node-level textual descriptions, such as descriptions of individual variables.
• Global / graph-level textual descriptions, such as high-level explanations of causal structure.
• Implicitly edge-level descriptions, encoded as part of graph-level text embeddings.

Node- and graph-level descriptions are embedded into

$\mathbf{Q} \in \mathbb{R}^{D \times H},$9

and

$$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$0

respectively. Alignment is then performed through symmetric InfoNCE objectives between variable features or node tokens and $$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$1, and between graph tokens and pooled $$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$2.

A central complication is that latent variables are permutation-invariant. M3S therefore uses a Hungarian assignment to match latent variables to annotated node identities before computing local losses. The matching is defined as

$$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$3

with $$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$4 the set of valid nodes, after which

$$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$5

This matching mechanism is what allows heterogeneous supervision to be mapped onto a single latent interface. The paper’s examples illustrate this at both granularities. In the CausalVLBench, Circuit scenario, local descriptions for circuit elements supervise node tokens, while the circuit causal graph and a high-level explanation supervise $$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$6 and $$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$7. In Causal3D, programmatic causal rules can supervise edge structure globally, while node-level supervision could associate specific latent variables with visual entities. Because M3S activates losses conditionally on availability, fully annotated scenes, partially annotated scenes, and samples with no causal supervision can all be trained under the same interface.

4. Objective formulation and training behavior

The base training objective is the autoregressive language modeling loss

$$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$8

M3S adds three kinds of auxiliary terms.

First, when ground-truth adjacency is available, edge supervision directly constrains the induced routing probabilities: $$\tilde{\mathbf{A}} = f_L(\mathbf{Q})\,f_R(\mathbf{Q})^{\top}, \qquad \mathbf{P} = \sigma(\tilde{\mathbf{A}}) \in [0,1]^{D\times D}.$$9 where BCE is applied only over valid variables and observed entries.

Second, description alignment is imposed through symmetric InfoNCE. After Hungarian matching yields permuted node representations $\mathbf{H}^{(0)} = \mathbf{Q},$0,

$\mathbf{H}^{(0)} = \mathbf{Q},$1

$\mathbf{H}^{(0)} = \mathbf{Q},$2

and

$\mathbf{H}^{(0)} = \mathbf{Q},$3

Third, when no oracle graph is available, DAG regularization encourages acyclicity using a NOTEARS-style penalty: $\mathbf{H}^{(0)} = \mathbf{Q},$4

$\mathbf{H}^{(0)} = \mathbf{Q},$5

These terms are combined as

$\mathbf{H}^{(0)} = \mathbf{Q},$6

and the final objective is

$\mathbf{H}^{(0)} = \mathbf{Q},$7

Typical weights reported in the appendix include $\mathbf{H}^{(0)} = \mathbf{Q},$8, $\mathbf{H}^{(0)} = \mathbf{Q},$9, $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$0, and $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$1. The interface is explicitly conditional: terms are activated only when the corresponding supervision is present. The result is a training formulation in which arbitrary supervision sources can be mixed across examples without changing the architecture (Yu et al., 10 Jun 2026).

Implementation details reinforce this interpretation. Images are encoded into $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$2, a fixed number $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$3 of learnable variable queries $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$4 cross-attend to $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$5, and node and graph descriptions are tokenized by the same text encoder as the LVLM, identified as Phi-4-MMI, into pooled embeddings $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$6. During training, the decoder always sees $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$7, while M3S losses are computed on internal states $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$8. At inference, no graphs or descriptions are supplied; the model still executes the same internal causal pipeline. M3S is therefore described as a training-only interface, but the learned internal causal representation is used at test time.

5. Empirical characteristics and ablation findings

The principal empirical claim is that how causal supervision is applied matters substantially. On CausalVLBench intervention tasks across Pendulum, Flow, and Circuit, Phi-4-MMI-7B with prompt-level causal traces achieves 33.2% Avg, whereas BridgeVLM + M3S (NodeAlign, no fancy decoding bias) reaches 54.4% Avg, with category scores of 36.0 on Pendulum, 43.6 on Flow, and 83.7 on Circuit. On Causal3D, prompt-level Phi-4 causal-trace reaches 43.6% intervention average and 81.0% counterfactual accuracy, while BridgeVLM + M3S reaches 49.0% intervention and 92.3% counterfactual (Yu et al., 10 Jun 2026).

The supervision ablation isolates which M3S components matter. GraphPrompt / NodePrompt, both prompt-level variants, offer little gain or even degradation relative to answer-only. GraphAlign alone, corresponding to only edge supervision, yields 31.0 Avg and does not improve performance. By contrast, NodeAlign alone, aligning Causal Tokens to node and global descriptions, raises intervention accuracy to 54.4% Avg. Adding AttBias and GraphAlign on top of NodeAlign pushes performance further, up to 60.2%, but the dominant effect is attributed to NodeAlign. The paper therefore states that local semantic grounding of Causal Tokens is the main driver of performance, while exact structural supervision gives diminishing returns unless node semantics are already aligned.

Structural diagnostics follow the same pattern. On CausalVLBench, directed-edge $\bar{\mathbf{P}} = \mathrm{RowNorm}(\mathbf{P}).$9 between the induced DAG and ground truth is 33.4% for BridgeVLM trained answer-only, 100% for + GraphAlign with oracle adjacency, and 75.1% for + NodeAlign with no graph labels. The text states that higher structural $\ell$0 correlates with better intervention performance across Pendulum, Flow, and Circuit. This suggests that semantic alignment alone can substantially improve the faithfulness of induced causal structure, even without explicit graph supervision.

Cross-benchmark transfer is also reported on CELLO. In zero-shot transfer from CausalVLBench, BridgeVLM reaches 63.1% versus 66.2% for Phi-4 without fine-tuning, which the paper notes as hard OOD behavior. After fine-tuning on CELLO, the Phi-4 baseline reaches 83.7% Avg, whereas BridgeVLM + M3S reaches 90.3% Avg. The reported pattern is therefore not merely in-domain performance gain, but improved adaptation once the causal interface has been trained.

6. Limitations, extensions, and broader positioning

Several limitations are stated explicitly. The induced DAG is described as a latent reasoning scaffold, not guaranteed to recover the true underlying causal graph without strong supervision or assumptions. The benchmarks and training setup assume that test scenarios resemble training scenarios, including the same variable types and meta-structure family, so generalization to completely new causal systems remains open. M3S also leverages relatively rich supervision, such as node or global descriptions and sometimes full graphs, which may be expensive or noisy in real-world settings (Yu et al., 10 Jun 2026).

The paper also outlines possible extensions. These include using weaker supervision, such as only outcome changes under interventions or implicit constraints from dynamics; automating graph induction with external LLMs as graph priors refined through M3S; extending the framework beyond static DAGs to temporal causal structures or continuous-time processes with generalized RAMP; and exploring self-supervised or distillation-based M3S, where internal causal supervision is derived from model-generated traces or teacher models. These are presented as directions that could transform M3S from a method based on multi-source causal annotations into a more self-contained causal learning paradigm.

Within the broader literature, M3S is positioned against approaches that keep causal graphs external, or that inject causal information as textual prompts or rationales. The paper states that many prior methods do not expose a variable-level, tokenized causal interface used directly in decoding, and that causal prompting or causal chain-of-thought can be weakly coupled to actual model reasoning. It further notes that methods such as CausalMM, CDC, and CLIP-ICM use causal ideas mainly for robustness or hallucination mitigation rather than for explicit structural supervision of internal variables for intervention and counterfactual reasoning. What is described as novel about M3S is that it is a first-class supervision interface targeting internal causal structure, supporting heterogeneous, partial supervision under a single loss template, and operationalizing causal supervision by shaping the internal computation that drives predictions.

In this sense, M3S denotes not merely a loss bundle, but a particular stance on causal supervision in LVLMs: the induced DAG and Causal Tokens are treated as the locus of causal reasoning, and supervision is directed there accordingly.