CauCo: Causal Coherence Metric

• CauCo is a metric designed to assess whether a natural-language causal chain forms a coherent cause–effect progression rather than just displaying lexical overlap.
• It employs a fine-tuned evaluator LLM to generate binary or probabilistic judgments, ensuring structured evaluation of relational validity within chains.
• The metric emphasizes local cause–effect alignment, chain completeness, and actor consistency, making it crucial for improved causal video question answering.

Searching arXiv for the cited paper and closely related context. arXiv search query: "(Parmar et al., 28 Aug 2025) ChainReaction CauCo causal coherence score" CauCo, short for Causal Coherence Score, is an evaluation metric for causality-oriented captioning introduced in the context of causal video question answering. It is designed to assess whether a natural-language causal chain—a sequence of short event descriptions intended to encode step-by-step reasoning—forms a causally coherent progression rather than merely exhibiting lexical or semantic overlap with a reference. In the formulation described in "ChainReaction! Structured Approach with Causal Chains as Intermediate Representations for Improved and Explainable Causal Video Question Answering" (Parmar et al., 28 Aug 2025), CauCo is implemented through a fine-tuned evaluator LLM that answers whether a candidate chain is causally coherent, yielding either a binary judgment or a probability-valued score.

1. Concept and motivation

CauCo was proposed in response to a specific deficiency of standard captioning metrics. BLEU, METEOR, ROUGE, CIDEr, and SPICE measure n-gram overlap, semantic similarity, or scene-graph propositional content, but they do not test whether a sequence of events actually encodes causal relations such as “cause $\rightarrow$ effect $\rightarrow$ next effect.” In settings where models generate explicit causal chains, this omission becomes consequential, because a chain may be lexically plausible while remaining causally invalid (Parmar et al., 28 Aug 2025).

The metric therefore targets causal coherence rather than surface correspondence. The central object of evaluation is a chain of events expressed in natural language, and the key question is whether each link in the chain holds up as a cause–effect relationship. In this sense, CauCo is not a generic caption-quality metric; it is specialized for outputs whose intended semantics is explicitly relational and directional.

This design places CauCo within a broader shift from overlap-based evaluation toward structure-sensitive evaluation. A plausible implication is that it is most informative when the generated output is itself structured as an ordered reasoning trace rather than as a free-form summary sentence.

2. Formalization

The metric is defined on a natural-language causal chain

$$C = (e_1 \rightarrow e_2 \rightarrow \dots \rightarrow e_k),$$

where the $e_i$ are events. Let

$$f_{\mathrm{eval}} : \mathrm{Chain} \rightarrow [0,1]$$

denote the evaluator LLM’s output probability that the chain is coherent. The per-chain score is then defined as

$\mathrm{CauCo}(C) = f_{\mathrm{eval}}(C).$

If the evaluator returns a hard True/False label rather than a probability, the metric is discretized as

$$\mathrm{CauCo}(C)= \begin{cases} 1,& \text{if evaluator says True} \ 0,& \text{if evaluator says False.} \end{cases}$$

For a dataset containing $N$ predicted chains $\{C_i\}_{i=1}^N$, the aggregate score is the arithmetic mean:

$$\text{CauCo}_{\text{avg}} = \frac{1}{N} \sum_{i=1}^{N} \mathrm{CauCo}(C_i) = \frac{1}{N} \sum_{i=1}^{N} f_{\mathrm{eval}}(C_i).$$

This formulation makes CauCo a learned evaluator over complete causal chains rather than a token-level or pairwise symbolic metric. It is therefore holistic: it assigns a coherence value to the chain as a whole.

3. Computation pipeline

CauCo is computed through a supervised evaluator-training procedure followed by inference on model-generated chains. The procedure has four stages (Parmar et al., 28 Aug 2025).

First, a fine-tuning dataset is assembled from gold causal chains obtained from human-verified annotations. Negative examples are then constructed by perturbing these gold chains using one of six operations: actor swapping, event negation, event removal, event order reversal, semantic modification, and chain shuffling. Gold chains are labeled positive $\rightarrow$0 and perturbed chains negative $\rightarrow$1.

Second, an evaluator LLM is fine-tuned using standard supervised learning with cross-entropy loss on chain–label pairs. The output can be either a two-way classifier (Coherent vs. Incoherent) or a single-sigmoid probability.

Third, at evaluation time, a predicted chain is passed to the fine-tuned evaluator through a prompt of the form: “Given the chain of events [ … ], is it causally coherent? Answer True or False.” The evaluator returns either a hard label or a probability $\rightarrow$2, and the per-chain CauCo score is set accordingly.

Fourth, scores are averaged over the test set to obtain $\rightarrow$3.

The implementation details specified for CauCo include a chain length limit of $\rightarrow$4 events, a negative-positive ratio such as $\rightarrow$5 or $\rightarrow$6, and a classification threshold $\rightarrow$7 when probabilities are binarized. The evaluator LLM backbone may be GPT-4o, Gemini 2.5, or a LLaMA-based model. Learning rate, batch size, and number of epochs are stated to follow standard LLM fine-tuning recipes.

4. Causal properties measured

CauCo is intended to capture several distinct properties of causal chains rather than only superficial plausibility. These properties are tied directly to the perturbation strategy used to generate negative examples (Parmar et al., 28 Aug 2025).

Cause–effect alignment is the most immediate target: the evaluator checks whether each adjacent event pair $\rightarrow$8 is a plausible cause–effect relation. This makes the metric sensitive to local relational validity.

Chain completeness is tested indirectly through event-removal negatives. Chains that omit crucial intermediate steps may be judged incoherent even if the remaining events are individually plausible.

Structural integrity is probed through event-order reversal, semantic modification, and chain shuffling. These perturbations target chains whose constituent events may all be reasonable in isolation but whose ordering or semantic continuity is defective.

Actor consistency is probed through actor swapping. This addresses whether the same agent is consistently tracked through the chain, which is often necessary for preserving causal interpretation.

Taken together, these properties make CauCo a metric of structured relational validity. A plausible implication is that it is especially useful when failure modes arise from missing links, temporal disorder, or agent mismatch rather than from lexical mismatch alone.

5. Empirical behavior

The reported empirical evaluation uses three base VideoQA datasets augmented with gold causal chains: NextQA, CausalVidQA, and CausalChaos! Baseline metrics include BLEU-1 to BLEU-4, METEOR, ROUGE, and SPICE. For each model, chains are generated on test splits and then scored with CauCo (Parmar et al., 28 Aug 2025).

Across all three datasets, the reported averaged results are as follows:

Model	Standard metrics	CauCo
Qwen-VL2.5 (one-shot)	BLEU-1=0.35, BLEU-4=0.11, METEOR=0.54, ROUGE=0.36, SPICE=0.40	0.75
Ours (fine-tuned CCE)	BLEU-1=0.63, BLEU-4=0.28, METEOR=0.61, ROUGE=0.50, SPICE=0.52	0.89

The paper also reports an out-of-domain transfer setting in which training is performed on CausalChaos! and testing on NextQA. In that setting, VILA zero-shot achieves CauCo=0.42, whereas Ours fine-tuned CCE achieves CauCo=0.84.

These results are presented as evidence that CauCo is sensitive to causal coherence in a way standard captioning metrics are not. The comparative pattern also suggests that improvements in chain-generation quality can be reflected more directly in CauCo than in purely overlap-based metrics.

6. Examples, interpretation, and limitations

The worked examples make the operational semantics of CauCo concrete. In one example, the ground-truth chain is [Person A flips switch] $\rightarrow$9 [Light turns on] $$C = (e_1 \rightarrow e_2 \rightarrow \dots \rightarrow e_k),$$0 [Room becomes visible], whereas the predicted chain is [Person A flips switch] $$C = (e_1 \rightarrow e_2 \rightarrow \dots \rightarrow e_k),$$1 [Light stays off] $$C = (e_1 \rightarrow e_2 \rightarrow \dots \rightarrow e_k),$$2 [Person B enters room]. When prompted with this chain, the evaluator answers False, and the score is CauCo = 0 (Parmar et al., 28 Aug 2025).

In a second example, the predicted chain [Driver sees red light] $$C = (e_1 \rightarrow e_2 \rightarrow \dots \rightarrow e_k),$$3 [Driver applies brake] $$C = (e_1 \rightarrow e_2 \rightarrow \dots \rightarrow e_k),$$4 [Car stops]

is judged True, yielding CauCo = 1. Averaging the two example scores gives

$$C = (e_1 \rightarrow e_2 \rightarrow \dots \rightarrow e_k),$$5

The strengths of CauCo are stated explicitly. It directly measures causal coherence rather than surface similarity, is sensitive to missing, reversed, or semantically altered steps, is easily interpretable, and leverages powerful LLM judgment without hand-crafting symbolic causal rules.

Its limitations are equally explicit. The binary or probabilistic judgment may miss subtle gradations of coherence; the metric depends on quality of fine-tuning data, especially the coverage of perturbation types; as an LLM-based evaluator it inherits biases and occasional hallucinations; and it does not measure finer-grained alignment, such as identifying which specific link in the chain is broken. The evaluation is therefore holistic rather than diagnostic.

7. Position within causal video question answering

CauCo is introduced as part of a broader modular framework for Causal-Why Video Question Answering in which causal chains serve as interpretable intermediate representations between video understanding and answer generation (Parmar et al., 28 Aug 2025). Within that framework, CauCo functions as the metric aligned to the intermediate representation itself rather than to the final answer alone.

This placement is significant because the framework distinguishes between a Causal Chain Extractor (CCE) and a Causal Chain-Driven Answerer (CCDA). CauCo evaluates the coherence of the generated chains that mediate between these components. This suggests that the metric is not only an assessment tool but also a mechanism for validating the quality of an explicit reasoning substrate.

In methodological terms, CauCo belongs to a family of learned evaluators that replace lexical overlap with model-based judgments over structured outputs. Its defining feature, however, is its focus on ordered causal chains and on perturbation-based supervision targeted at causal failure modes. A plausible implication is that CauCo is best understood not as a universal replacement for BLEU-, ROUGE-, or SPICE-style measures, but as a complementary metric for tasks in which causal structure is the primary object of evaluation.