OmniTrace: Cross-Modal Attribution

• OmniTrace is a generation-time attribution framework that dynamically traces token contributions from text, image, audio, and video inputs during autoregressive decoding.
• It aggregates raw attention, mean-pooled attention, and gradient signals into span-level cross-modal explanations without requiring retraining or supervision.
• The framework improves stability and interpretability in multimodal tasks by enforcing temporal coherence and structured curation during output generation.

Searching arXiv for the specified paper and closely related attribution work to ground the article. OmniTrace is a generation-time attribution framework for decoder-only omni-modal LLMs that process interleaved text, image, audio, and video inputs and generate open-ended responses autoregressively. It is formulated to address the problem of identifying which input sources support each generated statement in settings where existing attribution methods are primarily oriented toward classification, fixed prediction targets, or single-modality architectures. The framework treats attribution as a structured tracing problem over the causal decoding process, converts arbitrary token-level signals into span-level cross-modal explanations, and operates without retraining or supervision by reading attention- or gradient-derived signals during generation (Yan et al., 20 Mar 2026).

1. Problem setting and formalization

OmniTrace assumes a decoder-only omni-modal LLM that receives a single interleaved token sequence

$x = (x_1,\ldots,x_n),$

where each $x_i$ may be a text token, an image patch embedding, an audio frame embedding, or a video frame embedding. The input sequence is partitioned into $m$ contiguous source units

$\mathcal{S} = \{S_1,\ldots,S_m\},$

such as text spans, image regions, audio time intervals, or video shot intervals. The model then generates output tokens $y_1,\ldots,y_T$ autoregressively according to

$$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$

At each decoding step $t$, the framework extracts a token-level attribution signal

$a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$

which scores the influence of each context token on the generation of $y_t$. The paper lists three concrete instances: raw attention weight $\alpha_{i\to t}$ in the final layer, mean-pooled attention $x_i$0 across all layers and heads, and a gradient-based score

$x_i$1

The central objective is to map each generated token $x_i$2 back to one, or a small set, of input source units $x_i$3, and then aggregate token-level assignments into span-level attributions for semantically coherent output chunks. This formulation is specific to open-ended multimodal generation rather than fixed-label prediction, and it places the attribution procedure directly inside the decoding process rather than only after generation (Yan et al., 20 Mar 2026).

2. Projection from token-level signals to source units

The first transformation in OmniTrace is token-to-span projection. Given a per-token attribution signal $x_i$4, the framework aggregates the signal over each source unit:

$x_i$5

The generated token $x_i$6 is then assigned to its top-scoring source unit

$x_i$7

with an associated confidence

$x_i$8

The framework also permits cumulative or time-weighted aggregation, which can be used for visualization or debugging. The running attribution mass for source unit $x_i$9 up to time $m$0 is defined as

$m$1

where $m$2 may be a temporal decay or a uniform weight. In gradient-based form, the same quantity may be written as

$m$3

This design yields a unified protocol: arbitrary token-level signals are converted into source-unit scores with identical downstream handling. The abstraction is important because the framework does not depend on a single attribution mechanism. Raw attention, mean attention, gradient-based scores, and, as the paper explicitly states, LRP or future methods can be used as long as they produce a vector $m$4 (Yan et al., 20 Mar 2026).

3. Span-level curation and temporal coherence

After generation completes, OmniTrace segments the output sequence $m$5 into $m$6 semantically coherent chunks $m$7, for example by simple sentence splitting or a light parser. For each chunk $m$8, with token index set $m$9, the framework collects a source-ID sequence

$\mathcal{S} = \{S_1,\ldots,S_m\},$0

a confidence sequence

$\mathcal{S} = \{S_1,\ldots,S_m\},$1

and optionally a POS-tag sequence

$\mathcal{S} = \{S_1,\ldots,S_m\},$2

These sequences are passed to a curation function, CurateWithConf, which selects a minimal supporting set $\mathcal{S} = \{S_1,\ldots,S_m\},$3 whose normalized mass covers at least a fraction $\mathcal{S} = \{S_1,\ldots,S_m\},$4 of the total vote. The curation routine proceeds by computing per-token votes

$\mathcal{S} = \{S_1,\ldots,S_m\},$5

normalizing them into

$\mathcal{S} = \{S_1,\ldots,S_m\},$6

computing a run-length statistic $\mathcal{S} = \{S_1,\ldots,S_m\},$7 from the largest contiguous block of tokens assigned to $\mathcal{S} = \{S_1,\ldots,S_m\},$8, and then scoring each source unit using

$\mathcal{S} = \{S_1,\ldots,S_m\},$9

The top spans are chosen until cumulative $y_1,\ldots,y_T$0 reaches the coverage threshold $y_1,\ldots,y_T$1, while a strong single $y_1,\ldots,y_T$2 is allowed to override.

Temporal coherence enters through the run-length term. In the paper’s formulation, this is one of the mechanisms that turns token-level traces into concise span-level explanations rather than isolated per-token saliency assignments. The ablation results further indicate that POS weights, confidence-shaping, run-coherence, and $y_1,\ldots,y_T$3 filtering are all critical, with particular importance for stabilizing image attributions: image F1 collapses to approximately $y_1,\ldots,y_T$4 without these components (Yan et al., 20 Mar 2026).

4. Execution properties and model-agnostic scope

OmniTrace is described as lightweight and model-agnostic. The paper states that it runs entirely at generation time, requires no parameter updates, and introduces no supervised training stage. The only learning-like component is the fixed-heuristic curation routine, which does not require ground-truth annotations. In operational terms, the framework performs a single forward pass per generated token, with an optional backward pass when the chosen attribution signal $y_1,\ldots,y_T$5 is gradient-based.

Its computational complexity is given as $y_1,\ldots,y_T$6 per generation when $y_1,\ldots,y_T$7 is attention-only, and $y_1,\ldots,y_T$8 when gradients are used. The reported practical cost is approximately $y_1,\ldots,y_T$9–$$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$0 overhead relative to greedy decoding on modern GPUs. This suggests that the framework is intended to be integrated into standard autoregressive inference pipelines rather than into a separate post hoc analysis stage.

The framework’s algorithmic pipeline can be summarized in four stages. First, prompt and multimodal inputs are converted into the token sequence $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$1, and source units $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$2 are built. Second, decoding proceeds token by token while extracting an attribution signal with $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$3. Third, token-level signals are aggregated over source units to compute $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$4, the top source $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$5, and confidence $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$6. Fourth, the completed output is chunked into semantically coherent spans, and CurateWithConf produces the final support set for each chunk. The paper characterizes this as a plug-in procedure for any decoder-only LLM that exposes attention or gradient hooks during generation (Yan et al., 20 Mar 2026).

5. Experimental design and quantitative findings

The paper instantiates OmniTrace on two open-source omni-modal LLMs: Qwen2.5-Omni-7B and MiniCPM-o-4.5-9B. The evaluation covers four modalities through text+image QA, text+image summarization, audio QA/summarization, and video QA, over 759 total examples. Ground-truth attributions are generated by prompting GPT-5.2 and Gemini-3, and 26.6% of them are double-checked by humans with 88.2% agreement. Evaluation uses span-level F1 for discrete spans in text and image settings, and Time-F1 with 1 s bins for audio and video intervals (Yan et al., 20 Mar 2026).

On Qwen2.5-Omni, the mean-pooled attention instantiation OT_AttMean is reported to achieve text-summ F1 = 75.7, compared with self-attr 9.3 and embed 17.3; image-summ F1 = 76.6, compared with self-attr 40.6 and embed 14.6; image-QA = 56.6, compared with 61.0 for self-attr, though the paper notes that self-attr is semantically meaningless here; audio summarization Time-F1 = 83.1, compared with 4.4 for self-attr; audio QA Time-F1 = 49.9, compared with 29.0; and video QA Time-F1 = 40.2, compared with 13.7. On MiniCPM-o-4.5, OT_RawAtt generally performs best, including image QA F1 = 45.4 versus self 39.4, audio QA 49.2 versus self 0.08, and video QA 21.6 versus self 18.3.

Model / variant	Reported tasks	Reported result examples
Qwen2.5-Omni + OT_AttMean	text summarization, image summarization, image QA, audio summarization, audio QA, video QA	75.7 text-summ F1; 76.6 image-summ F1; 83.1 audio summarization Time-F1
MiniCPM-o-4.5 + OT_RawAtt	image QA, audio QA, video QA	45.4 image QA F1; 49.2 audio QA; 21.6 video QA
Baselines	self-attr, embed	examples include 9.3 text-summ F1 and 4.4 audio summarization Time-F1 for self-attr

The paper concludes from these experiments that generation-aware span-level attribution is more stable and interpretable than naive self-attribution and embedding-based baselines, while remaining robust across multiple underlying attribution signals. A plausible implication is that the main empirical advantage is not tied exclusively to a single saliency primitive, but to the structured generation-time tracing and curation procedure applied to that primitive (Yan et al., 20 Mar 2026).

6. Illustrative cases, interpretation, and points of clarification

A stylized example in the paper combines text, image, audio, and video inputs. The source units are defined as $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$7: text span “The weather is sunny.”; $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$8: an image region depicting a man in a red shirt; $$P(y_1,\ldots,y_T \mid x) = \prod_{t=1}^T P(y_t \mid x, y_{<t}).$$9: audio timestamp $t$0 containing “Good morning”; and $t$1: video timestamp $t$2 showing a man waving. The generated output is divided into two chunks: $t$3, “The man in the red shirt says ‘Good morning.’”, and $t$4, “He waves at the camera.”

In the described token-level trace, the tokens “man,” “red,” and “shirt” concentrate on $t$5; “says” mixes $t$6 and $t$7; and “Good” and “morning” concentrate on $t$8, with $t$9 peaking at 0.85 for $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$0 when emitting “morning.” After chunk $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$1 is completed, the sequence of source assignments is summarized as $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$2 with confidences approximately $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$3, and CurateWithConf selects $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$4. For $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$5, tokens attend to video frames in $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$6, yielding $a_t(i) \ge 0 \quad \text{for } i=1\ldots(n+t-1),$7 (Yan et al., 20 Mar 2026).

This example clarifies a common misconception in multimodal attribution. A token-level mechanism that simply points to a high-scoring context position is not, by itself, a span-level explanation for a generated statement. OmniTrace instead aggregates over predefined source units, tracks confidence over time, and enforces temporal coherence before returning a support set for a semantic chunk. The paper’s discussion of image-QA further underscores that a numerically competitive baseline can still be “semantically meaningless” in the intended explanatory sense. Another point of clarification concerns supervision: although the evaluation uses attributed references generated by GPT-5.2 and Gemini-3 with partial human checking, the framework itself is not trained on those annotations and does not require them for inference.

The paper positions these case studies, including those in Figures 1 and 4 and Appendix A, as demonstrations that the method can highlight the image region, audio clip, or video segment underlying each semantic unit of the output. This suggests a view of attribution not as a post hoc score attached to the final response, but as a dynamic record of which multimodal sources supported each part of the generation process (Yan et al., 20 Mar 2026).