Temporal Clip Banzhaf Interaction

• The paper introduces temporal clip Banzhaf interaction as a coalition-based mechanism for fine-grained, interpretable clip–phrase alignment in multimodal settings.
• It leverages hierarchical token merging via DPC-KNN clustering to construct dynamic clips without relying on manual temporal annotations.
• Empirical evaluations on video-language and text-to-motion tasks show improved retrieval accuracy and reduced redundancy compared to global contrastive methods.

Searching arXiv for the specified papers and closely related work on Temporal Clip Banzhaf Interaction / Hierarchical Banzhaf Interaction. Temporal Clip Banzhaf Interaction denotes a clip-level, coalition-based alignment mechanism in which temporally grouped visual or motion units interact with grouped linguistic units under a Banzhaf cooperative-game formulation. In the video-language literature, it is the action-level stage of Hierarchical Banzhaf Interaction (HBI), where merged video clips interact with merged textual phrases rather than only individual frames and words (Jin et al., 2023, Jin et al., 2024). In text-to-motion generation, the term is used explicitly as Temporal Clip Banzhaf Interaction (TCBI), where text clips and motion clips are treated as the players in a cooperative game and ranked by their semantic cooperation (Wang et al., 9 Jul 2025). Across these settings, the central objective is the same: replace coarse global matching with fine-grained temporal coalition reasoning that can capture clip–phrase or clip–clip correspondence, reduce redundancy, and provide a more interpretable interaction signal than global contrastive similarity alone.

1. Conceptual origin and scope

The immediate background is contrastive multimodal representation learning, especially CLIP-style video-language learning, in which supervision is typically derived from global semantic similarity between paired samples (Jin et al., 2023). The HBI line argues that such coarse alignment cannot capture entity-level matching, fails to model clip-phrase correspondence, lacks interpretability, and would otherwise require fine-grained labels that are unavailable (Jin et al., 2024). Temporal Clip Banzhaf Interaction addresses this by treating local temporal units as cooperative players whose joint contribution can be valued explicitly.

In the original HBI formulation, the method begins from frame-level video tokens and word-level text tokens and then stacks token merge modules to obtain a hierarchy with three semantic levels: entity level, action level, and event level (Jin et al., 2024). Temporal Clip Banzhaf Interaction is therefore not a separate standalone module in that framework; it is the action-level interaction among clip tokens and phrase tokens. The paper states: “By stacking token merge modules, HBI achieves hierarchical interaction, encompassing entity-level interactions among frames and words, action-level interactions among clips and phrases, and event-level interactions among segments and paragraphs” (Jin et al., 2024).

The later MOST framework adopts the same basic game-theoretic principle in a different domain: rare text-to-motion generation (Wang et al., 9 Jul 2025). There, the novelty is formulated directly as Temporal Clip Banzhaf Interaction in the retrieval stage, where text descriptions are matched to motion clips rather than to full motion sequences. This suggests a broader interpretation of the concept: it is a modality-agnostic mechanism for turning temporal alignment into a coalition valuation problem at clip granularity.

2. Cooperative-game-theoretic formulation

Both HBI and MOST derive Temporal Clip Banzhaf Interaction from multivariate cooperative game theory. In HBI, the set of players is defined over frame and word tokens:

$$\mathcal{N}=\{v^{i}_{f}\}^{N_v}_{i=1}\cup \{t^{j}_{w}\}^{N_t}_{j=1}.$$

The characteristic function $\phi$ maps a coalition of players to a payoff, and for fine-grained alignment the payoff is taken to be the cross-modality similarity measurement $S$ (Jin et al., 2024).

For a coalition $\{i,j\}$, the Banzhaf Interaction is defined as

$$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$

with $p(\mathcal{C})=\frac{1}{2^{n-2}}$ (Jin et al., 2023, Jin et al., 2024). The interpretation given in the papers is that a high interaction score means the coalition contributes more than the sum of isolated players, so the pair cooperates strongly.

In HBI, the interaction scores are turned into distributions over pairwise relations by softmax normalization, for example

$$p_{i,j}^{\mathcal{I}}=\frac{\exp\bigl(\mathcal{I}([\{v^{i}_{f}, t^{j}_{w}\}])\bigr)}{\sum_{k=1}^{N_t} \exp\bigl(\mathcal{I}([\{v^{i}_{f}, t^{k}_{w}\}])\bigr)},$$

with an analogous text-to-video normalization (Jin et al., 2024). Training aligns these interaction-induced distributions with predicted fine-grained relation distributions through a KL-divergence-based Banzhaf loss (Jin et al., 2023).

MOST uses the same cooperative principle but changes the players. Its player set is formed by text clips and motion clips rather than frame and word tokens:

$N=\{m^i\}_{i=1}^{T_s}\cup\{t^j\}_{j=1}^{T_s}.$

The paper defines the Banzhaf value

$$B(i|N) = \sum_{S \subseteq N \backslash \{i\}} \rho(S)\big(v(S \cup \{i\}) - v(S)\big),$$

and the Banzhaf interaction

$$I(\{i,j\}) = \sum_{S \subseteq N \backslash \{i,j\}} \rho(S)\left[ v(S \cup \{i,j\}) + v(S) - v(S \cup \{i\}) - v(S \cup \{j\}) \right],$$

with $\phi$0 (Wang et al., 9 Jul 2025). In that formulation, a larger $\phi$1 again means stronger cooperation than expected from independent contributions.

A common misconception is to reduce this mechanism to ordinary attention. The papers distinguish them explicitly: Banzhaf Interaction is not just “where to attend,” but a coalition payoff that reflects how much the interaction between players contributes beyond isolated appearances (Jin et al., 2024). This is a materially different semantic object from an attention weight.

3. Temporal clip construction and hierarchical token merging

In HBI, clip-level interaction emerges by hierarchically merging lower-level tokens rather than by relying on hand-labeled temporal boundaries (Jin et al., 2024). At the lowest level, the model starts with frame-level video tokens and word-level text tokens:

$\phi$2

Video frames are evenly sampled; each sampled frame is encoded by CLIP/ViT and then aggregated by a 4-layer transformer (Jin et al., 2024).

Temporal clips are obtained through a token merge module using DPC-KNN clustering. For the visual stream, the model first applies a 1D convolution to strengthen temporal information, then computes a local density

$\phi$3

and a distance index

$\phi$4

Tokens with high $\phi$5 become cluster centers, and the remaining tokens are assigned to the nearest center (Jin et al., 2024). Cluster representatives are weighted averages with weight

$\phi$6

and the resulting merged tokens are used as queries in attention while original tokens remain keys and values (Jin et al., 2024).

This process yields the hierarchical structure central to HBI:

• Entity level: frame tokens and word tokens
• Action level: clip tokens and phrase tokens
• Event level: segment tokens and paragraph tokens

At the action level, coalitions $\phi$7 and $\phi$8 are first obtained from $\phi$9 and $S$0 using DPC-KNN, and the cluster representatives then produce action-level visual tokens $S$1 and action-level textual tokens $S$2 (Jin et al., 2024). Temporal Clip Banzhaf Interaction in HBI is precisely the Banzhaf computation carried out on these merged action-level tokens rather than on raw frame and word tokens.

MOST constructs clips differently. It starts from frame-level motion features

$S$3

and token-level text features

$S$4

then downsamples both into clip sequences via a 1D convolutional layer followed by self-attention:

$S$5

with $S$6 and implementation using $S$7 (Wang et al., 9 Jul 2025). The stated motivation is that motion semantics usually appear in short contiguous spans, not in isolated frames.

A further misconception is that “temporal clips” necessarily correspond to explicit human-defined intervals. In HBI they do not: the clip-level unit is a cluster of adjacent or semantically related frame tokens produced by clustering plus attention-based merging (Jin et al., 2024). In MOST, the clips are downsampled temporal units produced by convolution and self-attention rather than manual annotations (Wang et al., 9 Jul 2025).

4. Representation reconstruction, normalization, and training objectives

In HBI, the clip-level stage does not rely only on single-modal merged tokens. After obtaining action-level single-modal tokens, the model reconstructs them into mixed single-modal and cross-modal representations:

$S$8

$S$9

The coefficients are learned by MLPs:

$\{i,j\}$0

The paper states that this reconstruction preserves the fine granularity of clip tokens while also injecting query-adaptive cross-modal information, which helps the Banzhaf calculation become less biased (Jin et al., 2024).

The training scheme in HBI is hierarchical. At the semantic levels, the objectives are

$\{i,j\}$1

In the earlier HBI paper, the stacked levels are denoted entity, action, and event, with total loss

$\{i,j\}$2

for deep supervision and self-distillation from lower to higher levels (Jin et al., 2023). The action-level term $\{i,j\}$3 is the temporal clip interaction loss.

The original HBI paper also notes a computational issue: exact Banzhaf computation is NP-hard, and the candidate-set count is $\{i,j\}$4 (Jin et al., 2023). To make the method practical, it pre-trains a tiny estimator to approximate the interaction from the similarity matrix of video frames and text tokens; the estimator uses two CNN layers and a self-attention layer (Jin et al., 2023). This is an important implementation fact because clip-level coalition reasoning otherwise remains combinatorially expensive.

MOST also separates coarse and fine objectives. Before clip-level Banzhaf refinement, it computes a conventional text-motion alignment matrix

$\{i,j\}$5

and a coarse entity-level similarity

$\{i,j\}$6

It then defines a generalization degree ratio

$\{i,j\}$7

and forms a weighted revenue score $\{i,j\}$8 from $\{i,j\}$9 to emphasize motion richness and informativeness (Wang et al., 9 Jul 2025). The retrieval stage combines NCE-based overall alignment with a Banzhaf interaction predictor, and the formal retrieval loss aligns predicted interaction distributions with ground-truth Banzhaf interactions (Wang et al., 9 Jul 2025).

5. Functional role in multimodal alignment and retrieval

The primary problem Temporal Clip Banzhaf Interaction is designed to solve is the mismatch between local semantics and global supervision. In video-language learning, a short visual action may correspond to a phrase rather than to an entire sentence, and whole-video contrastive matching does not expose that structure (Jin et al., 2023). In motion generation, a text prompt may refer to only a small sub-action inside a long motion sequence, while the rest of the sequence introduces redundancy (Wang et al., 9 Jul 2025).

Within HBI, clip-level interaction is the mechanism that moves the model from raw frame–word correspondence to a more semantically coherent clip–phrase correspondence (Jin et al., 2024). The papers describe this as fine-grained, hierarchical, explainable, and less dependent on human annotations. A plausible implication is that the action level serves as the scale at which compositional temporal semantics become stable enough to support both retrieval and interpretation.

Within MOST, the retrieval pipeline makes this role explicit. The process is:

1. overall alignment with encoded text and motion features using NCE loss,
2. top entity retrieval using the similarity score $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$0, with implementation using $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$1,
3. clip segmentation with $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$2,
4. interaction prediction for each clip pair,
5. ranking by interaction score,
6. selection of the top $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$3 cooperative motion clips, with implementation using $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$4, and
7. extraction of the corresponding contiguous motion frames from the original sequence (Wang et al., 9 Jul 2025).

The generation stage then reuses the selected clips as compact prompts. MOST conditions a diffusion model on both text prompt $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$5 and retrieved motion references $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$6, and forms each prompt clip as a Banzhaf-weighted soft aggregation:

$$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$7

The motion transformer partitions noisy motion into clips and lets each clip attend to its corresponding prompt motion while also attending to text prompts (Wang et al., 9 Jul 2025). The significance claimed in the paper is that the generator receives semantically aligned clip prompts rather than a long, redundant reference sequence.

6. Empirical evidence, interpretability, and limitations

The empirical evidence for clip-level Banzhaf interaction appears in both the original HBI line and MOST. In the 2023 HBI paper, retrieval results are reported on MSRVTT, ActivityNet Captions, and DiDeMo, and VideoQA results are reported on MSRVTT-QA (Jin et al., 2023). The paper states that on MSRVTT text-to-video retrieval HBI reaches 48.6 R@1; on DiDeMo it reaches 46.9 R@1 for text-to-video and 46.2 R@1 for video-to-text; and on MSRVTT-QA it improves answer accuracy to 46.2% (Jin et al., 2023). The same paper reports that the tiny estimator reduces training cost dramatically relative to sampling-based Banzhaf computation while preserving accuracy, and that ablations show the Banzhaf interaction loss itself gives a clear boost (Jin et al., 2023).

The later HBI paper extends the framework into an encoder-decoder setting for retrieval, VideoQA, and captioning, and states that the full HBI V2 model improves over the previous HBI and prior methods on all three tasks (Jin et al., 2024). Although it does not isolate “clip interaction only” in a single table row, the paper reports that adding Banzhaf Interaction improves the baseline retrieval and QA results; that deep supervision and self-distillation further help; that cluster-number ablation shows hierarchical token merging is meaningful and performance-sensitive; and that visualization of hierarchical interaction shows coalitions at the action level produce higher semantic similarity than individual frame-word pairs (Jin et al., 2024). The example given is a word coalition such as $$\mathcal{I}([\{i,j\}])=\sum_{\mathcal{C} \subseteq \mathcal{N} \setminus \{i,j\}} p(\mathcal{C}) \Bigl[\phi(\mathcal{C}\cup \{[\{i,j\}]\})+\phi(\mathcal{C})-\phi(\mathcal{C}\cup\{i\})-\phi(\mathcal{C}\cup\{j\})\Bigr],$$8 aligning with a video coalition representing a man driving (Jin et al., 2024).

MOST provides a more direct ablation on the role of clip-level Banzhaf interaction in rare text-to-motion settings. On HumanML3D retrieval, the paper reports:

• MOST w/o Banzhaf: R@1 = 5.59, MedR = 29
• MOST: R@1 = 6.61, MedR = 25

It also reports rare-text generalization improvements on Tail 0–5%, Tail 0–15%, and Tail 0–25%, including HumanML3D FID comparisons such as ReMoDiffuse 0.87 vs. MOST 0.66 on Tail 0–5%, 0.43 vs. 0.34 on Tail 0–15%, and 0.39 vs. 0.19 on Tail 0–25% (Wang et al., 9 Jul 2025). In a motion redundancy experiment, the paper states that ReMoDiffuse uses about 196 frames, MOST uses only 39 frames, and yet MOST achieves higher average cosine similarity to ground truth motion prompts (Wang et al., 9 Jul 2025). The ablations further state that too few clips fragment semantics, too many clips reintroduce redundancy, and 5 clips is a good tradeoff (Wang et al., 9 Jul 2025).

Interpretability is a recurring claim. HBI visualizes hierarchical cross-modal links with different confidence colors, ranging from red to blue, across entity, action, and event levels (Jin et al., 2023). The qualitative examples show that coalitions such as “two men talking” align more strongly with a temporal clip depicting the men talking than any single word or single frame alone (Jin et al., 2023). This supports the paper’s claim that semantic meaning often emerges only after temporally grouping frames into clips.

The limitations are clearest in MOST. The paper states that the two-stage pipeline is inconvenient, long and detailed prompts remain difficult to segment finely, some generated motions still contain foot sliding and other physical artifacts, and performance depends on sufficient training diversity (Wang et al., 9 Jul 2025). It also suggests a potential future role for LLMs in simplifying rare text understanding and improving retrieval (Wang et al., 9 Jul 2025). In the HBI setting, a plausible limitation is the dependence on cluster design and hierarchy specification, since the cluster-number ablation is reported to be performance-sensitive (Jin et al., 2024).

Temporal Clip Banzhaf Interaction is therefore best understood as a fine-grained temporal alignment paradigm rather than a single architecture-specific block. In HBI, it is the action-level interaction between merged clips and phrases inside a hierarchical video-LLM (Jin et al., 2023, Jin et al., 2024). In MOST, it becomes an explicit clip-level retrieval mechanism for rare text-to-motion generation (Wang et al., 9 Jul 2025). In both cases, the defining feature is coalition reasoning over temporal clips: a clip pair is important not merely because the two units are similar, but because their joint participation contributes more than their isolated appearances would predict.