Blockwise Cross-Attention in Multimodal Models

• Blockwise cross-attention is an attention mechanism that organizes interactions into predefined blocks for effective multimodal feature fusion.
• It is implemented either by inserting attention wrappers at specific backbone stages or by partitioning tokens into chunks with causal masking.
• This design reduces computational cost by limiting attention to relevant blocks while preserving full within-chunk interactions and temporal causality.

Searching arXiv for the cited works and closely related terminology to ground the article in the relevant papers. arxiv_search(query="(Chi et al., 2019) Two-Stream Video Classification with Cross-Modality Attention", max_results=5) arxiv_search(query="(Tang et al., 23 Jan 2025) M3PT blockwise attention", max_results=5) A blockwise cross-attention module is an attention mechanism in which interaction structure is organized around explicit blocks, either at the level of network insertion or at the level of token grouping. In the 2019 video-classification formulation, the Cross-Modality Attention (CMA) block is a wrapper that takes a query feature map from one stream and key/value feature maps from another stream, computes cross-attention, and returns a residual output that can be plugged into existing two-stream backbones (Chi et al., 2019). In the 2025 multimodal social-signal formulation, blockwise attention is defined by flattening person $\times$ modality $\times$ time inputs into chunk-major blocks and imposing a closed-form causal mask that permits full intra-chunk cross-attention while forbidding look-ahead to future chunks (Tang et al., 23 Jan 2025). Taken together, these formulations define a family of modules in which cross-attention is constrained or structured by block design rather than treated as an unrestricted dense interaction pattern.

1. Conceptual scope and problem setting

In “Two-Stream Video Classification with Cross-Modality Attention,” the motivating problem is multimodal fusion for video classification. The paper states that “the most popular method up to now is still simply fusing each stream’s prediction scores at the last stage,” and asks whether “there exists a more effective method to fuse information cross modality” (Chi et al., 2019). Its answer is a cross-modality attention operation implemented as a compatible CMA block that can be plugged into many existing architectures. The underlying setting is a standard two-stream backbone with an RGB branch and an optical-flow branch.

In “M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention,” the problem setting is different: multimodal social-signal prediction in multi-party conversations. The model processes multiple social cues across multiple participants and their temporal interactions with a causal transformer architecture using modality and temporal blockwise attention masking (Tang et al., 23 Jan 2025). Inputs are organized by participant, modality, and time-chunk, and the block structure is intrinsic to the sequence representation.

These two uses of blockwise cross-attention are not identical. In CMA, the module is “blockwise” in the sense that attention wrappers are inserted “every other” residual block within selected backbone stages. In M3PT, “blockwise” refers to the partition of the token sequence into time-chunk blocks of size $B=N\cdot K$. A plausible implication is that the term designates a design principle rather than a single canonical operator: block structure may be imposed through architectural placement, through token partitioning, or through both.

2. Cross-modality attention in two-stream video models

The CMA block begins from two feature maps of identical shape. A query feature map is denoted $X\in\mathbb{R}^{N\times C\times T\times H\times W}$ from one stream, and a key/value feature map is denoted $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$ from the other stream (Chi et al., 2019). Internally, the block applies three $1\times1(\times1)$ convolutions:

• $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$, with $d_k=C/2$
• $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$ after $2\times2$ spatial max-pooling
• $\times$0 with the same pooling

After flattening spatiotemporal dimensions, $\times$1 and $\times$2, yielding $\times$3, $\times$4, and $\times$5 (Chi et al., 2019). The attention itself is defined positionwise. Let $\times$6 be the $\times$7-dimensional spatial-temporal feature at position $\times$8 in the query branch, and $\times$9 the $B=N\cdot K$0-dimensional feature at position $B=N\cdot K$1 in the other branch. With linear embeddings $B=N\cdot K$2, $B=N\cdot K$3, and $B=N\cdot K$4 to $B=N\cdot K$5 dimensions, the attended output is

$B=N\cdot K$6

where

$B=N\cdot K$7

and

$B=N\cdot K$8

In compact $B=N\cdot K$9-$X\in\mathbb{R}^{N\times C\times T\times H\times W}$0-$X\in\mathbb{R}^{N\times C\times T\times H\times W}$1 form,

$X\in\mathbb{R}^{N\times C\times T\times H\times W}$2

The final block output is

$X\in\mathbb{R}^{N\times C\times T\times H\times W}$3

where $X\in\mathbb{R}^{N\times C\times T\times H\times W}$4 is another $X\in\mathbb{R}^{N\times C\times T\times H\times W}$5 convolution mapping $X\in\mathbb{R}^{N\times C\times T\times H\times W}$6 back to $X\in\mathbb{R}^{N\times C\times T\times H\times W}$7 channels (Chi et al., 2019). A BatchNorm with scale initialized to $X\in\mathbb{R}^{N\times C\times T\times H\times W}$8 follows $X\in\mathbb{R}^{N\times C\times T\times H\times W}$9 “to stabilize training and start as identity.”

The significance of this construction is explicit in the paper’s comparison with late fusion. Cross-attention is used to obtain information from the other modality “in a more effective way than two-stream,” rather than restricting fusion to final prediction scores (Chi et al., 2019).

3. Blockwise token partitioning and masked cross-attention in M3PT

M3PT defines blockwise attention by first constructing a multimodal, multi-party token sequence. The model observes $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$0 participants and $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$1 modalities, and chunks time into $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$2 non-overlapping segments of equal duration $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$3; in the HHCD experiments, $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$4 and $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$5 (Tang et al., 23 Jan 2025). For each participant $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$6, modality $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$7, and time-chunk $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$8, a pretrained VQ-VAE encoder produces a single $Y\in\mathbb{R}^{N\times C\times T\times H\times W}$9-dimensional codebook embedding $1\times1(\times1)$0.

The collection $1\times1(\times1)$1 is flattened in the order

$1\times1(\times1)$2

so that the full sequence $1\times1(\times1)$3 with $1\times1(\times1)$4 is arranged in “time $1\times1(\times1)$5 person $1\times1(\times1)$6 modality” order (Tang et al., 23 Jan 2025). Three learned positional embeddings are then added: a cyclic temporal-chunk embedding $1\times1(\times1)$7, a participant embedding $1\times1(\times1)$8, and a modality embedding $1\times1(\times1)$9.

This ordering creates natural blocks of size $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$0 corresponding to each time-chunk. Within a given time $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$1, the $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$2 tokens are further logically grouped into $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$3 sub-blocks of size $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$4, one sub-block per person (Tang et al., 23 Jan 2025). The intended attention pattern is then stated explicitly: each token at time $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$5 may “attend freely within its own time-chunk to all other participants’ modalities” and may also “attend to all tokens from earlier time-chunks $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$6,” but may not look ahead into future chunks $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$7.

The projection stage uses the usual linear maps

$$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$8

with $$\mathrm{conv}_q: X\rightarrow Q\in\mathbb{R}^{N\times d_k\times T\times H\times W}$$9 and therefore $d_k=C/2$0 (Tang et al., 23 Jan 2025). The blockwise causal mask $d_k=C/2$1 is defined using consecutive blocks of equal size $d_k=C/2$2 and the block index function

$d_k=C/2$3

The mask is

$d_k=C/2$4

Equivalently,

$d_k=C/2$5

The final attention is

$d_k=C/2$6

Because all pairwise entries inside a block satisfy $d_k=C/2$7, the mask permits full within-chunk cross-person and cross-modality attention, while causally excluding future chunks (Tang et al., 23 Jan 2025). This directly addresses a common misunderstanding: the blockwise mask does not suppress within-chunk multimodal interaction; rather, it preserves it completely.

4. Architectural integration and implementation choices

The CMA block is explicitly described as a wrapper and is therefore intended to be plug-and-play. In the reported experiments, CMA blocks are inserted “every other” residual block within ResNet’s res3 and res4 stages, giving five blocks per branch (Chi et al., 2019). Blocks can be placed symmetrically in both branches, with information flowing in both directions. The branch backbone may be ResNet-50 with 2D convolutions or P3D ResNet / I3D with 3D convolutions. Additional design choices include $d_k=C/2$8, spatial max-pooling $d_k=C/2$9 on $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$0 and $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$1 inputs, single-head self/cross attention with no multi-head splitting, and $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$2BatchNorm(scale=$$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$3 init) as identity at start. The data pipeline uses TSN segments with $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$4 and $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$5, together with flip, crop, and scale jitter augmentation (Chi et al., 2019).

M3PT makes different implementation commitments because it is a causal transformer rather than a two-stream CNN wrapper. Its reported hyperparameters are $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$6 modalities—gaze, head-pose, body-pose, word-embedding, speaking-status, and bite-status—$$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$7 participants, block size $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$8 tokens, model width $$\mathrm{conv}_k: Y\rightarrow K'\in\mathbb{R}^{N\times d_k\times T/1\times H/2\times W/2}$$9, head dimension $2\times2$0, 12 layers, and 16 heads (Tang et al., 23 Jan 2025). The pseudocode constructs $2\times2$1, $2\times2$2, and $2\times2$3 by matrix multiplication with learned projections, makes the blockwise mask, computes masked scores, applies softmax, and returns $2\times2$4. For batching, sequences of shape $2\times2$5 are stacked into $2\times2$6, and the same block mask of shape $2\times2$7 is broadcast across the batch and heads. The paper notes that in PyTorch one can pass $2\times2$8 as the “attention_mask” to torch.nn.MultiheadAttention, or implement the operation via torch.einsum plus masking (Tang et al., 23 Jan 2025).

A plausible implication is that the two implementations occupy complementary points in the design space. CMA is a lightweight wrapper around pre-existing two-stream backbones, whereas the M3PT formulation builds block structure into the tokenization, masking, and transformer stack.

5. Computational structure and efficiency

In CMA, the baseline global attention cost is given as approximately $2\times2$9 (Chi et al., 2019). The reduction strategy has four explicit components: channel reduction with $\times$00, spatial pooling that halves $\times$01 and $\times$02 and therefore yields $\times$03, use of a single head, and implementation of all $\times$04 projections as $\times$05 convolutions with matrix multiplication via BLAS. Zero-initialized BatchNorm on $\times$06 ensures that early training sees identity. The design therefore reduces the cost of cross-attention without changing the global scope of the receptive field over the downsampled key/value positions.

In M3PT, full attention over $\times$07 tokens costs $\times$08 (Tang et al., 23 Jan 2025). With blockwise causal attention, only allowed block pairs are computed. There are $\times$09 allowed block pairs, including intra-block pairs, each of size $\times$10 attention, which yields

$\times$11

This is compared directly to vanilla full attention,

$\times$12

with ratio approximately

$\times$13

For the reported setting $\times$14, the cost is “$\times$15 the cost of full attention” (Tang et al., 23 Jan 2025). Empirically, “on a single NVIDIA V100, M3PT processes $\times$16 fps for batch=8, $\times$17 tokens, comparable to a standard transformer of the same width/length.”

These efficiency patterns are structurally different. CMA reduces the cardinality of attended keys and values through pooling and channel reduction, while M3PT reduces the set of legal query-key pairs through a causal block mask. This suggests that “blockwise cross-attention” may be implemented either as compressed dense attention or as masked structured attention, depending on the computational bottleneck.

6. Empirical behavior, comparisons, and interpretation

For Kinetics, the CMA paper reports the following comparisons between two-stream baselines, CMA, and non-local blocks (Chi et al., 2019).

Model / setting	top-1	top-5
ResNet50 two-stream baseline (R+F late-fusion)	71.21	89.92
CMA$\times$18-R (RGB-branch only)	72.17	90.70
CMA$\times$19-S (weighted two-stream)	72.62	91.04
ResNet50+nonLocal (R+F)	71.67	89.87

The same source states that CMA uses “1.8$\times$20 ResNet50 vs. 2$\times$21 for naïve two-stream” and that adding non-local blocks atop CMA “gives no further gain, showing CMA subsumes global self-attention” (Chi et al., 2019). For 3D-CMA on P3D ResNet-152, the paper reports:

• P3D(12f): $\times$22
• CMA(12f): $\times$23
• P3D(16f): $\times$24
• CMA(16f): $\times$25

For transfer to UCF-101, it reports:

• ResNet50-S (two-stream fine-tuned): $\times$26
• CMA$\times$27-S: $\times$28

The same section notes that this is “Comparable to I3D(64f)+flow at 98% using far heavier 3D nets” (Chi et al., 2019).

The attention visualizations in CMA are also described concretely. In the last CMA block of res4, query pixels on the RGB frame frequently “point” to moving hands, tools or persons in the flow map. Attention is global; for example, a query on the right boxer attends also to the left boxer in a boxing clip. Only discriminative motions such as the “bow tie” region and swinging arms attract high weights, whereas background queries sometimes remain self-focused in failure cases. Compared side-by-side with non-local attention, CMA maps exhibit stronger focus on motion cues complementary to appearance (Chi et al., 2019).

For M3PT, the reported ablations identify dependencies among modalities and context length (Tang et al., 23 Jan 2025). Removing modalities such as gaze, words, and speaking status “hurts bite-timing F1 drastically.” Removing bite status hurts speaking-status prediction. Increasing total context $\times$29 from $\times$30 up to $\times$31 “steadily improves bite timing (F1 from 0.89→0.95) but not speaking status.” Increasing segment length $\times$32 beyond $\times$33 causes “VQ-VAE mode collapse and a crash in both tasks.”

These results support two distinct interpretations. First, in the CMA setting, cross-attention appears to be most beneficial when one modality supplies complementary cues for another, particularly motion cues for appearance features. Second, in the M3PT setting, the blockwise mask preserves the temporal causality required for prediction while retaining rich within-chunk cross-person and cross-modality interaction.

7. Relation to neighboring attention mechanisms

The CMA paper explicitly compares its module with two common alternatives: standard two-stream late fusion and non-local self-attention (Chi et al., 2019). The comparison is not merely architectural but functional. Late fusion combines prediction scores at the end, whereas CMA allows one stream to query another stream’s features throughout the backbone. Non-local blocks compute global self-attention, but the reported absence of additional gain from stacking non-local blocks on top of CMA is interpreted in the source as evidence that CMA “subsumes global self-attention.” In this formulation, the critical distinction is that attention is cross-modal rather than only intra-stream.

The M3PT formulation should likewise be distinguished from unrestricted full attention (Tang et al., 23 Jan 2025). Its blockwise mask does not create local windows or hard person-specific silos within a time-chunk. Instead, it allows reciprocal within-time interaction across persons and modalities and imposes only one prohibition: future chunks cannot be attended. The paper further states that the same design can be adapted whenever “chunks” of inputs should see one another at the same chunk index but must preserve causality across chunks, including streaming audio/video, multi-agent simulations, or multimodal-multi-party transformers.

Across these formulations, the central idea is consistent: cross-attention is shaped by explicit block structure so that information exchange follows the semantics of the task. In video classification, this means directing appearance features toward motion-derived evidence. In multimodal social-signal prediction, this means organizing interaction by time-chunk while preserving both within-chunk reciprocity and cross-chunk causality.