Group Causal Convolution (GCConv)

• GCConv is a module that enhances video VAE performance by partitioning sequences into groups and applying 3D convolution with specialized causal padding.
• It divides video frames into fixed-size groups, using head and tail padding to enforce causal constraints while allowing bidirectional interaction within each group.
• The approach improves convergence and reconstruction fidelity by mitigating artifacts from strict causality, offering efficient temporal compression in generative models.

Group Causal Convolution (GCConv) is a module introduced in the context of improving temporal compression and reconstruction quality in latent video generative models, particularly variational autoencoders (VAEs) for video. GCConv divides video sequences into frame groups, applies 3D convolution with intra-group context, and enforces temporal causality at the group level by specialized logical padding. This design addresses problems associated with existing temporal compression approaches, specifically unequal information interaction between frames and difficulties in frame reconstruction attributed to strict causality. GCConv maintains global causal constraints while enabling bidirectional frame interaction within fixed-size groups, leading to improved convergence and balanced temporal modeling.

1. Mathematical Formalism of GCConv

Let $$Z \in \mathbb{R}^{T \times C_\mathrm{in} \times H \times W}$$ denote a latent tensor containing $T$ video frames with $C_\mathrm{in}$ channels, height $H$, and width $W$. The temporal compression rate $t_c \in \mathbb{N}$ sets the group size $M = t_c$ (e.g., $M=4$). The convolution kernel is parameterized by temporal size $K_t$ (e.g., $K_t=3$), spatial sizes $K_h$ and $K_w$ (e.g., $3 \times 3$), shared convolutional weights $$W \in \mathbb{R}^{C_\mathrm{out} \times C_\mathrm{in} \times K_t \times K_h \times K_w}$$, and bias $b \in \mathbb{R}^{C_\mathrm{out}}$.

The sequence is divided into $G = \lceil T / M \rceil$ frame groups. For group $g=1\ldots G$, with group start index $t_{start} = (g-1) \cdot M$ and group length $M_g = \min(M, T-t_{start})$, the group tensor is $Z^{(g)} = Z[t_{start} : t_{start} + M_g]$. Temporal padding $p = \lfloor K_t/2 \rfloor$ is split into:

• Head-pad $$H^{(g)} \in \mathbb{R}^{p \times C_\mathrm{in} \times H \times W}$$, from the tail of $Z^{(g-1)}$ for $g>1$ or by replicating $Z[0]$ when $g=1$.
• Tail-pad $$T^{(g)} \in \mathbb{R}^{p \times C_\mathrm{in} \times H \times W}$$ of zeros, preventing access to future frames.

The padded group $$Z_\mathrm{pad}^{(g)} = \mathrm{concat}_{\mathrm{time}}\left(H^{(g)}, Z^{(g)}, T^{(g)}\right)$$ undergoes a standard 3D convolution, yielding $$Y^{(g)} = \mathrm{Conv3D}(Z_\mathrm{pad}^{(g)}; W, b)$$ of shape $[M_g, C_\mathrm{out}, H', W']$. The output $Y$ is formed by assigning each $Y^{(g)}$ into its corresponding temporal slot.

2. Grouping and Temporal Causality

GCConv defines group-level receptive fields to control information flow. The input is split into contiguous, fixed-size groups of $M$ frames, except possibly the last group which may be shorter if $T$ is not a multiple of $M$. The first group requires special treatment—its head padding is constructed by replicating its first frame rather than propagating past frames.

Causality is maintained globally: each group processes only its current and previous temporal context, explicitly preventing leakage of future group information via tail padding with zeros. Head padding leverages the trailing frames of the prior group to enable context accumulation. This ensures for any group $g$, only data from groups $1$ to $g$ informs outputs, supporting autoregressive or sequential generation schemes.

3. Intra-Group Convolution and Inter-Frame Equivalence

Within each group, a standard 3D convolution operates over the padded tensor. The symmetric kernel of size $K_t$ ensures that each frame in a group interacts bidirectionally with its temporal neighbors. Thus, inter-frame equivalence is preserved within the group: all frames receive the same bidirectional context and the same spatial convolution as in an image-based VAE. This approach addresses the "starvation" problem found in strictly causal convolutions, where the initial frame or early group positions lack sufficient context, leading to reconstruction artifacts or imbalanced performance.

For variable-length sequences where the final group is shorter than $M$, head and tail padding are determined as above. The convolution proceeds with no alteration to group treatment, providing robust handling for video clips of arbitrary length.

4. Pseudocode, Implementation, and Operational Characteristics

A forward pass of GCConv operates as follows:

def GCConv(Z, W, b, M, K_t): # Z: [T, C_in, H, W] # W: conv weights; b: bias; M: group size; K_t: temporal kernel size p = K_t // 2 T_total = Z.shape[0] G = ceil(T_total / M) Y = zeros([T_total, C_out, H_out, W_out]) prev_tail = None for g in range(1, G + 1): t_start = (g - 1) * M M_g = min(M, T_total - t_start) Zg = Z[t_start : t_start + M_g] # Head padding if g == 1: Hpad = repeat(Z[0:1], repeats=p, axis=0) else: Hpad = prev_tail Tpad = zeros([p, C_in, H, W]) Z_pad = concat([Hpad, Zg, Tpad], axis=0) Yg = Conv3D(Z_pad, W, b) # [M_g, C_out, H', W'] Y[t_start : t_start + M_g] = Yg prev_tail = Zg[-p:] return Y

This process is visualized as: input frames $\rightarrow$ head pad (past) $\rightarrow$ tail pad (future zeros) $\rightarrow$ 3D conv $\rightarrow$ output frames.

The computational complexity matches that of a conventional 3D convolution: $$O(T \cdot C_\mathrm{in} \cdot C_\mathrm{out} \cdot K_t \cdot K_h \cdot K_w \cdot H' \cdot W')$$. Memory overhead for padding is limited to $O(p \cdot C_\mathrm{in} \cdot H \cdot W)$ per group. Only $p$ frames of temporal history are required to be retained, enabling streaming and efficient processing of long videos.

5. Benefits for Temporal Compression and Model Training

GCConv was developed to address the limitations identified in inflated 3D causal VAEs, where causal reasoning produced unbalanced performance across frames and stunted temporal compression when initialized from pretrained image VAEs. By introducing intra-group bidirectionality, GCConv delivers “inter-frame equivalence,” ensuring all frames in a group have similar reconstruction difficulty. This yields smoother training and mitigates flicker or artifacts—common in standard causal convolutions—by avoiding over-dependence on past-only information for early frames in each group.

Furthermore, GCConv accelerates convergence in the latent video VAE training by maintaining a compromise between the causality constraints (needed for autoregressive or generative video sampling) and the reconstruction capacity of bidirectional convs. Extensive benchmark experiments have shown state-of-the-art performance for the associated IV-VAE model in both video reconstruction and generative tasks.

6. Integration and Usage Scenarios

GCConv functions as a drop-in replacement for “pure” causal 3D conv layers in video VAEs, diffusion models, or any temporal latent neural architecture with strict causality requirements. The grouping and padding logic is lightweight and mechanical, making it practical for high-performance research pipelines. The single, shared convolutional weight $W$ is optimized end-to-end as in standard UNet or VAE blocks.

Its design is compatible with variable-length videos and adapts to data streams, provided a manageable value for the group size $M$ is chosen according to dataset temporal coherence and computational constraints. The explicit padding and group structure make it straightforward to interface with modern deep learning frameworks.

7. Significance in Video Latent Modeling

The GCConv module, introduced in “Improved Video VAE for Latent Video Diffusion Model” (Wu et al., 2024), represents a structured approach to balancing expressivity and causality in deep video generative modeling. By partitioning the temporal axis and constraining cross-group information flow, it ensures global causal consistency (required for autoregressive sampling) while leveraging intra-group convolutions for localized temporal context, which refines reconstruction and generative fidelity. This approach is particularly relevant in VAEs powering large-scale video generation models such as OpenAI's Sora and serves as an exemplar of architectural innovations targeting the unique demands of high-dimensional, temporally-extended data.