Temporal-Spatial Codebook Learning (CTSL)

• CTSL is a representation-learning paradigm where a learned codebook jointly organizes temporal and spatial features.
• It employs methods like vector quantization, probabilistic mixtures, and continuous interpolation to capture dynamic patterns.
• CTSL improves temporal stability and downstream performance in applications ranging from video restoration to Cine MRI risk prediction.

Searching arXiv for the cited CTSL-related papers to ground the article in current preprints. Codebook-based Temporal-Spatial Learning (CTSL) denotes a family of representation-learning schemes in which a learned codebook, prototype set, or interpolation-friendly memory is used to organize temporal and spatial structure jointly rather than framewise or pointwise. In the supplied literature, CTSL appears in several distinct but technically related forms: vector-quantized latent bottlenecks for video restoration, temporal codebooks in spiking VQ-VAEs, continuous spatio-temporal feature planes for human avatars, causal prototype codebooks for federated dynamic graphs, multi-scale codebooks for multivariate time-series anomaly detection, and a self-supervised temporal/spatial codebook architecture for Cine MRI risk prediction (Chen et al., 16 Jun 2025, Feng et al., 2024, Zhan et al., 17 Mar 2025, Liu et al., 31 Mar 2026, Park et al., 2 Feb 2026, Su et al., 22 Jul 2025). Across these works, the common principle is that temporal coherence, temporal order, or transferable temporal structure is not left entirely to generic feature extractors; instead, it is anchored in a codebook-mediated latent space that is explicitly queried, quantized, interpolated, or aligned.

1. Conceptual definition and scope

Within the supplied papers, CTSL is most explicitly formulated in the Cine MRI work that names the framework itself, where temporal and spatial features are decoupled and quantized with separate trainable codebooks, denoted $\mathcal{C}_\tau$ and $\mathcal{C}_\sigma$, in order to learn dynamic spatiotemporal representations from raw Cine data without segmentation masks (Su et al., 22 Jul 2025). A closely aligned formulation appears in DicFace, which is described not merely as a video face restoration model but as a concrete instance of CTSL because it begins from a pretrained spatial codebook prior, replaces hard per-frame code assignment with probabilistic code mixtures, and then learns temporal dynamics over those codebook-derived latent variables with a spatio-temporal Transformer (Chen et al., 16 Jun 2025).

The broader supplied corpus suggests that CTSL is not a single architectural template but a design pattern. In the spiking VQ-VAE paper, the codebook itself is temporalized: each codeword is a $T \times c$ latent trajectory rather than a static $c$-vector, so quantization occurs over temporal slices attached to spatial feature positions (Feng et al., 2024). In R$^3$-Avatar, the “codebook” is not discrete vector quantization but a continuous hex-plane spatio-temporal lookup structure indexed by canonical coordinates and time, with retrieval performed in pose-sequence space (Zhan et al., 17 Mar 2025). In SC-FSGL, the codebook is a shared learnable prototype matrix $\delta \in \mathbb{R}^{\phi \times d}$ used to align shared causal temporal and spatial factors across clients rather than to quantize all latent states (Liu et al., 31 Mar 2026). In COMET, the codebook stores representative normal multiscale patch patterns for multivariate time series and serves simultaneously as a quantizer, memory bank, and adaptation anchor (Park et al., 2 Feb 2026).

A plausible synthesis is that CTSL is best characterized by three jointly necessary ingredients. First, the model has a codebook-like latent organizing structure. Second, temporal information affects either the entries, the assignments, or the retrieval dynamics of that structure. Third, the learned codes remain spatially or structurally localized, whether over image grids, canonical 3D coordinates, graph nodes, or multivariate channels. The supplied works also indicate that CTSL is not confined to discrete tokenization: it includes hard nearest-neighbor quantization, soft assignment, Dirichlet mixture representations, continuous interpolation over spatio-temporal planes, and prototype-based alignment (Chen et al., 16 Jun 2025, Zhan et al., 17 Mar 2025, Liu et al., 31 Mar 2026).

2. Codebook representations and latent parameterizations

The most classical CTSL instantiation in the supplied papers uses VQ-style discrete codebooks. In COMET, for each scale $k$, a codebook $$\mathbf{C}_k=\{\mathbf{c}_1,\dots,\mathbf{c}_M\}\in\mathbb{R}^{M\times d}$$ is learned, and each fused patch embedding $\mathbf{z}_{e,k}^{(i,j)}$ is assigned to its nearest codeword by

$$q_k^{(i,j)} = \arg\min_{m} \left\| \mathbf{z}_{e,k}^{(i,j)}-\mathbf{c}_m \right\|_2^2, \qquad \mathbf{z}_{q,k}^{(i,j)} = \mathbf{c}_{q_k^{(i,j)}}.$$

The activated training codewords are then retained as a coreset memory bank representing normal patterns (Park et al., 2 Feb 2026). The Cine MRI CTSL framework uses an analogous nearest-neighbor vector quantization formulation, but with separate temporal and spatial codebooks, each of size $\mathcal{C}_\sigma$0 and embedding dimension $\mathcal{C}_\sigma$1 (Su et al., 22 Jul 2025).

DicFace departs from hard quantization by reformulating code assignment as a convex-mixture variable on the simplex. At each spatial location, the latent is

$\mathcal{C}_\sigma$2

and the posterior over $\mathcal{C}_\sigma$3 is Dirichlet-distributed rather than one-hot (Chen et al., 16 Jun 2025). This is a central CTSL move because code usage becomes temporally smooth in codebook space rather than abruptly switching via $\mathcal{C}_\sigma$4. The paper explicitly frames this as a bridge from static spatial codebook priors to dynamic temporal-spatial codebook priors (Chen et al., 16 Jun 2025).

The spiking VQ-VAE paper temporalizes the codebook entries themselves. A standard codebook $\mathcal{C}_\sigma$5 with $\mathcal{C}_\sigma$6 is replaced by

$\mathcal{C}_\sigma$7

Quantization is then performed by matching a spatial site’s full temporal latent slice $\mathcal{C}_\sigma$8 to the nearest temporal codeword (Feng et al., 2024). The paper states that each temporal code vector represents a spatiotemporal trajectory, and its destructive experiment shows that disturbing temporal ordering inside codewords degrades image quality (Feng et al., 2024).

R$\mathcal{C}_\sigma$9-Avatar uses a different representational regime. Its “codebook” is implemented by six learned feature planes

$T \times c$0

where the temporal planes $T \times c$1 record appearance variations caused by clothing at different times, while the spatial planes store canonical static information (Zhan et al., 17 Mar 2025). Because retrieval is by continuous interpolation in $T \times c$2, not by discrete index lookup, this paper establishes that CTSL need not imply VQ-style discrete tokens. The same conclusion follows from SC-FSGL, whose codebook consists of trainable causal prototypes used as semantic anchors rather than explicit quantized replacements (Liu et al., 31 Mar 2026).

A concise comparison is useful.

Work	Codebook form	Temporalization mechanism
DicFace	Pretrained latent codebook with simplex mixtures	Dirichlet-distributed codeword mixtures over frames
Temporal-codebook SNN	$T \times c$3	Codeword stores a full latent temporal trajectory
R$T \times c$4-Avatar	Continuous hex-plane spatio-temporal encoder	Query by $T \times c$5 and retrieve timestamps
SC-FSGL	Prototype matrix $T \times c$6	Shared temporal/spatial causal factors aligned to prototypes
COMET	Scale-specific VQ codebooks $T \times c$7	Multiscale temporal patches quantized as normal prototypes
CTSL (Cine MRI)	Separate temporal and spatial codebooks	Decoupled temporal and spatial student features quantized separately

This range of parameterizations suggests that “codebook-based” in CTSL is broader than classical VQ. It includes dictionary atoms, latent prototypes, interpolation-friendly spatio-temporal fields, and probabilistic convex combinations, provided they structure temporal-spatial learning rather than merely serving as auxiliary memory.

3. Temporal-spatial coupling mechanisms

The supplied papers differ sharply in where temporal-spatial coupling occurs. DicFace uses a spatio-temporal Transformer with alternating spatial and temporal self-attention blocks. For a clip $T \times c$8, each frame is encoded to $T \times c$9, and the Transformer alternates attention over $c$0 spatial positions and over the $c$1 frame dimension, with sinusoidal positional encodings added to queries and keys (Chen et al., 16 Jun 2025). Crucially, this Transformer does not just fuse features; it predicts the Dirichlet concentration parameters $c$2 that govern codeword mixtures. Temporal context therefore shapes the probability distribution over codebook usage itself (Chen et al., 16 Jun 2025).

The spiking VQ-VAE paper places temporal coupling in two locations simultaneously. First, the encoder-decoder is built from spiking components with Leaky Integrate-and-Fire dynamics,

$c$3

so temporal processing occurs inside the neuron model. Second, the temporal codebook quantizes the concatenated latent tensor $c$4, so the discrete bottleneck itself preserves temporal order (Feng et al., 2024). The paper’s ablation shows that increasing timesteps harms a vanilla codebook but helps or remains stable with the temporal codebook, which directly supports the claim that temporal neural computation alone is insufficient if the latent discretization remains static (Feng et al., 2024).

COMET uses a lighter CTSL strategy. Temporal structure is captured by multi-scale patching, with patches

$c$5

and spatial structure is defined as cross-variable correlation in multivariate time series, modeled by a shared encoder acting on concatenated multivariate patches $c$6 (Park et al., 2 Feb 2026). The paper is explicit that there is no sequential Transformer or RNN over patch indices; temporal learning is primarily within-patch and across scales rather than through explicit long-range inter-patch sequence modeling (Park et al., 2 Feb 2026). This is an important corrective to a common misconception: CTSL does not necessarily entail full sequence attention.

R$c$7-Avatar couples time and space through canonical coordinates and timestamps. A query $c$8 is mapped to a spatio-temporal feature $c$9, decoded into Gaussian attributes, then warped to posed space by Linear Blend Skinning and rendered by 3D Gaussian Splatting (Zhan et al., 17 Mar 2025). Temporal variability is recorded in canonical space and retrieved later by pose-sequence matching with body-part decomposition, delta pose sequences, sliding-window validity constraints, and weighted interpolation between the top two timestamps (Zhan et al., 17 Mar 2025). This mechanism indicates that CTSL can be retrieval-driven rather than feedforward temporal aggregation.

SC-FSGL couples temporal and spatial learning through disentanglement and shared prototype alignment. Spatial and temporal latent features are first separated into shared causal and client-specific components by the Conditional Separation Module, then only the shared components are matched to the causal codebook through learned query mappings and softmax scores $^3$0 and $^3$1 (Liu et al., 31 Mar 2026). This suggests a different CTSL principle: the codebook need not organize the full latent state; it can be restricted to the transferable subspace.

In the Cine MRI CTSL framework, temporal-spatial coupling occurs after multi-view distillation. Stage II obtains temporal and spatial student queries $^3$2 and $^3$3, quantizes them with separate codebooks, and recombines them by

$^3$4

The paper presents this as a temporal-spatial interaction that yields compact image tokens for downstream survival modeling (Su et al., 22 Jul 2025). Because the exact architectural branching that produces $^3$5 and $^3$6 is not specified, the decoupling mechanism is clear at the level of objective and representation, but not fully transparent at the level of module internals (Su et al., 22 Jul 2025).

4. Objectives, optimization, and implicit versus explicit consistency

A recurring property of the supplied CTSL methods is that temporal consistency is often achieved implicitly rather than with a dedicated temporal penalty. DicFace is a clear example. Its objective combines an ELBO over Dirichlet latent variables with LPIPS: $^3$7 and the reconstruction term derives from a Laplacian likelihood, yielding an $^3$8-style loss (Chen et al., 16 Jun 2025). The paper explicitly states that there is no adversarial loss, no explicit temporal consistency term, and no optical-flow consistency or warping loss. Flicker reduction emerges from temporal attention, continuous codeword mixtures, KL-regularized simplex structure, and overlapping center-frame inference (Chen et al., 16 Jun 2025).

The temporal-codebook SNN paper uses the standard VQ-style codebook and commitment objective

$^3$9

together with MSE reconstruction, perceptual loss, and discriminator-guided loss, though the exact GAN objective is not written in the provided text (Feng et al., 2024). Here again, there is no explicit temporal smoothness regularizer; temporal consistency is induced structurally by SNN dynamics and temporal quantization (Feng et al., 2024).

COMET uses a more conventional VQ reconstruction objective,

$\delta \in \mathbb{R}^{\phi \times d}$0

followed at inference by online test-time adaptation using pseudo-labels derived from whether a codebook entry was activated during training (Park et al., 2 Feb 2026). The adaptation objective combines normal-only reconstruction/VQ updates with a supervised contrastive loss over pseudo-labels (Park et al., 2 Feb 2026). This is notable because the codebook is not only a representational bottleneck but also a criterion for deciding which test samples are sufficiently normal to trust during adaptation.

SC-FSGL combines prediction loss, prototype alignment, and invariance regularization: $\delta \in \mathbb{R}^{\phi \times d}$1 The paper also notes an inconsistency: one section says the prediction loss is measured by MAE, whereas the local training formulation defines MSE (Liu et al., 31 Mar 2026). This is emblematic of a broader feature of the supplied CTSL literature: several papers are methodologically clear at the conceptual level but contain local ambiguities in notation or implementation details. DicFace notes inconsistencies regarding whether the codebook remains frozen and whether the temporal Transformer has 8 or 9 layers (Chen et al., 16 Jun 2025). The temporal-codebook SNN paper notes that the temporal quantization equation appears to contain a typo (Feng et al., 2024). The Cine MRI CTSL paper does not specify the decoder architecture, exact codebook update mechanism, or train/validation/test split (Su et al., 22 Jul 2025).

These details matter because CTSL methods often sit at the intersection of discrete latent learning, temporal modeling, and downstream task optimization. A plausible implication is that reproducibility hinges not only on the existence of a codebook but on how codebook updates, temporal context windows, assignment granularity, and downstream objectives are operationalized.

5. Application domains and empirical evidence

The supplied works show that CTSL is not domain-specific. Its application range includes blind video face restoration, video inpainting, and facial colorization (Chen et al., 16 Jun 2025); SNN-based image generation on static, neuromorphic, and high-resolution datasets (Feng et al., 2024); photorealistic human avatar reconstruction and animation (Zhan et al., 17 Mar 2025); federated forecasting on dynamic spatio-temporal graphs (Liu et al., 31 Mar 2026); multivariate time-series anomaly detection (Park et al., 2 Feb 2026); and non-contrast cardiac risk prediction from Cine MRI (Su et al., 22 Jul 2025).

Empirically, DicFace reports state-of-the-art performance and particularly strong temporal metrics on VFHQ-Test, with PSNR 29.099, SSIM 0.831, LPIPS 0.246, IDS 0.908, AKD 2.093, FVD 336.015, and TLME 1.091 for blind video face restoration (Chen et al., 16 Jun 2025). The paper highlights that this improves TLME by about 5.6% relative to the best listed baseline temporal metric and shows that Dirichlet variational aggregation outperforms Top-$\delta \in \mathbb{R}^{\phi \times d}$2 and Average aggregation in PSNR, AKD, FVD, and TLME (Chen et al., 16 Jun 2025). This is direct evidence that probabilistic codebook-space modeling improves both temporal smoothness and structural fidelity.

The temporal-codebook SNN paper reports that the temporal codebook outperforms several SNN generative baselines across multiple datasets, including FID 13.11 on CelebA $\delta \in \mathbb{R}^{\phi \times d}$3, FID 12.97 on LSUN Bedroom $\delta \in \mathbb{R}^{\phi \times d}$4, FID 4.60 on MNIST, and FID 3.67 on FashionMNIST (Feng et al., 2024). Its most CTSL-specific ablation compares vanilla versus temporal codebooks across timesteps. On CelebA, vanilla codebook FID degrades from 28.00 at $\delta \in \mathbb{R}^{\phi \times d}$5 to 73.09 at $\delta \in \mathbb{R}^{\phi \times d}$6, whereas the temporal codebook remains around 13.57–14.87; on LSUN Bedroom, vanilla degrades from 44.04 to 121.38 while the temporal codebook improves from 15.24 to 12.90 (Feng et al., 2024). This is unusually direct evidence for the importance of time-aware latent discretization.

R$\delta \in \mathbb{R}^{\phi \times d}$7-Avatar reports best novel-view rendering results on the four listed datasets and strong animation metrics, including on DNA-Rendering and HiFi4G with complex clothing (Zhan et al., 17 Mar 2025). Its recording-phase ablation on DNA-Rendering shows Full at 32.43/0.974/25.2 versus 30.46/0.966/36.8 without hex-plane and 29.13/0.958/53.0 with no recording-phase architecture, which strongly supports the spatio-temporal codebook representation (Zhan et al., 17 Mar 2025). The retrieval ablations further show that body partitioning, sliding window constraints, weighted timestamps, and temporal jitter smoothing materially affect temporal stability (Zhan et al., 17 Mar 2025).

SC-FSGL reports best average performance across five heterogeneous traffic datasets, with average MAE 4.21, 3.67, and 3.29 for 60, 30, and 15 minute horizons, respectively (Liu et al., 31 Mar 2026). The most relevant evidence for the codebook is the ablation labeled “w/o Book,” which significantly increases MAE (Liu et al., 31 Mar 2026). COMET states that the full model achieves best performance in 39 out of 45 metrics, while the adaptive version further improves 35 out of 45 metrics relative to the non-adaptive version (Park et al., 2 Feb 2026). Its ablations show that removing multi-scale encoding, quantization score, memory score, local scaling, normalization, or test-time adaptation degrades performance (Park et al., 2 Feb 2026). The Cine MRI CTSL paper reports C-index values of 0.788, 0.826, and 0.863 across three private cohorts, outperforming CoxPH, DeepSurv, DSM, SurvRNC, and Sparse BagNet (Su et al., 22 Jul 2025). Its ablation attributes a mean $\delta \in \mathbb{R}^{\phi \times d}$8C-index of +0.118 to Stage I distillation over the baseline and +0.078 to Stage II over Stage I on average, although RJCCM decreases relative to Stage I distilled features alone (Su et al., 22 Jul 2025).

This distribution of evidence supports a general, but still carefully delimited, conclusion: in the supplied literature, CTSL tends to help most when the task is sensitive to temporal instability, ambiguous pose-to-appearance mappings, non-stationarity, client heterogeneity, or the mismatch between static codebooks and temporally structured data.

6. Limitations, misconceptions, and open directions

A frequent misconception is that CTSL necessarily means discrete VQ tokens over video frames. The supplied papers contradict that narrow reading. R$\delta \in \mathbb{R}^{\phi \times d}$9-Avatar uses continuous, interpolation-based hex-plane features and retrieval over timestamps rather than explicit quantization (Zhan et al., 17 Mar 2025). SC-FSGL uses prototype alignment without commitment loss or quantized reconstruction (Liu et al., 31 Mar 2026). DicFace uses Dirichlet-distributed simplex mixtures rather than one-hot assignments (Chen et al., 16 Jun 2025). CTSL therefore refers more fundamentally to codebook-mediated temporal-spatial organization than to any one quantization rule.

A second misconception is that temporal consistency in CTSL is always explicitly supervised. In several supplied works, it is not. DicFace has no separate temporal consistency loss (Chen et al., 16 Jun 2025). The temporal-codebook SNN paper has no explicit temporal regularizer beyond quantization over temporal slices (Feng et al., 2024). COMET has no explicit long-range temporal sequence model and relies on multi-scale patching plus codebook-based scoring and adaptation (Park et al., 2 Feb 2026). The literature therefore supports the narrower claim that temporal stability is often induced structurally by the latent organization and inference scheme rather than by dedicated consistency penalties.

The supplied papers also reveal recurring limitations. Many methods are domain-specific: DicFace relies on a face codebook pretrained on high-quality portraits (Chen et al., 16 Jun 2025); R$k$0-Avatar depends on multi-view synchronized video and sufficient temporal coverage (Zhan et al., 17 Mar 2025); the Cine MRI CTSL framework uses private in-house datasets and omits key split and event statistics (Su et al., 22 Jul 2025). Motion handling is often implicit rather than explicit: DicFace has no optical flow or deformable alignment module (Chen et al., 16 Jun 2025), while COMET’s temporal modeling is local rather than fully sequential (Park et al., 2 Feb 2026). Prototype granularity and update rules are sometimes under-specified: SC-FSGL does not clearly define whether assignments are per node, per snapshot, or pooled along some axis (Liu et al., 31 Mar 2026), and the Cine MRI CTSL paper does not explicitly provide a separate codebook-update loss (Su et al., 22 Jul 2025).

Several forward-looking design principles nevertheless recur. One is to replace hard code selection with smoother or more semantically structured alternatives, such as Dirichlet mixtures or weighted timestamp interpolation (Chen et al., 16 Jun 2025, Zhan et al., 17 Mar 2025). Another is to apply the codebook only where transferability matters, as in SC-FSGL’s shared causal subspace (Liu et al., 31 Mar 2026). A third is to let codebook activations govern online adaptation, as in COMET’s seen-codebook pseudo-labeling (Park et al., 2 Feb 2026). A fourth is to decouple temporal and spatial factors explicitly, as in the Cine MRI CTSL framework (Su et al., 22 Jul 2025). These patterns suggest that future CTSL systems may vary chiefly along three axes: whether codes are discrete or continuous, whether temporalization lies in code entries or assignment dynamics, and whether the codebook organizes the full latent state or only a selected subspace.

Taken together, the supplied literature supports a precise encyclopedic characterization: CTSL is a research pattern in which codebook structures are promoted from static latent lookup tables to active carriers of temporal-spatial organization. The codebook may store temporal trajectories, interpolate over temporal-spatial fields, constrain probabilistic code mixtures, align transferable causal prototypes, or anchor normal multiscale dynamics. What unifies these variants is the claim that temporal structure should be represented in the code-mediated latent space itself rather than treated as an afterthought to otherwise static spatial representations (Chen et al., 16 Jun 2025, Feng et al., 2024, Zhan et al., 17 Mar 2025, Liu et al., 31 Mar 2026, Park et al., 2 Feb 2026, Su et al., 22 Jul 2025).