---
title: Temporal-Spatial Codebook Learning (CTSL)
url: https://www.emergentmind.com/topics/codebook-based-temporal-spatial-learning-ctsl
type: topic
---

# Temporal-Spatial Codebook Learning (CTSL)

Searching arXiv for the cited CTSL-related papers to ground the article in current preprints.
Codebook-based Temporal-Spatial Learning (CTSL) denotes a family of representation-learning schemes in which a learned codebook, prototype set, or interpolation-friendly memory is used to organize temporal and spatial structure jointly rather than framewise or pointwise. In the supplied literature, CTSL appears in several distinct but technically related forms: vector-quantized latent bottlenecks for video restoration, temporal codebooks in spiking VQ-VAEs, continuous spatio-temporal feature planes for human avatars, causal prototype codebooks for federated dynamic graphs, multi-scale codebooks for multivariate time-series anomaly detection, and a self-supervised temporal/spatial codebook architecture for Cine MRI risk prediction [2506.13355], [2405.14474], [2503.12751], [2603.29384], [2602.01635], [2507.16612]. Across these works, the common principle is that temporal coherence, temporal order, or transferable temporal structure is not left entirely to generic feature extractors; instead, it is anchored in a codebook-mediated latent space that is explicitly queried, quantized, interpolated, or aligned.

## 1. Conceptual definition and scope

Within the supplied papers, CTSL is most explicitly formulated in the Cine MRI work that names the framework itself, where temporal and spatial features are decoupled and quantized with separate trainable codebooks, denoted \(\mathcal{C}_\tau\) and \(\mathcal{C}_\sigma\), in order to learn dynamic spatiotemporal representations from raw Cine data without segmentation masks [2507.16612]. A closely aligned formulation appears in DicFace, which is described not merely as a video face restoration model but as a concrete instance of CTSL because it begins from a pretrained spatial codebook prior, replaces hard per-frame code assignment with probabilistic code mixtures, and then learns temporal dynamics over those codebook-derived latent variables with a spatio-temporal Transformer [2506.13355].

The broader supplied corpus suggests that CTSL is not a single architectural template but a design pattern. In the spiking VQ-VAE paper, the codebook itself is temporalized: each codeword is a \(T \times c\) latent trajectory rather than a static \(c\)-vector, so quantization occurs over temporal slices attached to spatial feature positions [2405.14474]. In R\(^3\)-Avatar, the “codebook” is not discrete vector quantization but a continuous hex-plane spatio-temporal lookup structure indexed by canonical coordinates and time, with retrieval performed in pose-sequence space [2503.12751]. In SC-FSGL, the codebook is a shared learnable prototype matrix \(\delta \in \mathbb{R}^{\phi \times d}\) used to align shared causal temporal and spatial factors across clients rather than to quantize all latent states [2603.29384]. In COMET, the codebook stores representative normal multiscale patch patterns for multivariate time series and serves simultaneously as a quantizer, memory bank, and adaptation anchor [2602.01635].

A plausible synthesis is that CTSL is best characterized by three jointly necessary ingredients. First, the model has a codebook-like latent organizing structure. Second, temporal information affects either the entries, the assignments, or the retrieval dynamics of that structure. Third, the learned codes remain spatially or structurally localized, whether over image grids, canonical 3D coordinates, graph nodes, or multivariate channels. The supplied works also indicate that CTSL is not confined to discrete tokenization: it includes hard nearest-neighbor quantization, soft assignment, Dirichlet mixture representations, continuous interpolation over spatio-temporal planes, and prototype-based alignment [2506.13355], [2503.12751], [2603.29384].

## 2. Codebook representations and latent parameterizations

The most classical CTSL instantiation in the supplied papers uses VQ-style discrete codebooks. In COMET, for each scale \(k\), a codebook \(\mathbf{C}_k=\{\mathbf{c}_1,\dots,\mathbf{c}_M\}\in\mathbb{R}^{M\times d}\) is learned, and each fused patch embedding \(\mathbf{z}_{e,k}^{(i,j)}\) is assigned to its nearest codeword by
\[
q_k^{(i,j)} = \arg\min_{m} \left\| \mathbf{z}_{e,k}^{(i,j)}-\mathbf{c}_m \right\|_2^2,
\qquad
\mathbf{z}_{q,k}^{(i,j)} = \mathbf{c}_{q_k^{(i,j)}}.
\]
The activated training codewords are then retained as a coreset memory bank representing normal patterns [2602.01635]. The Cine MRI CTSL framework uses an analogous nearest-neighbor vector quantization formulation, but with separate temporal and spatial codebooks, each of size \(n_e=128\) and embedding dimension \(d_c=512\) [2507.16612].

DicFace departs from hard quantization by reformulating code assignment as a convex-mixture variable on the simplex. At each spatial location, the latent is
\[
\hat{v}_{i,j} = \hat{w}_{i,j}^T \mathbf{c},
\qquad
\hat{w}_{i,j}\ge 0,\quad \hat{w}_{i,j}^T\mathbf{1}=1,
\]
and the posterior over \(\hat{w}_{i,j}\) is Dirichlet-distributed rather than one-hot [2506.13355]. This is a central CTSL move because code usage becomes temporally smooth in codebook space rather than abruptly switching via \(\arg\max\). The paper explicitly frames this as a bridge from static spatial codebook priors to dynamic temporal-spatial codebook priors [2506.13355].

The spiking VQ-VAE paper temporalizes the codebook entries themselves. A standard codebook \(Q=\{q_k\}_{k=1}^{N_q}\) with \(q_k\in\mathbb{R}^c\) is replaced by
\[
\tilde{Q} = \{\tilde{q}_k\}_{k=1}^{N_q},
\qquad
\tilde{q}_k \in \mathbb{R}^{T \times c}.
\]
Quantization is then performed by matching a spatial site’s full temporal latent slice \([\tilde z]_{ij}\in\mathbb{R}^{T\times c}\) to the nearest temporal codeword [2405.14474]. The paper states that each temporal code vector represents a spatiotemporal trajectory, and its destructive experiment shows that disturbing temporal ordering inside codewords degrades image quality [2405.14474].

R\(^3\)-Avatar uses a different representational regime. Its “codebook” is implemented by six learned feature planes
\[
(P_{xy}, P_{xz}, P_{yz}, P_{xt}, P_{yt}, P_{zt}),
\]
where the temporal planes \(P_{xt}, P_{yt}, P_{zt}\) record appearance variations caused by clothing at different times, while the spatial planes store canonical static information [2503.12751]. Because retrieval is by continuous interpolation in \((\mathbf{x}_c,t)\), not by discrete index lookup, this paper establishes that CTSL need not imply VQ-style discrete tokens. The same conclusion follows from SC-FSGL, whose codebook consists of trainable causal prototypes used as semantic anchors rather than explicit quantized replacements [2603.29384].

A concise comparison is useful.

| Work | Codebook form | Temporalization mechanism |
|---|---|---|
| DicFace | Pretrained latent codebook with simplex mixtures | Dirichlet-distributed codeword mixtures over frames |
| Temporal-codebook SNN | \(\tilde q_k \in \mathbb{R}^{T\times c}\) | Codeword stores a full latent temporal trajectory |
| R\(^3\)-Avatar | Continuous hex-plane spatio-temporal encoder | Query by \((\mathbf{x}_c,t)\) and retrieve timestamps |
| SC-FSGL | Prototype matrix \(\delta \in \mathbb{R}^{\phi \times d}\) | Shared temporal/spatial causal factors aligned to prototypes |
| COMET | Scale-specific VQ codebooks \(\mathbf{C}_k\) | Multiscale temporal patches quantized as normal prototypes |
| CTSL (Cine MRI) | Separate temporal and spatial codebooks | Decoupled temporal and spatial student features quantized separately |

This range of parameterizations suggests that “codebook-based” in CTSL is broader than classical VQ. It includes dictionary atoms, latent prototypes, interpolation-friendly spatio-temporal fields, and probabilistic convex combinations, provided they structure temporal-spatial learning rather than merely serving as auxiliary memory.

## 3. Temporal-spatial coupling mechanisms

The supplied papers differ sharply in where temporal-spatial coupling occurs. DicFace uses a spatio-temporal Transformer with alternating spatial and temporal self-attention blocks. For a clip \(\{\mathbf{x}^r\}_{r=1}^R\), each frame is encoded to \(\mathbf{z}^r\in\mathbb{R}^{m\times n\times d}\), and the Transformer alternates attention over \(mn\) spatial positions and over the \(R\) frame dimension, with sinusoidal positional encodings added to queries and keys [2506.13355]. Crucially, this Transformer does not just fuse features; it predicts the Dirichlet concentration parameters \(\hat{\alpha}_{i,j}^r\) that govern codeword mixtures. Temporal context therefore shapes the probability distribution over codebook usage itself [2506.13355].

The spiking VQ-VAE paper places temporal coupling in two locations simultaneously. First, the encoder-decoder is built from spiking components with Leaky Integrate-and-Fire dynamics,
\[
H[t] = V[t-1] + \frac{1}{\tau}\big(I[t] - (V[t-1]-V_{reset})\big),
\quad
S[t] = \Theta(H[t] - V_{th}),
\]
so temporal processing occurs inside the neuron model. Second, the temporal codebook quantizes the concatenated latent tensor \(\tilde z = [z_1,\ldots,z_T]\), so the discrete bottleneck itself preserves temporal order [2405.14474]. The paper’s ablation shows that increasing timesteps harms a vanilla codebook but helps or remains stable with the temporal codebook, which directly supports the claim that temporal neural computation alone is insufficient if the latent discretization remains static [2405.14474].

COMET uses a lighter CTSL strategy. Temporal structure is captured by multi-scale patching, with patches
\[
\mathbf{P}_k^{(i,j)} = \mathbf{X}_{j s_k : j s_k+p_k,\; i},
\]
and spatial structure is defined as cross-variable correlation in multivariate time series, modeled by a shared encoder acting on concatenated multivariate patches \(\tilde{\mathbf{P}}_k^{(j)}\in\mathbb{R}^{D p_k}\) [2602.01635]. The paper is explicit that there is no sequential Transformer or RNN over patch indices; temporal learning is primarily within-patch and across scales rather than through explicit long-range inter-patch sequence modeling [2602.01635]. This is an important corrective to a common misconception: CTSL does not necessarily entail full sequence attention.

R\(^3\)-Avatar couples time and space through canonical coordinates and timestamps. A query \((\mathbf{x}_c,t)\) is mapped to a spatio-temporal feature \(\mathbf{f}\), decoded into Gaussian attributes, then warped to posed space by Linear Blend Skinning and rendered by 3D Gaussian Splatting [2503.12751]. Temporal variability is recorded in canonical space and retrieved later by pose-sequence matching with body-part decomposition, delta pose sequences, sliding-window validity constraints, and weighted interpolation between the top two timestamps [2503.12751]. This mechanism indicates that CTSL can be retrieval-driven rather than feedforward temporal aggregation.

SC-FSGL couples temporal and spatial learning through disentanglement and shared prototype alignment. Spatial and temporal latent features are first separated into shared causal and client-specific components by the Conditional Separation Module, then only the shared components are matched to the causal codebook through learned query mappings and softmax scores \(\alpha_j^s\) and \(\alpha_j^t\) [2603.29384]. This suggests a different CTSL principle: the codebook need not organize the full latent state; it can be restricted to the transferable subspace.

In the Cine MRI CTSL framework, temporal-spatial coupling occurs after multi-view distillation. Stage II obtains temporal and spatial student queries \(Z_\tau^{(s)}\) and \(Z_\sigma^{(s)}\), quantizes them with separate codebooks, and recombines them by
\[
Q_{img} = \mathrm{Softmax}\Big(\frac{Q_\tau Q_\sigma}{\sqrt{d_c}}\Big)Q_\sigma.
\]
The paper presents this as a temporal-spatial interaction that yields compact image tokens for downstream survival modeling [2507.16612]. Because the exact architectural branching that produces \(Z_\tau^{(s)}\) and \(Z_\sigma^{(s)}\) is not specified, the decoupling mechanism is clear at the level of objective and representation, but not fully transparent at the level of module internals [2507.16612].

## 4. Objectives, optimization, and implicit versus explicit consistency

A recurring property of the supplied CTSL methods is that temporal consistency is often achieved implicitly rather than with a dedicated temporal penalty. DicFace is a clear example. Its objective combines an ELBO over Dirichlet latent variables with LPIPS:
\[
\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\mathtt{ELBO}} + \lambda_2 \mathcal{L}_{\mathtt{LPIPS}},
\qquad
\lambda_1=-1.0,\ \lambda_2=1.0,
\]
and the reconstruction term derives from a Laplacian likelihood, yielding an \(L_1\)-style loss [2506.13355]. The paper explicitly states that there is no adversarial loss, no explicit temporal consistency term, and no optical-flow consistency or warping loss. Flicker reduction emerges from temporal attention, continuous codeword mixtures, KL-regularized simplex structure, and overlapping center-frame inference [2506.13355].

The temporal-codebook SNN paper uses the standard VQ-style codebook and commitment objective
\[
L_{Q^T} = \|sg(\tilde{z}) - \tilde{z}^q\|_2^2 + \beta \|\tilde{z} - sg(\tilde{z}^q)\|_2^2,
\]
together with MSE reconstruction, perceptual loss, and discriminator-guided loss, though the exact GAN objective is not written in the provided text [2405.14474]. Here again, there is no explicit temporal smoothness regularizer; temporal consistency is induced structurally by SNN dynamics and temporal quantization [2405.14474].

COMET uses a more conventional VQ reconstruction objective,
\[
\mathcal{L}_{total} = \mathcal{L}_{rec} + \alpha \mathcal{L}_{cb} + \beta \mathcal{L}_{cm},
\qquad
\alpha=\beta=1.0,
\]
followed at inference by online test-time adaptation using pseudo-labels derived from whether a codebook entry was activated during training [2602.01635]. The adaptation objective combines normal-only reconstruction/VQ updates with a supervised contrastive loss over pseudo-labels [2602.01635]. This is notable because the codebook is not only a representational bottleneck but also a criterion for deciding which test samples are sufficiently normal to trust during adaptation.

SC-FSGL combines prediction loss, prototype alignment, and invariance regularization:
\[
\mathcal{L}_{\text{local}} = \mathcal{L}_{\text{mse}} + \alpha \mathcal{L}_{\text{com}} + \beta \mathcal{L}_{\text{irm}}.
\]
The paper also notes an inconsistency: one section says the prediction loss is measured by MAE, whereas the local training formulation defines MSE [2603.29384]. This is emblematic of a broader feature of the supplied CTSL literature: several papers are methodologically clear at the conceptual level but contain local ambiguities in notation or implementation details. DicFace notes inconsistencies regarding whether the codebook remains frozen and whether the temporal Transformer has 8 or 9 layers [2506.13355]. The temporal-codebook SNN paper notes that the temporal quantization equation appears to contain a typo [2405.14474]. The Cine MRI CTSL paper does not specify the decoder architecture, exact codebook update mechanism, or train/validation/test split [2507.16612].

These details matter because CTSL methods often sit at the intersection of discrete latent learning, temporal modeling, and downstream task optimization. A plausible implication is that reproducibility hinges not only on the existence of a codebook but on how codebook updates, temporal context windows, assignment granularity, and downstream objectives are operationalized.

## 5. Application domains and empirical evidence

The supplied works show that CTSL is not domain-specific. Its application range includes blind video face restoration, video inpainting, and facial colorization [2506.13355]; SNN-based image generation on static, neuromorphic, and high-resolution datasets [2405.14474]; photorealistic human avatar reconstruction and animation [2503.12751]; federated forecasting on dynamic spatio-temporal graphs [2603.29384]; multivariate time-series anomaly detection [2602.01635]; and non-contrast cardiac risk prediction from Cine MRI [2507.16612].

Empirically, DicFace reports state-of-the-art performance and particularly strong temporal metrics on VFHQ-Test, with PSNR 29.099, SSIM 0.831, LPIPS 0.246, IDS 0.908, AKD 2.093, FVD 336.015, and TLME 1.091 for blind video face restoration [2506.13355]. The paper highlights that this improves TLME by about 5.6% relative to the best listed baseline temporal metric and shows that Dirichlet variational aggregation outperforms Top-\(K\) and Average aggregation in PSNR, AKD, FVD, and TLME [2506.13355]. This is direct evidence that probabilistic codebook-space modeling improves both temporal smoothness and structural fidelity.

The temporal-codebook SNN paper reports that the temporal codebook outperforms several SNN generative baselines across multiple datasets, including FID 13.11 on CelebA \(64\times64\), FID 12.97 on LSUN Bedroom \(64\times64\), FID 4.60 on MNIST, and FID 3.67 on FashionMNIST [2405.14474]. Its most CTSL-specific ablation compares vanilla versus temporal codebooks across timesteps. On CelebA, vanilla codebook FID degrades from 28.00 at \(T=2\) to 73.09 at \(T=6\), whereas the temporal codebook remains around 13.57–14.87; on LSUN Bedroom, vanilla degrades from 44.04 to 121.38 while the temporal codebook improves from 15.24 to 12.90 [2405.14474]. This is unusually direct evidence for the importance of time-aware latent discretization.

R\(^3\)-Avatar reports best novel-view rendering results on the four listed datasets and strong animation metrics, including on DNA-Rendering and HiFi4G with complex clothing [2503.12751]. Its recording-phase ablation on DNA-Rendering shows Full at 32.43/0.974/25.2 versus 30.46/0.966/36.8 without hex-plane and 29.13/0.958/53.0 with no recording-phase architecture, which strongly supports the spatio-temporal codebook representation [2503.12751]. The retrieval ablations further show that body partitioning, sliding window constraints, weighted timestamps, and temporal jitter smoothing materially affect temporal stability [2503.12751].

SC-FSGL reports best average performance across five heterogeneous traffic datasets, with average MAE 4.21, 3.67, and 3.29 for 60, 30, and 15 minute horizons, respectively [2603.29384]. The most relevant evidence for the codebook is the ablation labeled “w/o Book,” which significantly increases MAE [2603.29384]. COMET states that the full model achieves best performance in 39 out of 45 metrics, while the adaptive version further improves 35 out of 45 metrics relative to the non-adaptive version [2602.01635]. Its ablations show that removing multi-scale encoding, quantization score, memory score, local scaling, normalization, or test-time adaptation degrades performance [2602.01635]. The Cine MRI CTSL paper reports C-index values of 0.788, 0.826, and 0.863 across three private cohorts, outperforming CoxPH, DeepSurv, DSM, SurvRNC, and Sparse BagNet [2507.16612]. Its ablation attributes a mean \(\Delta\)C-index of +0.118 to Stage I distillation over the baseline and +0.078 to Stage II over Stage I on average, although RJCCM decreases relative to Stage I distilled features alone [2507.16612].

This distribution of evidence supports a general, but still carefully delimited, conclusion: in the supplied literature, CTSL tends to help most when the task is sensitive to temporal instability, ambiguous pose-to-appearance mappings, non-stationarity, client heterogeneity, or the mismatch between static codebooks and temporally structured data.

## 6. Limitations, misconceptions, and open directions

A frequent misconception is that CTSL necessarily means discrete VQ tokens over video frames. The supplied papers contradict that narrow reading. R\(^3\)-Avatar uses continuous, interpolation-based hex-plane features and retrieval over timestamps rather than explicit quantization [2503.12751]. SC-FSGL uses prototype alignment without commitment loss or quantized reconstruction [2603.29384]. DicFace uses Dirichlet-distributed simplex mixtures rather than one-hot assignments [2506.13355]. CTSL therefore refers more fundamentally to codebook-mediated temporal-spatial organization than to any one quantization rule.

A second misconception is that temporal consistency in CTSL is always explicitly supervised. In several supplied works, it is not. DicFace has no separate temporal consistency loss [2506.13355]. The temporal-codebook SNN paper has no explicit temporal regularizer beyond quantization over temporal slices [2405.14474]. COMET has no explicit long-range temporal sequence model and relies on multi-scale patching plus codebook-based scoring and adaptation [2602.01635]. The literature therefore supports the narrower claim that temporal stability is often induced structurally by the latent organization and inference scheme rather than by dedicated consistency penalties.

The supplied papers also reveal recurring limitations. Many methods are domain-specific: DicFace relies on a face codebook pretrained on high-quality portraits [2506.13355]; R\(^3\)-Avatar depends on multi-view synchronized video and sufficient temporal coverage [2503.12751]; the Cine MRI CTSL framework uses private in-house datasets and omits key split and event statistics [2507.16612]. Motion handling is often implicit rather than explicit: DicFace has no optical flow or deformable alignment module [2506.13355], while COMET’s temporal modeling is local rather than fully sequential [2602.01635]. Prototype granularity and update rules are sometimes under-specified: SC-FSGL does not clearly define whether assignments are per node, per snapshot, or pooled along some axis [2603.29384], and the Cine MRI CTSL paper does not explicitly provide a separate codebook-update loss [2507.16612].

Several forward-looking design principles nevertheless recur. One is to replace hard code selection with smoother or more semantically structured alternatives, such as Dirichlet mixtures or weighted timestamp interpolation [2506.13355], [2503.12751]. Another is to apply the codebook only where transferability matters, as in SC-FSGL’s shared causal subspace [2603.29384]. A third is to let codebook activations govern online adaptation, as in COMET’s seen-codebook pseudo-labeling [2602.01635]. A fourth is to decouple temporal and spatial factors explicitly, as in the Cine MRI CTSL framework [2507.16612]. These patterns suggest that future CTSL systems may vary chiefly along three axes: whether codes are discrete or continuous, whether temporalization lies in code entries or assignment dynamics, and whether the codebook organizes the full latent state or only a selected subspace.

Taken together, the supplied literature supports a precise encyclopedic characterization: CTSL is a research pattern in which codebook structures are promoted from static latent lookup tables to active carriers of temporal-spatial organization. The codebook may store temporal trajectories, interpolate over temporal-spatial fields, constrain probabilistic code mixtures, align transferable causal prototypes, or anchor normal multiscale dynamics. What unifies these variants is the claim that temporal structure should be represented in the code-mediated latent space itself rather than treated as an afterthought to otherwise static spatial representations [2506.13355], [2405.14474], [2503.12751], [2603.29384], [2602.01635], [2507.16612].

Source: https://www.emergentmind.com/topics/codebook-based-temporal-spatial-learning-ctsl