ASTC: Anchor-Based Spatio-Temporal Compression

• ASTC is a spatio-temporal compression technique that keeps the first frame as a high-fidelity anchor while compressing subsequent frames into a composite image.
• The method reduces visual token count in multimodal models—retaining only 20% of input tokens—while preserving essential temporal order and context.
• Implemented as a deterministic preprocessing step, ASTC outperforms pooling and merging alternatives in video segmentation tasks.

Anchor-Based Spatio-Temporal Compression (ASTC) denotes, in its clearest explicit arXiv formulation, a clip-wise compression strategy that preserves one frame as an anchor while compressing the remaining frames of the clip into a single spatio-temporal summary. In SVAC, ASTC is introduced for referring video object segmentation (RVOS) as a way to scale frame count without overwhelming the multimodal LLM (MLLM): each clip keeps its first frame unchanged as an anchor frame and replaces the other $m-1$ frames with one composite image that preserves temporal order in image space (Zhang et al., 28 Sep 2025). In a broader interpretive sense, the surrounding literature contains several anchor-like spatio-temporal mechanisms—reference frames, temporal memory states, voxel representatives, artificial references, and learned anchor nodes—but multiple papers explicitly distinguish those mechanisms from ASTC in the strict sense of anchor timesteps or anchor frames governing compression of neighboring content (Gong et al., 2024).

1. Terminological scope and lineage

The literature represented here contains two distinct patterns. The first is explicit ASTC, where the term itself is used for a concrete compression module. The second is anchor-like spatio-temporal compression, where the method preserves a reference object—such as a key frame, a recurrent memory state, a voxel representative, a global reference trajectory, or a solver restart state—but does not present itself as ASTC.

The explicit case is SVAC, where ASTC is a deterministic preprocessing module operating on sampled video frames before vision encoding. Its anchors are literal frames: the first frame of each clip is retained at full fidelity, and the rest are packed and resized into one composite image (Zhang et al., 28 Sep 2025). By contrast, several neighboring methods are explicitly described as not ASTC in the narrow sense. The MGARD-based scientific compression framework buffers multiple consecutive timesteps and compresses them jointly as a higher-dimensional block, but it has “no explicit key frames, anchor timesteps, reference states, predictor states, or temporal interpolation from selected frames” (Gong et al., 2024). The low-latency event-stream SNN method proposes a “spatio-temporal compression method” with equal-duration slicing and weighted sub-bin accumulation, but it is also described as “not an anchor-based method in the strict ASTC sense” (Xu et al., 2022).

Earlier and adjacent work nonetheless supplies important antecedents for ASTC-like reasoning. Neural video compression with spatio-temporal priors uses previously reconstructed frames, warped references, ConvLSTM hidden states, and hyperpriors as reusable support states; this is not called ASTC, but it is explicitly characterized as close in spirit to anchor- or prior-based spatio-temporal compression (Liu et al., 2019). Relative compression of trajectories uses an artificial reference and RLZ phrases, so that trajectories are stored relative to a shared representative sequence; this is again not phrased as ASTC, but it is a strong reference-relative analogue (Brisaboa et al., 2018). In turbulence databases, sparse global checkpoints and boundary-plane histories function as physics-based anchors for local re-simulation, even though the paper formulates the method as spatio-temporal sub-sampling and local re-simulation rather than anchor-based coding (Wu et al., 2019).

A practical implication is that ASTC is currently best treated as a specific family resemblance rather than a single universal architecture. The narrow term is precise in SVAC; elsewhere, anchor-based interpretation is often retrospective or comparative.

2. Canonical formulation in SVAC

In SVAC, ASTC is a clip-based, anchor-preserving, image-space compression method inserted on the visual input path to the MLLM. The input video is

$$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$

The sequence is partitioned into $N$ non-overlapping clips of length $m$,

$$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$

Within each clip, the first frame $I_1^{(i)}$ is retained unchanged and designated the anchor frame. The remaining frames are concatenated into a 2D spatial grid in temporal order, left-to-right and top-to-bottom,

$$I_{\mathit{agg}^{(i)}} = \mathit{Concat}(I_2^{(i)}, I_3^{(i)}, \dots, I_m^{(i)}) \in \mathbb{R}^{H' \times W' \times 3},$$

with zero padding if needed. The aggregate is then resized back to single-frame resolution using bicubic interpolation,

$$\tilde{I}_{\mathit{agg}^{(i)}} = \mathit{Bicubic\_Resize}(I_{\mathit{agg}^{(i)}}, H \times W \times 3) \in \mathbb{R}^{H \times W \times 3}.$$

The clip representation is therefore

$\{I_1^{(i)}, \tilde{I}_{\mathit{agg}^{(i)}}\}.$

This construction makes the term “anchor-based” literal: the anchor is not learned, not attention-selected, and not optimized by a placement algorithm. It is simply the first frame in the clip. The method is also explicitly not token merging, learned routing, or adaptive pruning. It is a deterministic image-space transformation applied once before the vision encoder (Zhang et al., 28 Sep 2025).

The token-count argument is central to the method. If the vision encoder yields $s$ visual tokens per frame, then a clip of $$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$0 frames normally produces

$$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$1

After ASTC, the same clip produces

$$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$2

so the retained token fraction is

$$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$3

For $$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$4, the retained token fraction is $$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$5, i.e. $$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$6 of the original frame-token budget (Zhang et al., 28 Sep 2025).

The design rationale is equally specific. The anchor frame preserves a full-resolution, uncompressed visual snapshot for each clip, while the resized composite preserves coarse temporal evolution of the rest of the clip. The paper explicitly positions this against pooling methods such as LongVLM and LLaMA-VID, which may compromise spatial resolution or temporal coherence, and against merging methods such as VisionZip and TokenPacker, which may require structural changes to the MLLM and can lose fine-grained detail (Zhang et al., 28 Sep 2025).

3. Architectural role and empirical behavior

ASTC belongs to stage (1) of the SVAC pipeline. Text and video frames are encoded and projected into a shared embedding space, then fed to the LLM; the hidden states of $$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$7 tokens are extracted from the LLM output; and the SAM-2 decoder combines those token states with SAM-2 visual features to produce masks. ASTC does not directly interact with the segmentation decoder. Its role is upstream: it reduces the number of frame-equivalents presented to the vision encoder and thereby reduces the number of visual tokens sent to the LLM (Zhang et al., 28 Sep 2025).

This placement determines its system function. ASTC addresses the input scaling / visual token bottleneck, while CSA addresses the output prompt granularity / temporal dynamics bottleneck. The paper treats them as complementary. In the main setup, each video is sampled to 100 frames, ASTC divides them into 10 clips, each clip contains 10 frames, and the resulting 100 sampled frames become 20 ASTC images. The LLM maximum sequence length is 12,288; the perception model is frozen; the LLM is fine-tuned with LoRA; and experiments run on 8 NVIDIA A800 GPUs with 40GB memory (Zhang et al., 28 Sep 2025).

The clearest direct evidence for ASTC comes from the compression-method ablation on MeVIS with the 1B model, 80 sampled frames, and 25% compression ratio. Under that matched budget, Average Pooling gives $$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$8, $$V = \{I_1, I_2, \dots, I_T\}, \qquad I_t \in \mathbb{R}^{H \times W \times 3}.$$9, $N$0; Max Pooling gives $N$1, $N$2, $N$3; Token Prune gives $N$4, $N$5, $N$6; Token Merge gives $N$7, $N$8, $N$9; and ASTC gives $m$0, $m$1, $m$2. The paper summarizes this as ASTC outperforming all other methods by at least 1 point (Zhang et al., 28 Sep 2025).

A second result is less about matched baselines and more about the logic of ASTC itself. Using ASTC with 10-frame clips and a 20% compression ratio, increasing sampled frames from 10 to 100 yields consistent improvement in $m$3, $m$4, and $m$5. This supports the paper’s central claim that the bottleneck in RVOS is not merely per-frame fidelity but underuse of available temporal evidence (Zhang et al., 28 Sep 2025).

The full SVAC system, where ASTC is a necessary enabler rather than the sole novelty, reaches $m$6 on MeVIS, $m$7 on Ref-DAVIS17, and $m$8 on ReVOS with 8B parameters, surpassing Sa2VA-26B by $m$9, $$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$0, and $$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$1, respectively. The paper’s framing makes clear that these gains rely on scaling frame input enough for the MLLM to exploit richer video-language context (Zhang et al., 28 Sep 2025).

4. Adjacent formulations and broader research context

Several neighboring papers do not present ASTC directly, but they sharpen what the anchor principle does and does not mean.

In scientific time-series compression, the MGARD-based framework for cyclone tracking is explicitly not ASTC in the sense of anchor selection, yet it provides one of the strongest empirical arguments for preserving temporal continuity. At equal storage cost, 6-hourly temporally decimated data yielded only about 70.71\% generally matched tracks and 9.29\% missed tracks, whereas hourly datasets compressed by uniform MGARD, SZ, and ZFP achieved 100\% matching rate. The same paper compresses every 120 consecutive timesteps jointly, applies tighter error bounds in detected critical regions, and shows that preserving all timesteps at reduced precision can dominate preserving fewer timesteps at full precision for trajectory-sensitive downstream analysis (Gong et al., 2024). This does not instantiate ASTC, but it supplies a strong rationale for why anchor-based methods should usually be judged against temporal decimation rather than only against framewise precision loss.

At the token level, V-CAST provides a closely related but distinct answer to long-video inference. It casts token compression as a trajectory approximation problem, uses curvature-guided temporal allocation, and ranks spatial tokens with a dual-anchor score

$$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$2

where the two anchors are the frame-mean contextual vector and the feature-activation magnitude. It is training-free, plug-and-play, and coordinate-preserving: unlike token merging, it keeps retained tokens at their original $$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$3 coordinates. Under $$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$4, it preserves 98.6\% of vanilla Qwen3-VL-8B performance and outperforms the second-best method by +1.1\% on average; it also reduces peak memory and total latency to 86.7\% and 86.4\% of vanilla Qwen3-VL-8B-Instruct (Lin et al., 29 Mar 2026). In a broad interpretive sense, this is anchor-based spatio-temporal compression at the token-selection level rather than the clip-image level.

STAC pushes the same idea into streaming 3D reconstruction. Its working temporal cache is

$$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$5

and the anchor tokens are selected by decayed cumulative attention

$$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$6

Evicted tokens are not simply dropped; they are compressed into voxel-aligned long-term representatives. The method is described as training-free and plug-and-play, and it reports nearly 10× memory reduction and 4× inference acceleration while maintaining state-of-the-art reconstruction quality (Wang et al., 18 Mar 2026). Here the temporal anchors are explicit, while the spatial “anchors” are voxel representatives rather than named anchor points.

Other related lines contribute complementary anchor-like mechanisms. Neural video compression with spatio-temporal priors uses previous reconstructed frames, warped references, ConvLSTM hidden states, and hyperpriors as multi-level support states, achieving averaged 38% BD-Rate improvement over HEVC Main Profile under MS-SSIM (Liu et al., 2019). Relative compression of trajectories uses an artificial reference and RLZ phrases so that trajectories are represented relative to a shared representative sequence (Brisaboa et al., 2018). In fine-grained spatio-temporal event prediction, GSTPP introduces trainable anchor coordinates $$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$7 and a Self-Adaptive Anchor Graph, which is not a compression method but is highly relevant as a compact continuous-space anchor representation (Zhou et al., 15 Jan 2025). In distributed optimization, observer-based spatio-temporal compression communicates $$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$8, where $$C^{(i)} = \{I_1^{(i)}, I_2^{(i)}, \dots, I_m^{(i)}\}, \quad i=1,\dots,N.$$9 is a maintained reference state; this is structurally close to residual coding around an anchor-like predictor, though the paper analyzes it as a stable dynamical subsystem rather than as ASTC (Ren et al., 2024).

Taken together, these works suggest that ASTC is best understood not as one fixed codec template but as a recurring strategy: retain a small number of high-value reference states and compress the rest relative to them while preserving the geometry of time, space, or both.

5. Misconceptions, limitations, and boundary conditions

A common misconception is that any joint spatio-temporal compression is ASTC. The scientific MGARD framework explicitly rejects this equivalence: it compresses buffered space-time blocks and preserves regions of interest, but it has “no explicit key frames, anchor timesteps, reference states, predictor states, or temporal interpolation from selected frames” (Gong et al., 2024). The event-stream SNN method likewise performs aggressive spatio-temporal aggregation with weighted sub-bin accumulation, yet it also does not implement explicit anchors (Xu et al., 2022). These cases are best regarded as ASTC-relevant rather than ASTC proper.

A second misconception is that ASTC in SVAC is a learned token compressor. It is not. The paper explicitly states that ASTC is not learned attention, clustering, token merging, or adaptive routing; it is a deterministic preprocessing strategy in image space (Zhang et al., 28 Sep 2025). This matters because its advantages and failure modes are correspondingly specific. It preserves exact full-resolution appearance only for the anchor frame. For non-anchor frames, it preserves coarse temporal evolution, temporal order via deterministic layout, and clip-level spatial context, but weakens full per-frame fidelity and precise high-frequency spatial detail after resize (Zhang et al., 28 Sep 2025).

The explicit limitations in SVAC follow directly from that design. Objects that appear briefly may be problematic because their temporal information might be lost during compression and attention computation. Extreme occlusions or dramatic appearance changes remain difficult. The method also has several inferred but text-supported weaknesses: the anchor is always the first frame in the clip, so a blurred or poorly centered first frame yields a suboptimal high-fidelity reference; compression is fixed per clip regardless of motion complexity; resize-induced loss can erase small objects or subtle motion cues; and there is no learned temporal weighting among non-anchor frames (Zhang et al., 28 Sep 2025).

There is also a positional subtlety. In coordinate-sensitive VideoLLMs, token merging can misalign MRoPE-style discrete $I_1^{(i)}$0 bindings. V-CAST argues that pruning retained tokens at original coordinates is safer than merge-based compression for such backbones (Lin et al., 29 Mar 2026). This does not invalidate ASTC, but it constrains which ASTC variants are compatible with position-sensitive tokenization schemes. Similarly, STAC shows that in streaming transformer settings, long-term memory compression must preserve certain temporal heavy hitters and spatially revisitable structures; naive eviction can sharply degrade long-sequence consistency (Wang et al., 18 Mar 2026).

6. Research directions and likely extensions

The clearest open direction is to combine explicit anchors with the stronger space-time fidelity machinery developed elsewhere. The MGARD-based climate paper states that an explicit extension toward ASTC would likely introduce anchor timesteps inside each buffered window, selecting a subset of timesteps with high coefficient energy, high event density, or large temporal novelty as anchors, then compressing residuals of neighboring timesteps relative to those anchors while reusing region-of-interest masks and adaptive error bounds (Gong et al., 2024). Because the paper presents this as an extension rather than a reported result, it is best read as a plausible design blueprint rather than an established ASTC algorithm.

A second direction is the separation of when to preserve more from what to preserve. V-CAST decouples temporal budget allocation from spatial token selection: curvature identifies temporal transition points, and dual anchors preserve diverse and salient evidence within each frame (Lin et al., 29 Mar 2026). This suggests that future ASTC systems need not bind temporal and spatial anchoring into one mechanism. A plausible implication is that anchor placement in time and fidelity allocation in space may benefit from different signals.

A third direction concerns persistent memory rather than clipwise summarization. STAC’s top-$I_1^{(i)}$1 temporal anchor tokens and voxel-local representatives indicate that ASTC can be recast as bounded long-term memory rather than as frame packing. This suggests a streaming ASTC variant in which anchors are neither fixed clip starts nor full keyframes, but dynamically promoted tokens or voxel representatives chosen by long-horizon reuse (Wang et al., 18 Mar 2026).

A fourth direction is domain transfer. In event-stream SNNs, extremely short sequences remain viable when each retained step encodes within-window temporal order through $I_1^{(i)}$2-weighted sub-bin accumulation, and performance at strong compression improves with temporal smoothing in the synaptic convolutional block (Xu et al., 2022). In distributed optimization, observer-based flows compress $I_1^{(i)}$3, where $I_1^{(i)}$4 is a maintained reference state, and prove convergence by analyzing the stability of $I_1^{(i)}$5 (Ren et al., 2024). These are not ASTC results, but they suggest that anchor/reference states plus residual compression form a reusable design pattern across very different spatio-temporal systems.

The current state of the literature therefore supports a restrained conclusion. ASTC, in the explicit sense, is presently exemplified by SVAC’s anchor-frame-plus-composite-image design for RVOS. At the same time, a wider body of work shows that the same structural idea—preserve a compact set of reference states, trajectories, tokens, or regions, and compress surrounding spatio-temporal content relative to them—has become a recurring principle in video understanding, scientific data reduction, streaming memory management, trajectory indexing, and distributed optimization. The term is narrow; the design logic is broader.