SceneTilling Algorithm: Efficient Memory Tokenization

Updated 2 June 2026

SceneTilling algorithm is a technique that partitions continuous visual and multimodal inputs into context-aware spatial and temporal tokens.
It compresses and indexes video and scene data by dynamically merging, pooling, and updating memory tokens for efficient long-range retrieval.
The approach underpins advances in video transformers and embodied agents, balancing memory efficiency and high-fidelity spatio-temporal reasoning.

SceneTilling Algorithm refers to a family of techniques for compressing, indexing, and retrieving temporally-evolving scene representations as tokenized “tiles” within large sequence models—especially in the context of video understanding, spatio-temporal reasoning, and persistent memory architectures for long-horizon tasks. The central idea is to segment streams of visual (and optionally multimodal or geometric) input into manageable, context-aware memory tokens that are organized, fused, and updated over time to enable efficient, long-range retrieval with high fidelity. SceneTilling algorithms are core to contemporary video LLM, vision transformer, and embodied agent architectures, and underpin much of the progress in scalable, memory-efficient video and scene reasoning.

1. Formalization of SceneTilling and Temporal Memory Tokens

SceneTilling denotes the process of partitioning continuous sequences (e.g., video frames, spatio-temporal scans) into tokenized “tiles” or memory units, each encoding local or semi-global portions of the scene, with explicit temporal assignment. These memory tokens constitute the "tiles" of the overall context. Key components are:

Spatial Partitioning: Decompose the visual or point cloud input at each timestep into spatially local tokens, e.g., patch embeddings (CLIP/Vision Transformer), object-centric tokens (via segmentation), or geometric tiles (3D point/voxel features) (Hu et al., 28 May 2025, Huang et al., 13 Mar 2026).
Temporal Assignment: Associate each token with a precise timestamp or temporal index, often via sinusoidal/time embeddings or explicit time tags (Su et al., 12 Jan 2026, Huang et al., 13 Mar 2026).
Memory Bank Construction: Aggregate, pool, or compress tokens over multiple timesteps, creating a memory bank that encodes a history of the scene as a tiling of "memory tokens" (Lan et al., 2024, Tao et al., 2024, Kim et al., 12 Mar 2026).
Dynamic Update and Write: As new observations arrive, memory banks are updated through FIFO policies (Lan et al., 2024, Sun et al., 8 Mar 2026), exponentially weighted moving averages (Kim et al., 12 Mar 2026), or learned fusion (Yoo et al., 15 Apr 2026).

The tiles can be spatial (2D/3D patches, object masks), temporal (event-based, clip-based), or spatio-temporal supertokens (e.g., Spatio-Temporal Fusion Tokens—STF-Tokens) (Huang et al., 13 Mar 2026).

2. SceneTilling Algorithms: Memory Construction and Compression

SceneTilling algorithms focus on managing token growth, redundancy reduction, and efficient long-term storage:

Spatial Pooling and Downsampling: Use convolutional pooling (e.g., Pool(·) in multiscale transformers) or Farthest Point Sampling (FPS) in 3D embeddings to compress spatial token sets per frame or clip (Lan et al., 2024, Hu et al., 28 May 2025).
Temporal Merging/Pruning: Merge temporally redundant tokens across frames using cosine similarity, average pooling, or joint selection strategies, e.g., Token Temporal Merging (TTM) in DyCoke (Tao et al., 2024). Adaptive Key Selection (AKS) selects least similar (most informative) patches per frame to avoid storing near-duplicates (Agarwal et al., 20 Feb 2026).
Fixed-Size Buffering: Apply sliding-window or FIFO caches to bound memory—e.g., Window $W_t^i$ in streaming video transformers or episodic FIFO buffers in transformer layers (Lan et al., 2024, Sun et al., 8 Mar 2026).
Attention-Based Memory Write: In memory-augmented transformers, new tile tokens may be appended, replaced, or aggregated conditionally, incorporating attention scores, structural anchors, and selection gates to determine retention (Basu, 13 Apr 2026, Agarwal et al., 20 Feb 2026).
Dual-Stream or Hierarchical Compression: Maintain parallel memory streams with different update rates to balance stability (long-term) and adaptivity (short-term), as in dual memory tokens (LT/ST) in MemRoPE (Kim et al., 12 Mar 2026).

SceneTilling thus encodes a dynamic tiling of the evolving scene, efficiently summarizing both spatial and temporal context.

3. Retrieval, Fusion, and Temporal Reasoning with SceneTilled Tokens

Retrieval from SceneTilled representations supports fine-grained, long-range spatio-temporal reasoning:

Cross-Attention and Content Addressing: Query tokens (e.g., current scene patches or text prompts) attend over the tiled memory bank using dot-product or cosine-similarity attention (Hu et al., 28 May 2025, Agarwal et al., 20 Feb 2026, Hu et al., 28 May 2025).
Temporal Bias and Episodic Retrieval: Inductive biases (primacy/recency) shape retrieval: LLMs and SSMs exhibit U-shaped recall over memory positions, favoring tokens at scene beginnings or ends (Bajaj et al., 26 Oct 2025).
External and MoE Retrieval: Mixture-of-Expert strategies fuse scores/ranks from internal memory and external models (e.g. CLIP, PECore), employing reciprocal-rank fusion for robust identification of relevant scene tiles (Agarwal et al., 20 Feb 2026).
Object/Region Persistence: Object-centric tokens augmented with geometric and temporal metadata (STF-Tokens) enable persistent tracking and causal spatio-temporal graph construction for reasoning over occlusion, action consequences, and causal chains (Huang et al., 13 Mar 2026).
Selective Read and Feature Propagation: Sophisticated attention/aggregation pipelines (e.g., memory-based feature refiner, memory updater modules (Yoo et al., 15 Apr 2026)) filter, project, and fuse memory tiles, propagating context into new predictions.

This memory tiling framework allows flexible, content-aware querying over extensive temporal windows and complex spatial layouts.

4. Token Lifecycle: Write, Retention, and Update Mechanisms

The lifecycle of a SceneTilled tile (memory token) is governed by systematic write, update, and pruning protocols:

Append or Update: New scene observations are encoded into tokenized tiles and either appended to memory (if memory is expandable) or used to update/summarize existing tiles (Lan et al., 2024, Kim et al., 12 Mar 2026).
Equivariance or Consistency-Driven Update: Tokens may be explicitly trained (or dynamically adjusted) for temporal consistency, e.g., via equivariance-driven updates in MAMo (Yasarla et al., 2023), cycle consistency in ChronoTrack (Yoo et al., 15 Apr 2026), or temporal consistency loss (Yoo et al., 15 Apr 2026).
Dynamic Pruning and Sponsorship: Under hard memory budgets, eviction is based on combined attention statistics, sponsorship via structural anchors, or documented utility (Transactional Attention (Basu, 13 Apr 2026)) to ensure critical but low-attention tokens (e.g. credentials, rare events) are never discarded.
Norm-Preserved Residual Updates: When integrating retrieved context, normalization steps prevent distribution shift under frozen weights (as in TempoFit (Sun et al., 8 Mar 2026)).
Compression Across Model Depth: SceneTilling can be implemented at multiple levels: patch-level (encoder), layer-wise (KV-cache or intermediate activations), and decoded memory heads (final LLM or vision output) (Tao et al., 2024, Agarwal et al., 20 Feb 2026).

This lifecycle maintains a compact yet high-fidelity temporal tiling as the explicit memory substrate.

5. Empirical Impact, Benchmarks, and Model Deployments

SceneTilling algorithms have demonstrated significant improvements across efficiency, fidelity, and reasoning capacities:

Model/Method	Key Metric	Result/Improvement
DyCoke (Tao et al., 2024)	Inference speedup	1.5× over baseline
	Memory reduction	1.4× less than baseline
	Retained accuracy	Negligible drop
MemStream (Agarwal et al., 20 Feb 2026)	QA Accuracy (CG-Bench)	+8.0 pp over ReKV
VidCompress (Lan et al., 2024)	Video QA, retention	3–6% gain, >256× compression
ChronoTrack (Yoo et al., 15 Apr 2026)	3D-SOT FPS	42 FPS, SOTA accuracy
RoboStream (Huang et al., 13 Mar 2026)	RLBench (long-horizon)	90.5% (vs. 11.1% baselines)
3DLLM-Mem (Hu et al., 28 May 2025)	3DMem-Bench SOTA	+16.5% on hardest tasks

The practical advances stem from the ability of SceneTilling methods to maintain high retrieval precision on long contexts, reduce memory and latency overhead, and support stateful multi-task reasoning in streaming and embodied agent settings.

6. Architectural Variants and Extensions

SceneTilling is realized in several forms, tailored to context and modality:

Spatio-Temporal Fusion Tokens (STF-Tokens): Object-centric tokens encoding both geometry and appearance plus temporal tags for robotics and manipulation memory graphs (Huang et al., 13 Mar 2026).
Dual-Memory Compression Streams: Parallel slow/fast memory token streams for global/short-term video context (Kim et al., 12 Mar 2026).
Durative and Point-Wise Memory Tokens: For dialogue agents, both discrete event tokens and durative, cluster-abstracted segments summarizing persistent semantic intervals (Su et al., 12 Jan 2026).
Memory Cycles and Consistency Objectives: Specialized consistency losses (cycle consistency, equivariance) to preserve token stability and discriminate scene parts (Yoo et al., 15 Apr 2026, Yasarla et al., 2023).
Context-Aware Pruning/Retrieval: Dynamic and sponsored retention for rare or non-attended temporal tiles ensures generalization to critical contextual cues (Basu, 13 Apr 2026).

SceneTilling thus serves as a unifying abstraction for token-based temporal memory across video, vision-language, dialog, and 3D spatial contexts.

7. Relation to Temporal Biases and Memory in Sequence Models

Underlying the success of SceneTilling is the exploitation—or control—of intrinsic temporal biases in sequence models:

Primacy and Recency Geometry: Positional encoding and architectural dynamics bias retrieval to tokens near temporal boundaries in sequence models; SceneTiling can exploit these by strategic placement or repetition of anchor tokens (Bajaj et al., 26 Oct 2025).
Architectural Convergence: Both transformers and state-space models display similar serial recall and temporal U-biases, indicating that SceneTilling algorithms are robust to model class (Bajaj et al., 26 Oct 2025).

Strategic SceneTilling—via prompt engineering or architectural enhancements—thus enables fine-tuned manipulation of memory specificity, context separation, and retrieval accuracy, paralleling properties of human episodic memory.

References: See (Lan et al., 2024, Tao et al., 2024, Agarwal et al., 20 Feb 2026, Kim et al., 12 Mar 2026, Huang et al., 13 Mar 2026, Yasarla et al., 2023, Sun et al., 8 Mar 2026, Su et al., 12 Jan 2026, Hu et al., 28 May 2025, Yoo et al., 15 Apr 2026, Bajaj et al., 26 Oct 2025, Basu, 13 Apr 2026) for canonical approaches and empirical results on token-based scene tiling and temporal memory in contemporary research.