--- title: 'MemSinks: Memory & Isolation in Video, LLMs, MEMS' url: https://www.emergentmind.com/topics/memsinks type: topic --- # MemSinks: Memory & Isolation in Video, LLMs, MEMS MemSinks refers to a family of mechanisms, architectures, or device paradigms that serve as context-anchoring, information-localization, or decoupled-sensing elements in computational models and MEMS-based sensors. Three distinct but technically rigorous usages of the term are prevalent in recent literature: as long-range memory anchors in autoregressive video generation, as neuron blocks isolating memorized content in large language model (LLM) training, and as viscosity-decoupled readout structures in MEMS ultrasonic density sensors. ## 1. MemSinks in Autoregressive Video Generation In streaming autoregressive diffusion transformer models for long video synthesis, MemSinks (also termed "frame sinks") are explicit allocations of KV-cache state corresponding to previously generated frames that are retained as long-range anchors throughout the generation process. Standard workflows, as in Caus Vid, Self-Forcing, and LongLive, employ static early-frame MemSinks: an initial block of $S$ frames is kept in memory throughout generation, augmented by a short-term sliding window of $W$ most recent frames for continuity [2605.21028]. The limitation of this static allocation is twofold: (1) as the video progresses and diverges from the starting scene, the relevance of these early frames as context declines rapidly, yet they persist in memory while intermediate (potentially more semantically aligned) frames are evicted under a fixed memory budget $S+W$; (2) repeated attention to these outdated sinks—exacerbated by RoPE-induced phase realignment—can induce "sink collapse", where multiple heads converge in attention to the static sinks, causing generative content to regress to earlier states and resulting in periodic, cyclic, or abruptly resetting motion artifacts. ## 2. DySink: Dynamic Retrieval-Based MemSinks Framework The DySink framework addresses static-sink limitations by replacing fixed early-frame caches with a dynamic, retrieval-based memory bank that stores blockwise descriptors and key-value (KV) caches, supporting adaptive context selection [2605.21028]. ### 2.1 Memory Structure and Update Each entry in the DySink memory bank $M = \{ (f_i, K_i) \}$ consists of: - $f_i \in \mathbb{R}^d$: a normalized latent visual descriptor for block $i$, constructed by mean-pooling over $N_i$ frames after encoding with a frozen visual encoder and applying $L_2$ normalization. - $K_i$: a per-layer set of KV-cache states. A deduplication threshold $T_{\text{dedup}}$ (e.g., 0.98) governs admission to $M$, preventing near-duplicates, as only blocks with $\max_{m\in M} \cos(f_j, f_m) < T_{\text{dedup}}$ are retained. ### 2.2 Retrieval and Injection At each generation block $j$, the descriptors for the last $W$ sliding-window blocks are used as queries. Historical blocks in $M$ are scored for relevance by their mean cosine similarity $s(i,j) = \frac{1}{|W_j|} \sum_{w\in W_j} \cos(f_w, f_i)$. The top-$k$ are selected, and their KV caches are concatenated with the sliding window caches for each transformer layer. This enables conditioning on contextually salient (rather than temporally oldest) history. ## 3. Stability: Sink Anomaly Gate and Collapse Avoidance To suppress failure modes arising from collective inter-head convergence onto retrieved context (i.e., sink collapse), DySink employs a per-layer, per-block binary anomaly gate [2605.21028]. For layer $\ell$, the affinity of each head $h$ to local and retrieved KV sets is computed, with the fraction $\rho_e$ of heads preferring retrieval over locality tallied for each block $e$. If $\rho_e \geq T_{\text{gate}}$ (typ. 0.5), the problematic context is dropped ($g_{j,e,\ell}=0$), otherwise retained. This low-cost, hard gating scheme prevents homogenization of attention and cyclic resets. Ablation confirms the necessity of the anomaly gate: disabling it drops dynamic degree from 63.52 to 37.98 at 50s despite preserved framewise metrics. ## 4. Experimental Protocols and Quantitative Outcomes in Video Models DySink is trained in two stages: short-horizon distillation via Distribution Matching Distillation (DMD) from a causal generator, followed by long-horizon LoRA fine-tuning enabling full retrieval and gating behavior [2605.21028]. Benchmarks use VBench and MovieGen for multi-pronged assessment: text alignment, temporal quality, dynamic degree. Key improvements: - At 50s: Dynamic Degree increases from 42.40 (LongLive static sinks) to 63.52 (+21.12), Temporal Quality rises from 89.29 to 91.63. - At 75–100s: Dynamic Degree remains $>60$ for DySink vs. $\sim40$ for static sinks, with higher temporal stability. - Benefits persist beyond the fine-tuned 60s horizon, demonstrating robust generalization. Table summarizing key metrics at 50s: | Model | Dynamic Degree | Temporal Quality | |----------------|---------------|-----------------| | LongLive (static) | 42.40 | 89.29 | | DySink | **63.52** | **91.63** | ## 5. MemSinks for Memorization Isolation in LLMs In large language model training, "Memorization Sinks" (MemSinks) refer to a subnetwork design in which a fixed pool of neurons per transformer MLP layer is allocated for sequence-specific memory [2507.09937]. During pretraining, each sequence is tagged with an integer sequence ID $k$. The hidden dimension $d_h$ is split into shared ($gd_h$) and sink-pool ($(1-g)d_h$) components. A deterministic binary mask $S^{(\ell)}(k)$, sampled from $k$, is applied to the sink-pool per layer, activating only a unique set of sink neurons for each sequence. Under this regime: - Repeated passages trigger memorization to accumulate in the associated sink neurons. - At inference, zeroing out these neurons (i.e., dropping the corresponding subset by mask) erases the memorized sequence from the model, mitigating both privacy and copyright concerns. - The primary method of isolation is sequence-tied dropout; no explicit memorization regularizer is required. ## 6. Learning, Forgetting Dynamics, and Empirical Results in MemSinks-LLM Standard training under entangled representations induces irreversible mixing of memorization and generalization; post-hoc removal of memorized material damages general performance. The MemSinks mechanism enforces separation: the sink-blockes accumulate sequence-specific logits while the shared block maintains generalization, as formalized in Theorem H.3 of [2507.09937]. Quantitative highlights: - For TinyStories (344M), after dropping MemSink neurons, the cross-entropy loss on repeated docs rises from ≈0.1 to ≈0.95 (near “never-seen” baseline), while held-out val loss shift is <0.02. - At billion-token scale (SmolLM 360M), memorization loss is halved (from 1.2 to 2.3), with validation perplexity unchanged (4.05 vs 4.2). Empirical ablation shows standard gradient-based forgetting cannot disentangle memory from generic language ability; MemSinks achieves selective erasure without collateral degradation. ## 7. MemSinks in MEMS Ultrasonic Density Sensing In MEMS-based fluid-density sensors, "MemSinks" designates a paired structure of AlN-on-SOI PMUTs (Piezoelectric Micromachined Ultrasonic Transducers) operating in a transmit–receive modality [2601.01173]. One PMUT transmits a CW ultrasonic signal into the liquid; the wave reflects at the liquid–air interface and is detected by the receiver PMUT. The receiver output amplitude $A(\rho)$ is directly modeled as a function of fluid density $\rho$ via the "added mass" principle, with negligible dependence on viscosity for typical operation ($\text{Stk}\gg1$). Three frequency intervals are used to map $A(\rho)$ monotonically over 0–100% glycerol concentration without requiring frequency sweeps. Performance summary: - Measurable density range: 0.789–1.261 g/cm³ - Error rate: <0.125% in low-viscosity fluids after cross-liquid calibration; <2.5% in high-viscosity media (80–100% glycerol), or one-fifth the error of frequency-based densitometers. - Sensing is rapid (