Memory Transfer Learning

• Memory Transfer Learning is a paradigm that uses explicit, model-external memory stores to transfer knowledge across tasks and domains for efficient and robust adaptation.
• It incorporates diverse architectures such as memory-augmented coding agents, recurrent memory banks, and kernel-based models to optimize performance and mitigate negative transfer.
• Empirical results demonstrate measurable gains including improved Pass@3 scores, reduced MSE in spatiotemporal tasks, and significant memory savings for edge-device adaptations.

Memory Transfer Learning (MTL) refers to a suite of methodologies that leverage explicit, model-external memory structures to support transfer across tasks, domains, temporal regimes, or representations. MTL formalizes and operationalizes the exploitation of memory—whether non-parametric stores of experience, learned memory kernels, recurrent state banks, or algorithmic memory models—to improve adaptation, generalization, and efficiency in diverse machine learning settings. The MTL paradigm subsumes frameworks varying from transfer of meta-knowledge in coding agents, unsupervised knowledge distillation from multiple pretrained recurrent neural networks, memory-efficient model adaptation for edge devices, to non-parametric continual learning architectures that resist catastrophic forgetting via compositional memory modules (Kim et al., 15 Apr 2026, Yao et al., 2020, Chiang et al., 2022, Wang et al., 2020).

1. Foundations and Formalization of Memory Transfer Learning

MTL frameworks share the central principle that learning is augmented by retrieving or transferring knowledge encoded in explicit memories, which are distinct from the parametric weights of the target model. In formal terms, let $\mathcal{D}_\text{src}$ and $\mathcal{D}_\text{tgt}$ denote (potentially heterogeneous) source and target domains or tasks. MTL maintains a memory pool $\mathcal{M}$ representing episodic experience, memory kernels, or feature traces accumulated during (or prior to) source task exposure.

At test time or fine-tuning, an MTL-enabled agent for input $x$ typically computes a retrieval score over $\mathcal{M}$ based on embedding similarity, gating, or kernel-matched context, and adapts inference or learning dynamics accordingly (Kim et al., 15 Apr 2026, Wang et al., 2020). Transfer is thus realized non-parametrically—by in-context augmentation, feature distillation, or dynamic recurrence—rather than by mere parameter adaptation.

This explicit memory mediation allows the agent to exploit meta-knowledge, mitigate catastrophic forgetting, and support sample-efficient out-of-distribution adaptation, rendering MTL fundamentally different from classical (parameter-)transfer paradigms.

2. Core Architectures and Mechanistic Designs

2.1 Memory-Augmented Coding Agents

In cross-domain coding, MTL operationalizes a non-parametric memory store built from trajectories, workflow sequences, summaries, or abstract “insights” obtained by previous inference over diverse programming tasks. For a new task $x$, the agent retrieves top-$N$ most similar memories by cosine similarity of embeddings $e(x), e(m)$:

$$\mathcal{R}(x) = \operatorname*{arg\,topN}_{m\in\mathcal{M}}\,\cos(e(x), e(m))$$

These memories are prepended to the model’s prompt, guiding completion with high-level meta-knowledge such as validation routines, environment constraints, or testing best practices. Empirically, abstraction level in memory format (e.g., “insight” vs. “workflow”) governs transferability, with abstracted content ($z_\text{inv}(m)$) maximizing cross-domain gains and minimizing negative transfer from overly specific low-level traces ($\mathcal{D}_\text{tgt}$0) (Kim et al., 15 Apr 2026).

2.2 Transferable Memory Bank for Recurrent Neural Networks

In spatiotemporal predictive tasks, MTL frameworks such as the Transferable Memory Unit (TMU) maintain a bank of $\mathcal{D}_\text{tgt}$1 frozen teacher RNNs, each providing rich memory state trajectories $\mathcal{D}_\text{tgt}$2. The TMU recurrent cell dynamically distills and adaptively gates teacher memory, fusing it with its own evolving cell state in a candidate update:

$\mathcal{D}_\text{tgt}$3

with $\mathcal{D}_\text{tgt}$4 a transfer gate weighting teacher $\mathcal{D}_\text{tgt}$5's memory, and $\mathcal{D}_\text{tgt}$6 a distilled (projected) version of the TMU’s state aligned to that of teacher $\mathcal{D}_\text{tgt}$7. This architecture enables the target model to leverage, filter, and blend diverse spatiotemporal dynamic priors (Yao et al., 2020).

2.3 Kernel Memory Transfer in Coarse-Grained Physical Models

MTL appears in the transfer of non-Markovian memory kernels in multi-parameter dynamical systems. Here, the memory kernel $\mathcal{D}_\text{tgt}$8 is efficiently represented by proper orthogonal decomposition (POD) and inferred for out-of-sample parameters via Gaussian process regression (GPR) on the expansion coefficients. This enables scalable, uncertainty-aware construction of memory kernels that generalize across thermodynamic and molecular weight regimes (Ma et al., 2021).

2.4 Memory-Efficient Model Adaptation and On-Device MTL

Memory Transfer Learning also encompasses frameworks for reducing memory and compute footprints during transfer learning in neural models. MobileTL, for instance, abandons storing full-precision activation maps (replacing them with 1-bit masks), fine-tunes only a subset of normalization layer shifts, and restricts back-propagation to top layers. This results in dramatic peak memory and FLOPs savings with negligible or positive impact on accuracy for edge devices (Chiang et al., 2022).

Advanced schemes (e.g., SHERL, MDPD) decouple adaptation routes, blend anti-redundancy compressed feature maps via attention mechanisms, and combine mutually distilled side-networks with fading or late-stage regulation, achieving a best-in-class parameter vs. memory vs. accuracy trade-off in resource-limited settings (Diao et al., 2024, Zhang et al., 10 Apr 2026).

2.5 Modular Memory-based Continual Learning

Frameworks such as the Forget-Me-Not (FMN) process instantiate a combinatorial MTL at the neuron level. Each neuron maintains a pool of up to $\mathcal{D}_\text{tgt}$9 locally optimal “solutions.” A Bayesian ensemble over all possible combinations of these memory states across network neurons provides exponential coverage of pseudo-tasks without explicit task boundary supervision. Gated Linear Networks enforce modularity, feeding FMN-smoothed outputs through context-gated local weights, which enables robust continual learning, positive forward and backward transfer, and resistance to catastrophic forgetting (Wang et al., 2020).

3. Experimental Protocols, Metrics, and Empirical Findings

3.1 Coding Agents: Cross-Domain Memory Transfer

In cross-benchmark coding, MTL with memory-based retrieval yields average Pass@3 improvements of 3.7% vs. zero-shot, with gains scaling with pool diversity and abstraction (Kim et al., 15 Apr 2026). Table 1 below summarizes performance across six coding benchmarks.

Format	Avg Δ vs. ZS	Max per-task Δ	Highlights
Trajectory	+1.1%	+2.0%	Modest gains, possible negative tr.
Workflow	+1.5%	+4.5%	Substantial on CLI/Replicate tasks
Summary	+2.3%	+8.3%	Broad, robust improvements
Insight	+3.7%	+8.3%	Maximized transfer, esp. MLBench

Performance gains derive predominantly from meta-knowledge and are highest with insight-level memories. Overly specific traces can induce negative transfer (Kim et al., 15 Apr 2026).

3.2 Spatiotemporal Prediction: TMU Distillation

In moving digit (MNIST), human motion, and precipitation nowcasting, TMU-equipped models outperform finetuning by 8–21% in MSE/SSIM. Notably, even irrelevant source domains are occasionally filtered and used beneficially by the gating mechanisms (Yao et al., 2020).

3.3 Physical Modeling: Memory Kernel Transfer

Across star-polymer and peptoid systems, ROM-GPR-based MTL produces under 3% relative $\mathcal{M}$0 error on interpolated tests and maintains prediction fidelity for out-of-domain parameters (up to 8.6% error in extrapolation), at orders of magnitude (10^4–10^6) lower computational cost versus direct GPR (Ma et al., 2021).

3.4 Edge/Resource-Limited Adaptation

MobileTL achieves up to 91% memory savings (MobileNetV2/3), 64% FLOP savings, and even a mild accuracy boost (+0.6%–0.4%) on CIFAR-10/100 by fine-tuning only the top blocks and storing only minimal information during training (Chiang et al., 2022). SHERL further recoups nearly all parameter-efficient transfer learning accuracy at 10–20% of memory load, with deployment guidance explicated for both NLP and vision–language backbones (Diao et al., 2024).

3.5 Continual/Incremental Settings

Memory-based modular learners achieve zero catastrophic forgetting ($\mathcal{M}$1), positive forward transfer ($\mathcal{M}$2), and competitive or superior accuracy to both replay-free and replay-based continual learning methods on challenging benchmarks (Split MNIST, Permuted MNIST, electricity data) (Wang et al., 2020).

4. Theoretical Analyses and Design Principles

Several analytic outcomes and empirical design recommendations have emerged:

• Abstraction–Transferability Trade-off: Expected transfer gain increases with abstraction level of retrieved memory, as more abstract representations yield higher domain invariance and minimize spurious specificity (Kim et al., 15 Apr 2026).
• Scalability in Coverage: Memory transfer effectiveness scales with the diversity and size of the memory pool, due to maximized chances for meta-knowledge retrieval (Kim et al., 15 Apr 2026).
• Combinatorial Generalization: In memory-augmented modular networks, the ensemble of pseudo-task networks covers an exponential class of solutions, with memory costs only linear in the number of modules and retained states (Wang et al., 2020).
• Memory–Efficiency–Accuracy Pareto Frontier: Techniques such as MobileTL, SHERL, and MDPD define the current Pareto frontier for accuracy achieved per unit memory or computational constraint across edge and large-backbone transfer settings (Chiang et al., 2022, Diao et al., 2024, Zhang et al., 10 Apr 2026).

5. Limitations, Risks, and Future Directions

Observed limitations include:

• Negative Transfer: Excessive reliance on low-level, domain-specific traces can produce transfer detriment (“negative transfer”) (Kim et al., 15 Apr 2026).
• Inference Overhead: Some memory-efficient methods (side networks) incur inference-time penalties unless methods like “fading” are used to eliminate extra computation at test time (Zhang et al., 10 Apr 2026).
• Selection and Reranking: More sophisticated retrieval or adaptation mechanisms may be necessary to avoid spurious memory anchors or overfitting to misleading content (Kim et al., 15 Apr 2026).
• Model–Memory Fusion: Integration of MTL with parametric fine-tuning holds promise for further gains, notably by locking in transferable insights found by non-parametric memory frameworks.

Proposed future research directions include adaptive memory routing, extension to multi-modal contexts, enhancement of memory abstraction mechanisms, and unification of memory and gradient-based transfer in a hybrid framework (Kim et al., 15 Apr 2026, Zhang et al., 10 Apr 2026).

6. Applications and Broader Impact

Memory Transfer Learning has been successfully applied in:

• Cross-benchmark coding agents for robust code synthesis and repair (Kim et al., 15 Apr 2026)
• Unsupervised multi-domain spatiotemporal prediction (Yao et al., 2020)
• Memory-kernel extrapolation for transferable coarse-grained physical models (Ma et al., 2021)
• Resource-constrained learning for edge devices and mobile deployment (Chiang et al., 2022, Diao et al., 2024)
• Lifelong and continual learning with robust transfer and no catastrophic forgetting across changing input regimes (Wang et al., 2020)

These advances collectively position MTL as a paradigm for scalable, robust, and interpretable adaptation in modern machine learning.

---

References

• (Kim et al., 15 Apr 2026) Memory Transfer Learning: How Memories are Transferred Across Domains in Coding Agents
• (Yao et al., 2020) Unsupervised Transfer Learning for Spatiotemporal Predictive Networks
• (Chiang et al., 2022) MobileTL: On-device Transfer Learning with Inverted Residual Blocks
• (Diao et al., 2024) SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning
• (Zhang et al., 10 Apr 2026) Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation
• (Ma et al., 2021) Transfer Learning of Memory Kernels in Coarse-grained Modeling
• (Wang et al., 2020) A Combinatorial Perspective on Transfer Learning