Joint Knowledge Transfer & Preservation

• Joint knowledge transfer and preservation is a framework that selectively integrates external insights while protecting native activations and representations.
• It employs methods like modular merging, frozen residual integration, and subspace constraints to balance adaptation with retention.
• This approach supports efficient model adaptation and organizational memory by controlling which knowledge is transferred versus preserved.

Searching arXiv for the cited papers and topic to ground the article in current literature. Joint knowledge transfer and preservation denotes a class of methods and analytical frameworks that seek to import useful information from one or more sources while preventing destructive overwriting of the recipient system’s own structure, task solution, or memory. In contemporary research, this joint objective appears in compact-model training, multi-source model merging, parameter-efficient fine-tuning, heterogeneous domain adaptation, knowledge-graph representation learning, organizational tacit-knowledge management, and citation-network analysis, although several works treat it as a design principle rather than as a standardized formal term (Tchenko et al., 13 Aug 2025, Osial et al., 26 Aug 2025, Luo et al., 29 May 2025, Benderoth et al., 27 Aug 2025, Cunningham et al., 2024). Across these settings, transfer is rarely treated as wholesale copying; instead, the central technical question is how to borrow selectively while preserving native activations, self-learned representations, target-domain structure, organizational memory, or long-lived knowledge communities.

1. Conceptual foundations

A recurrent formulation in the literature is that transfer should be subordinated to the recipient’s own objective rather than replacing it. In "HKT: A Biologically Inspired Framework for Modular Hereditary Knowledge Transfer in Neural Networks" (Tchenko et al., 13 Aug 2025), the child remains optimized for the supervised task,

$$\arg\min_{\theta_\mathcal{C}} \mathbb{E}_{(x,y)}\, \mathscr{L}(\mathcal{C}(x),y),$$

while the pretrained parent acts as an auxiliary source of structured hints. In "Efficient Multi-Source Knowledge Transfer by Model Merging" (Osial et al., 26 Aug 2025), preservation is enforced structurally by freezing the merged singular directions and adapting only a small subset of principal singular values. In "SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA" (Luo et al., 29 May 2025), the trade-off is made explicit through a subspace objective that favors directions informative for the fine-tuning task and suppresses directions associated with knowledge that should be preserved. In "Knowledge Transfer, Knowledge Gaps, and Knowledge Silos in Citation Networks" (Cunningham et al., 2024), the same duality appears at the level of scientific fields: transfer is citation-mediated interaction between communities, while preservation is the persistence and continuity of communities in a cumulative citation graph.

Taken together, these studies suggest several recurring design motifs. First, transfer is made selective rather than uniform. Second, preservation is usually implemented structurally—through residual integration, frozen reference models, sparse projection, or constrained parameterization—rather than through an abstract desideratum alone. Third, preservation is not identical across domains: in compression it often means preserving compactness and native representations; in transfer learning it means preserving target-specific signal; in organizations it means retaining tacit expertise; and in science-of-science it means preserving long-lived intellectual traditions while still enabling cross-community borrowing.

A common misconception is to equate joint transfer and preservation with simple compromise. The surveyed work does not present it as a midpoint between “copy everything” and “copy nothing.” Rather, the objective is asymmetric: borrow what improves the target and explicitly block what is redundant, mismatched, or harmful. This suggests that preservation is best understood not as passivity, but as controlled resistance to indiscriminate transfer.

2. Compression and distillation: from imitation to selective inheritance

Recent neural compression work makes the transfer-preservation tension especially explicit. HKT models parent and child as aligned functional blocks $\{p_1,\dots,p_n\}$ and $\{c_1,\dots,c_n\}$, and transfers knowledge through the ETM triad: Extraction $\mathcal{E}$, Transfer $\mathcal{T}$, and Mixture $\mathcal{M}$. Given parent signal $x$ and child signal $x'$,

$$\mathcal{E}(x)=z,\qquad \tau=\mathcal{T}(z),\qquad \phi=\mathcal{M}(x',\tau).$$

Its Genetic Attention mechanism uses the child as query and transferred parent activation as key and value,

$Q=x',\qquad K=V=\tau(x),$

computes attention over flattened feature dimensions, and injects only a residual parent component: $\{p_1,\dots,p_n\}$0 The full loss

$\{p_1,\dots,p_n\}$1

formalizes that the child’s native task loss remains dominant. Empirically, HKT improved RAFT-S on Sintel and KITTI while preserving the child’s 1.0M parameter count and 50 ms inference; analogous gains were reported for ResNet-20 on CIFAR-10 and PocketNet/Mini U-Net on LiTS without increasing parameter count or inference time (Tchenko et al., 13 Aug 2025).

"Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data" (Xu et al., 2023) addresses a different failure mode: under domain shift, naïve distillation may preserve source-biased teacher behavior while failing to transfer target-relevant invariances. UNI-KD separates teacher knowledge into universal feature-level knowledge and joint logit-level knowledge. A feature-domain discriminator aligns student and teacher features across both domains, while a data-domain discriminator defines sample weights

$\{p_1,\dots,p_n\}$2

so the student imitates the teacher more strongly on domain-shared samples and less on domain-specific ones. The resulting student objective combines adversarial feature matching, domain-discriminator learning, weighted logits distillation, and source-label supervision. The reported average macro F1 is best among the compared students on UCI HAR, HHAR, FD, and SSC, supporting the claim that selective cross-domain transfer can preserve target-relevant behavior better than source-skewed imitation.

Earlier work already pointed toward relational rather than purely output-level preservation. "Learning Deep Representations with Probabilistic Knowledge Transfer" (Passalis et al., 2018) transfers a teacher’s neighborhood structure by matching conditional probability distributions over sample relations. For teacher and student embeddings, PKT defines

$\{p_1,\dots,p_n\}$3

and minimizes KL divergence between the two distributions. The paper argues that this preserves neighborhood affinities and, through a quadratic mutual information argument, preserves label-relevant relational structure even when labels are unknown. Its experiments include retrieval, handcrafted-feature transfer, cross-modal transfer, and object-detection transfer, indicating that preservation can target representation geometry rather than logits alone.

"Knowledge Transfer via Dense Cross-Layer Mutual-Distillation" (Yao et al., 2020) pushes preservation into depth. Teacher and student are trained collaboratively from scratch, auxiliary classifiers are attached to hidden layers, and dense bidirectional distillation is performed both between corresponding supervised layers and across different stages. After training, all auxiliary classifiers are discarded, so no extra final-model parameters remain. The method can be read as preserving intermediate discriminative semantics by continuously supervising hidden layers and using them as teachers, rather than allowing semantic content to accumulate only at the final layer.

3. Multi-source merging, parameter-space transfer, and PEFT

When transfer originates from many source models rather than a single teacher, preservation is often moved from representation space into parameter space. AXIS starts from source task vectors

$\{p_1,\dots,p_n\}$4

and decomposes each matrix-valued module with singular value decomposition,

$\{p_1,\dots,p_n\}$5

All singular components from all sources are pooled, globally ranked by singular value, and the top-$\{p_1,\dots,p_n\}$6 components are reconstructed into a merged matrix. A second SVD,

$\{p_1,\dots,p_n\}$7

re-orthogonalizes the merged basis. Adaptation to the target task then updates only the top-$\{p_1,\dots,p_n\}$8 singular values while keeping $\{p_1,\dots,p_n\}$9, $\{c_1,\dots,c_n\}$0, and the remaining spectrum fixed. This is a direct preservation mechanism: the geometry of the merged source knowledge is frozen, and target learning is reduced to reweighting the strongest directions. On the 22-dataset CLIP benchmark used in the paper, AXIS averaged $\{c_1,\dots,c_n\}$1 Top-1 test accuracy with 16 task vectors on ViT-B-32, compared with $\{c_1,\dots,c_n\}$2 for aTLAS, while maintaining constant adaptation-time memory and runtime with respect to the number of sources (Osial et al., 26 Aug 2025).

ProjectionTL addresses an analogous problem statistically rather than spectrally. It constructs a source-guided prior

$\{c_1,\dots,c_n\}$3

where $\{c_1,\dots,c_n\}$4 is a dummy no-transfer component. Global source relevance is encoded by Dirichlet weights based on cosine similarity, and local preservation is enforced by projecting posterior draws through a weighted $\{c_1,\dots,c_n\}$5-penalized problem,

$\{c_1,\dots,c_n\}$6

This cleanly separates global source selection from local feature selection. The theory gives contraction of the projected transfer posterior at the sparse rate under stated assumptions and includes a non-transferable regime in which the dummy component absorbs posterior mass, thereby mitigating negative transfer (Pal et al., 7 Jun 2026).

SC-LoRA recasts the same joint objective for LLM fine-tuning. It defines a subspace reward

$\{c_1,\dots,c_n\}$7

with

$\{c_1,\dots,c_n\}$8

and chooses the LoRA output subspace from the top eigenspace of $\{c_1,\dots,c_n\}$9. Initialization is

$\mathcal{E}$0

so the LoRA branch initially equals the projection of the pretrained layer output onto the selected subspace. Here transfer means rapid adaptation to $\mathcal{E}$1, while preservation means minimizing interference with the knowledge associated with $\mathcal{E}$2. On the reported experiments, SC-LoRA improves downstream utility while preserving safety and world knowledge better than vanilla LoRA, PiSSA, and CorDA variants, though the paper explicitly notes that it is only an initialization method and preservation can still degrade after longer or harder fine-tuning (Luo et al., 29 May 2025).

4. Heterogeneous domains, modalities, and structured semantics

In heterogeneous domain adaptation, preservation often concerns structure that would otherwise be destroyed by aggressive alignment. "Joint Information Preservation for Heterogeneous Domain Adaptation" (Xu et al., 2019) combines distribution matching with three preservation mechanisms: paired-sample correlation, local structural preservation through a graph Laplacian, and global discriminative preservation through between-class and within-class scatter. Its generalized eigenvalue formulation joins these forces in a single optimization, indicating that in HDA transfer and preservation are not separable stages but coupled constraints on the shared subspace.

"Joint Semantic Transfer Network for IoT Intrusion Detection" (Wu et al., 2022) extends this idea to multi-source heterogeneous domain adaptation. It transfers three forms of semantics—scenario semantics, implicit semantics, and explicit semantics—while preserving categorical distribution relationships and class geometry. Scenario semantic transfer uses a confused domain discriminator and source-source categorical preservation; weighted implicit semantic transfer preserves source class-probability structure in the target; hierarchical explicit semantic alignment preserves centroid-level and representative-level geometry. The full objective combines supervised source loss, weighted implicit semantic loss, centroid alignment, representative alignment, domain confusion, and source-source categorical preservation. Reported experiments on five intrusion datasets show the best average accuracy among the compared methods, with the paper emphasizing an average gain of about $\mathcal{E}$3 over state-of-the-art comparators.

Bridged-GNN departs from domain-level alignment entirely. It defines a Bridged-Graph

$\mathcal{E}$4

learns pairwise similarities with an Adaptive Knowledge Retrieval module, retrieves top-$\mathcal{E}$5 beneficial samples for each target sample, and then performs graph knowledge transfer on the resulting graph. Its most explicit preservation term is a target-side reconstruction loss,

$\mathcal{E}$6

introduced to prevent the target encoder from forgetting original target-domain information. The broader preservation mechanism is selective scope definition: knowledge is transferred only along retrieved beneficial-benefited edges, rather than through global posterior alignment (Bi et al., 2023).

A related problem appears in multi-source knowledge graphs. "Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge Graphs" (Sun et al., 2023) first pre-trains a large teacher on linked background KGs, then extracts a linked subgraph for the target KG using entity alignment and distills teacher knowledge through feature KD, network KD, and prediction KD. The teacher remains frozen and reusable, so previously learned multi-source knowledge is preserved at the teacher level, while the student is locally re-trained on the target KG and linked subgraph, preserving target-specific relational structure rather than re-optimizing globally from scratch.

These works show that in heterogeneous settings preservation typically means one of three things: preserving paired correspondences, preserving target-domain information, or preserving explicit semantic geometry. This suggests that “preservation” is most informative when tied to a concrete structure—pairs, neighborhoods, centroids, linked subgraphs, or reconstructions—rather than invoked generically.

5. Organizational and scientific knowledge systems

Outside machine learning model transfer, joint knowledge transfer and preservation is treated as a socio-technical or ecosystem-level problem. "Socially Interactive Agents for Preserving and Transferring Tacit Knowledge in Organizations" (Benderoth et al., 27 Aug 2025) targets tacit knowledge that is highly personal, experience-based, and difficult to formalize. It proposes Socially Interactive Agents as knowledge transfer facilitators spanning three phases labeled in Figure 1 as “I Co-Creation,” “II Knowledge preservation,” and “III Knowledge application.” The same infrastructure—LLMs for contextually relevant dialogue, RAG for access to organizational knowledge, and CoT prompting for structured reflection—is meant first to elicit and externalize expert knowledge and then to support onboarding, mentoring, and knowledge application. The paper’s evidence is conceptual and architectural rather than experimental; it explicitly reports no dedicated experiments, user studies, or quantitative results for the proposed system. That limitation is substantial, but the paper is unusually clear that preservation and transfer are phases of one continuous process rather than separate systems.

At the scale of scientific fields, citation-network analysis provides a different formalization. "Knowledge Transfer, Knowledge Gaps, and Knowledge Silos in Citation Networks" (Cunningham et al., 2024) models a field as a cumulative dynamic citation network

$\mathcal{E}$7

detects step communities with OSLOM, links them into dynamic communities over time, and defines community interaction probabilities

$\mathcal{E}$8

Because the network is cumulative, earlier papers are preserved in later graphs; because inter-community edges change over time, transfer can be measured as citation interaction. Applied to XAI, the framework identifies limited transfer from foundational communities, isolated knowledge silos, and significant knowledge gaps. The key implication is that preservation of strong communities does not guarantee healthy transfer; preserved traditions can remain underused or become siloed.

A simpler organizational dynamics model appears in "Model of knowledge transfer within an organisation" (Kowalska-Styczeń et al., 2017). The organization is represented as an $\mathcal{E}$9 cellular automaton, each agent holds binary knowledge chunks, and an agent can learn one missing chunk from a neighbor only if the neighbor has exactly one more chunk overall. The study reports a significant role of the initial concentration of chunks of knowledge and suggests shortening social distance between people with different levels of knowledge or working out incentives to share knowledge. The model is primarily about transfer, but its coverage measures also imply a preservation perspective: knowledge widely distributed across members is less fragile than knowledge concentrated in a few individuals.

6. Limits, ambiguities, and open directions

Several limitations recur across the literature. Architectural assumptions are often strong. HKT works best when parent and child can be modularized into semantically aligned blocks, and the paper explicitly notes that transferring between heterogeneous architectures such as transformers and CNNs is harder because block correspondence becomes ambiguous (Tchenko et al., 13 Aug 2025). AXIS requires a common architecture and shared pretrained initialization across all source models (Osial et al., 26 Aug 2025). ProjectionTL assumes a common ambient parameter dimension for source coefficient summaries and target coefficients (Pal et al., 7 Jun 2026). The multi-source KG framework depends on entity alignment between KGs (Sun et al., 2023). Organizational SIA systems depend on trust, transparency, privacy protection, and a culture willing to externalize tacit knowledge (Benderoth et al., 27 Aug 2025).

Evidence quality is also uneven. HKT’s strongest empirical comparisons are in optical flow; outside that setting, the paper reports parent/child/HKT comparisons but no broad KD baselines for CIFAR-10 or LiTS (Tchenko et al., 13 Aug 2025). AXIS gives strong indirect evidence for tuning only principal singular values, but does not include a direct ablation against tuning minor singular values or singular vectors (Osial et al., 26 Aug 2025). SC-LoRA explicitly states that it is only an initialization method and that preservation can drop after longer or more complex fine-tuning (Luo et al., 29 May 2025). The SIA paper is architectural rather than empirical (Benderoth et al., 27 Aug 2025). Citation-network methods can identify gaps and silos, but they do not automatically supply intervention mechanisms (Cunningham et al., 2024).

A second ambiguity concerns what “preservation” actually means. In some papers it means preserving native activations, self-learned representations, architectural efficiency, and compactness, as in HKT (Tchenko et al., 13 Aug 2025). In others it means preserving source-derived directions in parameter space, as in AXIS (Osial et al., 26 Aug 2025). In SC-LoRA it means preserving safety or world knowledge (Luo et al., 29 May 2025). In Bridged-GNN it means preserving target-domain information during transfer (Bi et al., 2023). In citation networks it means preserving long-lived communities and knowledge ecosystems (Cunningham et al., 2024). This suggests that joint knowledge transfer and preservation is better viewed as an umbrella research program than as a single formal problem class.

Finally, preservation should not be conflated with freezing. Frozen teachers, frozen singular vectors, dummy no-transfer components, residual mixtures, or reconstruction losses are all preservation devices, but they preserve different objects. Nor is transfer synonymous with imitation. HKT explicitly criticizes uniform teacher imitation in favor of complementary inheritance (Tchenko et al., 13 Aug 2025), while PKT preserves relational structure rather than raw outputs (Passalis et al., 2018). The strongest general lesson is therefore methodological: effective transfer usually depends on choosing the right invariant to preserve. In current work, that invariant may be a blockwise computation, a singular basis, a covariance-defined subspace, a paired cross-domain relation, a class-correlation graph, a tacit reasoning trace, or a persistent citation community.