Architecture Shapes Transfer Specificity in Implicit Neural Representations

Abstract: Transfer in coordinate networks is often measured by warm-start gain, but whether that gain reflects source-specific structure or generic weight reuse is less clear. We study this question across three implicit neural representation (INR) families, SIREN, ReLU MLPs, and Fourier-feature MLPs, using controlled analytic tests, a 2D lid-driven-cavity Navier--Stokes benchmark, and 1D PDE reference-solution suites for heat, viscous Burgers, and focusing cubic NLS. The analytic tests use independent-seed random controls, while the PDE benchmarks use alternate same-family source controls and auxiliary ablations. Across settings, transfer magnitude and transfer specificity separate clearly. In a 10-seed controlled 1D geometric test, Fourier Features show the largest structured transfer ($33.1\times$), followed by SIREN ($23.0\times$) and ReLU ($10.7\times$), but ReLU is far more selective: random-control transfer is $0.41\times$ for ReLU versus $14.24\times$ for SIREN. On a controlled two-parameter 1D family, the ranking changes: ReLU gives the clearest structured-versus-control separation at default settings, whereas Fourier Features improve only after bandwidth retuning. In Navier--Stokes and the broader 1D PDE suite, no single architecture dominates every equation, yet the same pattern remains: SIREN often reuses weights broadly, whereas ReLU and, in some equations, Fourier Features are more source-selective. Static diagnostics remain weak, and the heuristic scaling law $A_{\text{transfer}} \propto 1/Δt^2$ is rejected in the implemented 1D audit. These results position transfer specificity as a useful diagnostic for coordinate networks and suggest that architecture selection in scientific machine learning should be evaluated under explicit control conditions, not by transfer magnitude alone.

• The paper demonstrates that architectural choices in INRs govern both the magnitude of transfer learning gains and the specificity to target problem families.
• It employs controlled analytic functions and PDE benchmarks to differentiate genuine representational continuity from generic weight reuse.
• The findings indicate that while ReLU MLPs yield sharper task specificity, SIREN and Fourier-feature based models require tuning for optimal transfer performance.

Architecture-Dependent Transfer Specificity in Implicit Neural Representations

Introduction

The study entitled "Architecture Shapes Transfer Specificity in Implicit Neural Representations" (2606.06827) addresses a foundational question in scientific machine learning using implicit neural representations (INRs): to what extent do architectural choices determine not only the magnitude of transfer learning gains, but also their specificity to a target family of parameterized problems? Instead of focusing on global approximation error or task accuracy, the work develops a systematic methodology for dissecting transfer learning behaviors in coordinate networks, with attention to distinguishing genuine representational continuity from generic weight reuse.

The paper centers on three canonical INR architectures—SIREN, ReLU MLPs, and Fourier-feature-based MLPs—evaluated across analytic controlled function families and partial differential equation (PDE) solution benchmarks. Explicit statistical controls and ensemble experiments underlie the analysis, revealing architecture-dependent trade-offs between transfer magnitude and source specificity.

Methods: Experimental Protocol and Control Structure

The investigations utilize both analytic function families and PDE-based benchmarks to rigorously probe transfer learning properties under warm-start fine-tuning. The methodology provides:

• Controlled Analytic Families: Two 1D families—a geometric target approaching a cusp, and a two-parameter smooth family using damped cosine modes—serve to decouple structural transfer from random weight reuse via phase-randomized targets as null controls.
• PDE Benchmarks: Parametric settings for 1D heat, Burgers, and nonlinear Schrödinger (NLS) equations, as well as a 2D lid-driven cavity flow under varying Reynolds number, extend the benchmarks to physically meaningful regimes.
• INR Architectures: Implementations compare SIREN (periodic activation, frequency-initialized), standard ReLU MLPs, and MLPs augmented with Fourier features (random frequency embedding), employing default and hyperparameter-swept configurations.
• Transfer Protocol: Transfer advantage is operationalized as a fixed-budget ratio of scratch-trained to fine-tuned MSE, after standardized pretraining and target adaptation phases.

To distinguish structure-exploiting transfer from generic weight reuse, both random controls (analytic family) and alternate same-family source controls (PDE benchmarks) are used. Additional ablations (shuffled weights, feature map re-initialization) further clarify the sources of transfer.

Results: Architecture Shapes Both Magnitude and Specificity

1D Analytic Families

On the strictly controlled 1D geometric test,

Figure 1: Architecture-dependent transfer magnitude and specificity in the controlled 1D geometric test over 10 seeds. Fourier features achieve the highest structured transfer, whereas SIREN shows high transfer on both structured and random targets.

Fourier-feature MLPs exhibit the highest structured transfer advantage ($33.1\times$), followed by SIREN ($23.0\times$) and ReLU ($10.7\times$). However, ReLU nearly eliminates transfer on random targets ($0.41\times$), marking strong specificity, while SIREN's random-control transfer remains substantial ($14.24\times$), indicating broad, non-specific weight reuse.

Shifting to the two-parameter analytic family, the architecture ranking shifts:

Figure 2: Paired structured-versus-random transfer in the two-parameter 1D family. ReLU achieves the clearest specificity; Fourier and SIREN show less strong separation.

• ReLU achieves high structured-vs-random separation under all parameter variations.
• SIREN maintains high structured transfer, but specificity is weak: transfer to random controls is nearly as high as to structured targets.
• Fourier features lose their 1D dominance; unless frequency scale is retuned, their absolute transfer advantage is modest.

These findings emphasize that architecture governs not only the observable transfer gain, but the nature of what is being transferred—distinguishing mechanistic transfer (task-specific continuity) from indiscriminate reuse.

Scaling Law Negative Result

A scaling-law audit of the heuristic $A_\mathrm{transfer} \propto 1/\Delta t^2$ finds that, for all three architectures, the decay in transfer with parameter distance is much slower than predicted by this naive model, and the expected linear log-log relationship is strongly rejected.

Figure 4: Cross-architecture scaling-law audit; observed transfer decays more slowly than the predicted quadratic law for SIREN, ReLU, and Fourier feature architectures.

Navier–Stokes and PDE Suite: Practical Settings

In the 2D Navier–Stokes lid-driven cavity, transfer patterns broadly mirror the analytic two-parameter family:

Figure 6: Navier–Stokes benchmark—designated-source vs. alternate-source transfer. ReLU demonstrates the clearest source specificity.

• SIREN provides substantial weight reuse, but no sharp preference between source conditions.
• ReLU exhibits marked sensitivity to the correct source, achieving highest transfer for pretraining on the matched parameter.
• Fourier features provide weak or unreliable transfer unless hyperparameter-tuned.

In 1D PDE reference benchmarks, architecture rankings vary by equation:

• SIREN dominates absolute transfer in the heat equation but is less selective.
• ReLU is most source-specific overall.
• Fourier features are superior on Burgers but require feature-map co-reuse (i.e., both random projection and the weights) to realize full transfer advantage.

Static Diagnostics and Null Controls

Analysis of static indicators—activation participation ratio, Hessian sharpness, CKA—shows weak or no correlation with transfer specificity. Proper null controls (randomized targets, alternate source/parameter, shuffled weights) are essential: substantial apparent transfer can occur under weak specificity, especially in SIREN, misleading evaluations based on transfer magnitude alone.

Implications and Theoretical Considerations

The central implication is that transfer advantage and its task specificity must be disentangled and reported separately. Architecture-dependent spectral bias, parameterization, and initialization effects critically shape both.

• SIREN supports broad transfer across parameter families but with limited selectivity, consistent with its spectral properties.
• ReLU MLPs, while potentially less effective in raw transfer magnitude, enforce higher specificity, making them preferable for tasks prioritizing sharp parametric discernment.
• Fourier-feature MLPs require explicit bandwidth tuning for optimal task transfer; coupled reuse of both the random map and trained MLP parameters is necessary.

Operator learning architectures (e.g., DeepONet, FNO) remain outside the scope of this study but are complementary in their explicit parameter conditioning. The present findings highlight the necessity for architecture-aware protocol selection in scientific ML workflows involving INRs, particularly where transfer learning is deployed in parametric sweeps or reduced-order modeling.

Practical Recommendations and Future Directions

For scientific ML practitioners:

• Always control for source specificity when reporting transfer advantages.
• Prefer ReLU MLPs for sharply target-specific transfer, SIREN for broad (but possibly indiscriminate) weight reuse, and Fourier features only after explicit bandwidth optimization.
• Include null controls (independent randoms, alternate same-family sources, and weight-shuffle ablations) in benchmarking protocols.

Future work should extend protocol designs to additional architectures (e.g., multi-resolution hash INRs, PINN variants), broader PDE classes, and more comprehensive meta-learning strategies. Theoretical modeling of transfer specificity, accounting for spectral overlap and architectural priors, remains an open challenge.

Conclusion

This study establishes that architecture fundamentally determines not only transfer magnitude, but also specificity, in implicit neural representations used for parametric scientific problems. The results caution against interpreting large transfer gains as inherently meaningful without appropriate null controls, and advocate for architecture-aware, specificity-sensitive evaluation methodologies in scientific machine learning transfer studies.