GrapNet: A Programmable Dynamic-Architecture Neural Graph Substrate

Abstract: Programmability is a missing first-class interface in fixed-tensor neural networks: editing a relation, freezing a subgraph, auditing a local function, or changing the execution backend should be an operation on the neural program rather than ad-hoc parameter surgery. GrapNet studies this graph-as-network setting. The graph is the architecture and executable program, not an input data graph. Each compute node owns its next-layer child references and a trainable allocation vector aligned with those references; deleting a relation physically removes both the child reference and the corresponding allocation coordinate. Structural rules and execution policies live outside the node core, so the same child-owned graph can be grown, frozen, structurally edited, grouped into trainable family blocks, routed by attention over active relations, or lowered to dense snapshots after topology stabilizes. GrapNet composes with conventional modules through a vector-valued parent interface: dense layers, CNN encoders, ResNet feature extractors, attention blocks, and transformer representations can all feed one sensory GrapNode per coordinate. The evaluation is organized as a programmability stress suite rather than as a new replay benchmark. In a matched ten-seed Split Fashion-MNIST study, a plastic GrapNet+ER head reaches 63.16 percent seen-class accuracy versus 51.08 percent for a parameter-larger dense MLP+ER under the same seen-class loss and replay memory, with paired delta 12.08 points and p=1.3e-5. On Split CIFAR-10 with a frozen ImageNet ResNet-18 encoder, the same substrate improves the online head over MLP-256 by 3.81 points, with p=0.0026. These results support GrapNet as an editable neural graph substrate whose core value is structural programmability with faithful execution views.

• The paper introduces a programmable, dynamic neural graph substrate that treats the network architecture as an editable, executable structure.
• It demonstrates algebraic equivalence with traditional dense models, ensuring precise gradient and parameter fidelity during training.
• Empirical results show improved continual learning, efficient structural editing, and accelerated execution through dynamic policy controls.

GrapNet: A Programmable Dynamic-Architecture Neural Graph Substrate

Introduction and Motivation

The paper introduces GrapNet, an editable, dynamic-architecture neural network substrate that treats the network itself as an explicitly programmable graph structure, rather than relying on static, globally parameterized tensor modules. This paradigm addresses inherent inflexibility in dense tensor-based neural networks, which cannot directly support structural edits, runtime program changes, or targeted parameter management without brittle parameter surgery.

Unlike traditional GNNs, which operate on data graphs with shared edge functions, GrapNet encodes the neural architecture itself as an executable object-level graph. Each computational node owns its references to next-layer child nodes and the associated, trainable allocation vectors. Relation addition or deletion is an atomic edit on this explicit representation—removing the both the execution reference and the associated training parameter—rather than indirectly manipulating dense matrices through masks or ad-hoc parameter reshaping.

Figure 1: Child-owned deletion versus matrix masking. GrapNet’s edit operation deletes both the child reference and the aligned allocation coordinate, in contrast to masked dense matrices where disabled relations persist in storage and optimizer state.

Architectural Design and Formal Properties

GrapNet's core abstraction is the canonical, child-owned node relation. Each GrapNode maintains local state, next-layer children, and the aligned allocation weights. The design enforces strict storage invariants: for each outgoing relation, the node owns both the reference and the trainable parameter, allowing local edits without costly global bookkeeping or indirect mapping. Node deletion, relation pruning, and optimizer refresh become explicit, first-class operations.

Two key technical propositions are substantiated:

• Local Storage Invariant and Edit Scope: Edits are restricted to the child list and allocation vector of the owner node; no global tensor surgery is required.
• Materialized Matrix and Gradient Equivalence: For any fixed topology, GrapNet's child-owned execution is algebraically equivalent (for both forward and reverse-mode AD) to a sparse or dense layer—guaranteeing parameter- and gradient-level fidelity with canonical architectures.

These invariants allow GrapNet to instantiate any feed-forward MLP or Transformer FFN as a special case by densely connecting each GrapNode, and the authors provide empirical audits showing exact numerical equivalence (max errors on the order of $10^{-7}$) in both outputs and gradients between GrapNet and matrix-based implementations.

Execution Model and Policy Flexibility

A distinctive aspect of GrapNet is the clean separation between canonical architectural representation and execution policy. The graph substrate stores the editable, node-owned network; external controllers and policies implement structural growth, freezing, pruning, node family-grouped execution, and attention-style routing.

• Family Execution: For dynamic acceleration, nodes with matched child patterns are grouped into "families," enabling fast batched computation and efficient gradient propagation.
• Dense Snapshots and Compiled Kernels: For stabilized topologies, GrapNet supports lowering the architecture into traditional dense matrices or compiled CPU kernels, with external controllers maintaining correspondence and execution efficiency.
• Attention Mechanisms: Input-dependent routing (i.e., attention) is realized as a dynamic policy over the child-owned graph, supporting soft/hard masking and programmable structural adaptation at execution time.

All gradients propagate directly through live, canonical parameters, ensuring policy-agnostic compatibility with standard learning objectives and optimizers.

Empirical Evaluation: Programmability Stress Tests

The evaluation suite is structured to empirically validate programmability and editable architecture claims, using continual- and online-learning tasks as stressors in addition to standard performance metrics.

• Structural Localization: Targeted deletion of task-owned relations on Split Fashion-MNIST directly and causally ablates only the associated function, with exact drops in task-specific accuracy and preservation of non-targeted performance. This demonstrates addressable, interpretable, and causally effective architecture editing.
• Continual Learning on Split Fashion-MNIST and CIFAR-10: With matched replay streams and parameter counts, a dynamic GrapNet head achieves 63.16% seen-class accuracy on Fashion-MNIST, a 12.08 point improvement ($p=1.3\times10^{-5}$) over a larger, dense MLP head. On Split CIFAR-10 using a frozen ResNet-18 encoder, GrapNet plastic achieves a 3.81 point gain over a dense MLP-256 head ($p=0.0026$). These results are robust to ablation by freezing, which isolates old hidden units and exposes the trade-off between plasticity and structural memory.
• Tabular Covariance Editing: On UCI tabular data, GrapNet supports structural pruning driven by feature-activation covariance, physically deleting relations and nodes, with negligible or slightly positive effects on accuracy. This showcases generality beyond image-centric benchmarks.
• Execution Policy Audits: Family acceleration yields 1.75–1.85x training runtime reductions; dense snapshot lowering provides up to 38.1x forward speedup post-edit. Attention policy over the canonical relation set produces numerically identical gradients and outputs pre- and post-edit.

Implications and Prospects

GrapNet provides a canonical, policy-driven interface for neural architecture programmability. The explicit, child-owned relation invariant enables new forms of runtime program manipulation: clean insertion/deletion, structural freezing, optimizer-aware parameter refresh, backend rebuilding, and input-dependent execution—all without ad-hoc parameter hacking. This architecture separates core graph structure from execution backend, facilitating research in dynamic neural program induction, continual and lifelong learning, and hybrid static-dynamic system design.

The empirical pattern emerging across domains is that structural programmability and replay drive improvements in global seen-class calibration and forgetting, while freezing introduces explicit capacity-memory trade-offs. These capacity- and edit-exposed axes can be decoupled and manipulated for downstream continual learning, transfer learning, and modular composition.

Potential theoretical extensions involve hybridizing GrapNet's granular edit protocol with task-conditioned architectures, dynamic module routing, or even neural program synthesis, where the network explicitly maintains and edits its own program structure. Practically, GrapNet's interface anticipates future systems where neural architectures are agents, edited, programmed, and interpreted as first-class programs rather than opaque weight containers.

Conclusion

GrapNet establishes a rigorous, editable network substrate organized as a canonical child-owned computation graph. The architecture exposes explicit, policy-driven programmability and supports seamless composition with dense, convolutional, and attention-based modules. Across a range of continual learning and dynamic editing protocols, GrapNet demonstrates both functional parity with matrix-based implementations and significant gains in class-incremental calibration and local structural addressability. The substrate's core value is the stable, executable, and editable graph interface, which decouples network structure from execution, and enables new research directions in dynamic, interpretable, and programmable neural networks.