Learning Compositional Latent Structure with Vector Networks

Abstract: Deep networks are powerful function approximators, but they typically store many different computations in shared weight matrices, making it difficult to selectively reuse or adapt parts of them when a familiar structure appears in novel combinations. We introduce the Vector Network (VN), a hierarchical recurrent architecture in which each layer replaces a fixed weight matrix with a library of reusable rank-1 weight atoms. For each input, VN minimizes a layer-local energy to infer a sparse set of active weight atoms and their coefficients, jointly constrained by bottom-up input reconstruction and top-down feedback consistency. These weight atom coefficients then compose an input-specific low-rank weight matrix for that sample. After convergence, slow learning updates only the selected weight atoms through local residual signals scaled by the inferred coefficients. We evaluate VN on four compositional benchmarks spanning 1D signals, 2D spatial decoding, N-body dynamics, and compositional MNIST. VN matches strong baselines in distribution while often achieving out-of-distribution error about an order of magnitude lower when familiar factors must be recombined in novel ways. Vector networks thus make compositional generalization a structural property of the architecture and inference process rather than a brittle byproduct of fitting many behaviors into one shared dense parameter substrate.

• The paper presents a novel architecture that replaces dense weight matrices with a library of rank-1 weight atoms, enabling explicit compositionality.
• It employs energy-based sparse inference and Atomic-Hebb updates to selectively update active weights, reducing entanglement and improving OOD performance.
• Empirical results across spatial, functional, dynamic, and digit composition tasks demonstrate VN’s robustness and its potential for scalable, interpretable deep models.

Learning Compositional Latent Structure with Vector Networks

Motivation and Architectural Approach

The paper introduces Vector Networks (VN), a hierarchical recurrent architecture designed to mitigate weight entanglement and enhance compositional generalization in deep networks (2605.28007). Standard deep networks often encode multiple behaviors in overlapping directions within shared dense weight matrices, which hinders selective reuse and composition of learned functions. VN replaces each layer’s weight matrix with a library of rank-1 weight atoms, each defined by learned code vector pairs. For every input, VN performs energy-based sparse inference to activate a small subset of weight atoms, thus composing a tailored, input-specific low-rank weight matrix. This inference is constrained by bottom-up reconstruction and top-down feedback consistency, and subsequent learning updates only the selected weight atoms via local residual signals scaled by inferred coefficients (Atomic-Hebb updates).

Figure 1: A VN layer composes input-specific transformations from sparse weight atoms; inference selects a minimal responsible set of rank-1 operators.

This architectural shift makes compositionality an explicit, structural property grounded in weight-space inference, rather than emerging from the implicit entanglement inherent in dense weight sharing.

Methodological Framework

VN’s foundation is the stack of rank-1 weight atoms at each layer. The architecture iteratively performs sparse inference over atom coefficients $g_{l,i}$ to minimize a layer-local energy function:

$$E_{l}(g_{l} \mid x_{l}, h_{l,\text{target}}) = \frac{1}{2}\|x_{l} - S_{l}g_{l}\|^2 + \lambda_{l}\|g_{l}\|_{1} + \frac{1}{2}\|h_{l,\text{target}} - U_{l}g_{l}\|^2$$

The process alternates between upward and downward passes: the upward pass propagates dense interface messages, while the downward pass updates atom coefficients using refreshed top-down targets. Sparse inference ensures only the most responsible atoms are active, imposing an explicit low-rank structure on the synthesized weight matrix per input.

Atomic-Hebb learning updates only those atoms in the active support, localizing plasticity and mitigating overlap between unrelated updates, thus preventing entanglement and promoting reuse.

Experimental Evaluation

VN was benchmarked on four compositional generalization tasks, each testing out-of-distribution (OOD) recombination of previously seen factors:

Spatial Composition: Novel Bump Positions

VN achieved uniform low error across both seen (ID) and unseen (OOD) spatial regions in the bump decoding task, outperforming CNN and VAE decoders, which deteriorated in the OOD region. Notably, VN’s OOD/ID error ratio remained stable as depth increased, confirming robustness against spatial weight entanglement.

Figure 2: VN exhibits stable, low spatial reconstruction error across unseen regions, unlike entangled CNN/VAEs.

Function Composition: Unseen Signal Combinations

In 1D function reconstruction, VN was intentionally sparsity-constrained and required to explain input via a competitive set of composable atoms. While sequence models (Transformer, Mamba) outperformed VN in direct reconstruction, masked-prediction settings reveal VN’s superiority: baseline models’ performance degraded severely, whereas VN maintained low error, indicating strong recovery and reuse of compositional structure—not mere interpolation.

Figure 3: VN remains robust under masked prediction in Hard-OOD regimes, outperforming standard baselines.

N-body Dynamics: Force Composition

VN was trained on single-force regimes and tested on multi-force OOD compositions with varying body counts. VN consistently achieved low error and displayed the highest robustness in OOD settings compared to MLP, GNN, Transformer, Mamba, and FiLM baselines. The log-log Pareto plot of ID versus OOD error further illustrated that VN’s generalization does not degrade as problem size changes, in contrast to baselines that memorized training-time body counts.

Figure 4: VN sustains low error under mixed-force OOD regimes and body count shifts; baselines suffer compositional collapse.

MNIST Digit and Transformation Composition

VN outperformed autoencoder baselines in ID and OOD scenarios involving digit composition and Rotation/Translation/Scaling (RTS) transformations. VN’s reconstruction error remained order-of-magnitude lower in OOD mixtures and motion tasks. Sparsity-matched AEs were consistently weaker both in ID and OOD, confirming that compositionality is not achieved via sparsity alone but by VN’s explicit weight-atom structure.

Figure 5: VN achieves superior OOD performance on digit mixtures and compositional RTS transformations.

Theoretical Implications and Extensions

The paper formalizes superposition preservation: linear sparse reconstruction guarantees exact compositional recovery if the sparse code for unseen compositions can be inferred. Sufficient conditions for this are established using classical sparse coding theory (coherence bounds), and the Atomic-Hebb update is shown to be the exact gradient of the equilibrium energy via Danskin’s theorem. VN’s rank-bound induces effective sample-specific hypothesis space constraints, promoting generalization by limiting capacity and enforcing reuse.

Top-down coupling in VN further aligns with predictive coding frameworks, integrating consistency signals from higher layers into each coefficient’s energy minimization. This facilitates hierarchical compositional inference while maintaining theoretical convergence guarantees (ISTA/FISTA-style proximal updates).

Practical Implications and Limitations

VN’s explicit compositional structure confers several advantages:

• Data Efficiency: New inputs are addressed via recomposition of learned atoms rather than memorizing each factorial combination.
• Continual Learning: Atomic responsibility localizes updates, allowing refinement of individual mechanisms without global entanglement.
• Interpretability: Sparse active atoms clarify which functional components explain each input.

However, the current VN implementation incurs computational overhead due to recurrent sparse inference. The authors suggest avenues for efficiency gains via learned support initializers and amortized inference, referencing prior work on approximation of sparse coding dynamics.

Future Directions

VN's design principle — compositionality as explicit weight-space responsibility — presents a pathway toward architectures with superior long-tail generalization and robust compositional reasoning. Extension to amortized sparse inference and integration with scalable backbone models could enable practical deployment in settings where rapid adaptation to unseen compositions is critical. Moreover, leveraging VN’s Atomic-Hebb updates for lifelong learning and task adaptation remains an open experimental and theoretical challenge.

Conclusion

Vector Networks offer a principled architectural and algorithmic solution to compositional generalization, shifting the locus of adaptivity from state routing (baseline models/attention) to sparse operator inference and weight-space responsibility. Empirical analysis demonstrates that strong ID performance does not ensure compositional robustness, and VN’s explicit structure and learning dynamics enable systematic reuse and adaptation of functional components in OOD settings. The approach sets foundational groundwork for future exploration into compositional architectures, lifelong learning, and interpretable modularity in AI.