When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry

Abstract: To preserve previously learned representations, continual learning systems must strike a balance between plasticity, the ability to acquire new knowledge, and stability. This stability-plasticity dilemma affects how representations can be reused across tasks: shared structure enables transfer when tasks are similar but may also induce interference when new learning disrupts existing representations. However, it remains unclear when and why structural separation influences this trade-off. In this study, we examine how network architecture, task similarity, and representational dimensionality jointly shape learning in a sequential task paradigm inspired by transfer-interference studies. We compare a task-partitioned modular recurrent network with a single-module baseline by systematically varying task similarity (low, medium, high) and the scale of weight initialization, which induces different learning regimes that we empirically characterize through the effective dimensionality of the learned representations. We find that architecture has minimal impact in high-dimensional regimes where representations are sufficiently unconstrained to accommodate multiple tasks without strong interference. In contrast, in lower-dimensional (rich) regimes, architectural separation is decisive: modular networks exhibit graded alignment of task-specific subspaces with overlap for similar tasks, partial orthogonalization for moderately dissimilar tasks, and stronger separation for dissimilar tasks. This graded geometry is absent in the single network baseline. Our findings suggest that representational dimensionality acts as a key organizing variable governing when structural separation becomes functionally relevant, and highlight adaptive geometry as a central principle for designing continual learning systems.

• The paper demonstrates that modular architectures significantly reduce interference in continual learning only under low-dimensional, rich representational regimes.
• It employs a transfer-interference paradigm with task similarity and initialization scaling to reveal how geometric organization of representations is shaped.
• The study implies that adaptive control of representational dimensionality can enhance system stability and plasticity in lifelong learning.

When and How Modularity Matters in Continual Learning: The Role of Representational Dimensionality

Introduction

The paper "When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry" (2604.27656) investigates the conditional benefits of modular neural architectures in continual learning paradigms. It scrutinizes the interplay between architectural modularity, task similarity, and the representational regime defined by effective dimensionality. The central claim is that the utility of modular structure is not inherent but emerges predominantly under low-dimensional constraints, shaping the geometric organization of task representations in a similarity-dependent manner.

Experimental Paradigm and Architectures

The study adapts a transfer-interference continual learning paradigm comprising a three-phase sequence (A1 → B → A2), where Task A is first learned, then superseded by Task B (with parametrizable similarity to A), and finally Task A is revisited to measure forgetting or interference (Figure 1).

Figure 1: Schematic of the sequential continual learning protocol and architectural manipulations used to interrogate the effects of dimensionality and modularity.

Three core manipulations were systematically explored:

• Task similarity: Task B could be the same, near, or far from Task A in rule space.
• Initialization scaling: All weights were rescaled by a global factor $\gamma$, allowing transition from lazy/high-dimensional (large $\gamma$) to rich/low-dimensional (small $\gamma$) representational regimes.
• Network architecture: Comparison between (i) a baseline single recurrent network and (ii) a modular network with task-partitioned recurrent modules converging on a shared readout.

Both architectures were trained identically, enabling direct attribution of observed effects to structure and representational regime.

Behavioral Analysis: Modularity, Transfer, and Interference

Empirical results on angular accuracy, transfer (facilitation from A to B), and interference (deterioration on A after B) unveil that modular systems offer pronounced advantages only in low-dimensional, rich regimes and for dissimilar task pairs. In high-dimensional (lazy) regimes, both architectures sustain high accuracy and low interference, regardless of task similarity, suggesting that abundant representational capacity trivially buffers against catastrophic forgetting, rendering architectural biases negligible.

Figure 2: Modularity’s utility emerges primarily in low-dimensional rich regimes, with the modular architecture showing pronounced resistance to interference relative to the single network in the “far” task condition.

As $\gamma$ is reduced, a clear divergence emerges: the single network becomes increasingly vulnerable to transfer-interference trade-offs, especially in the far similarity condition, while the modular network maintains stable performance and robust memory for Task A. This numerically substantiates the central claim that structural separation is beneficial only when the representational space is constrained.

Representational Geometry: Dimensionality as a Control Variable

A central contribution lies in mapping how initialization scaling ($\gamma$) acts as a lever on representational dimensionality, itself a determinant of subspace allocation and alignment. In high-$\gamma$ regimes, both architectures exhibit high effective dimensionality and similar, weakly structured subspace geometry across task pairs, as evidenced by principal component and principal angle analyses.

Figure 3: Dimensionality quantification and principal angles confirm that only in low-dimensional settings does architectural modularity drive graded, task-similarity-dependent separation of representations.

Conversely, low-$\gamma$ (rich) regimes induce compression: representations become low-dimensional, and the modular network expresses a graded organization—aligned for similar tasks, partially orthogonal for near, and well-separated for far. The single network exhibits substantially less structure and control under these constraints, often failing to disentangle dissimilar tasks.

Qualitative 3D PCA projections further illuminate this phenomenon. In the lazy regime, trajectories span broad, unstructured regions with weak task dependence (in both architectures). In the rich regime, the modular network’s representations become compact, with clear task-dependent geometric relationships—tight overlap for same tasks, partial reorganization for near, and pronounced separation for far—underscoring the contingent value of modularity (Figure 4).

Figure 4: Geometric organization of hidden-state trajectories, rendered via 3D PCA, highlights how reduced dimensionality accentuates modular control and similarity-dependent subspace allocation.

Implications and Future Directions

The study sharply refutes the general notion that modularity categorically mitigates forgetting or interference in continual learning. Instead, functional benefits of architectural separation are contingent on representational constraints: with unconstrained, high-dimensional code, modularity’s effect is muted; under compression, structure provides an organizing prior that enables adaptive, similarity-sensitive subspace management.

Practically, this suggests that continual learning systems may benefit from adaptive mechanisms that monitor and regulate effective dimensionality and modulate modularity deployment dynamically, rather than relying on static architectural priors. The findings also advocate for viewing continual learning as a problem of adaptive representational allocation, dictated by the triad of current task, its similarity to prior knowledge, and available representational resources.

Theoretically, these results bridge the fields of computational neuroscience and artificial intelligence. They support emerging perspectives that emphasize the critical role of underlying geometry, rather than parameter isolation per se, in the mitigation of catastrophic forgetting and the facilitation of transfer. This aligns with contemporary evidence from both biological systems and artificial modular RNNs that specialization and subspace allocation are adaptive processes modulated by ecological and architectural constraints.

Future Directions

Several concrete avenues are highlighted or implied:

• Dynamic control of representational dimensionality (e.g., via explicit regularization or capacity allocation mechanisms) to mediate modularity’s expressivity.
• Extended continual learning scenarios with variable similarity structures, to quantify the adaptability of geometric organization over time.
• Systematic comparison with other forms of architectural modularity (e.g., with inter-module communication) to dissect the nuances of coordination versus separation.
• Integration of metric learning or subspace-specific regularizers to encourage similarity-dependent overlap or separation adaptively.

Conclusion

This work demonstrates that in continual learning, the representational regime—specifically the effective dimensionality of internal codes—governs when and how modular architectures confer advantages. While modularity can dramatically buffer against interference and enable graded, similarity-sensitive allocation of representational resources, such effects are dormant in high-dimensional settings with unconstrained capacity. Future AI systems capable of self-regulating their representational geometry—and thus judiciously invoking modularity—are well positioned to reconcile stability and plasticity across lifelong learning.