Cognitively Inspired Self-Supervised Learning

• Cognitively inspired self-supervised learning schemes are techniques that mimic hippocampal and neocortical processes to form and update feature prototypes.
• They employ centroid-based concept learning to incrementally assimilate new information while preventing catastrophic forgetting.
• These methods use self-supervision and pattern separation, enabling efficient few-shot and continual learning in data-limited environments.

A cognitively inspired self-supervised learning scheme is an approach to representation learning, continual learning, or adaptation which draws direct inspiration from models and mechanisms found in cognitive neuroscience, such as concept learning in the hippocampus and neocortex, episodic memory integration and pattern separation, or biologically motivated forms of clustering and feature abstraction. Typically, these methods design architectures or processes that mimic human learning characteristics, including incremental assimilation of new concepts, resistance to catastrophic forgetting, and the ability to learn robust categories from few samples or in data-limited regimes. Unlike purely engineering-driven schemes, cognitively inspired self-supervised learning seeks to implement mechanistic analogues of cognitive functions or memory systems within machine learning frameworks, and to do so using unsupervised or internally generated supervisory signals.

1. Cognitive and Biological Foundations

Cognitively inspired self-supervised learning explicitly incorporates principles from human memory and concept formation. Notably, research has modeled inspiration from the hippocampus and neocortex—systems that support humans in integrating new knowledge without erasing old memories, through mechanisms such as memory integration and pattern separation. In this paradigm:

• Memory integration: When a new sensory experience (e.g., an image) is similar to a concept already known, it is incrementally assimilated, updating an internal "prototype" or "centroid" representing that category.
• Pattern separation: When the new experience is distinct from known concepts, a new prototype is created; information is encoded as unique.

Such models emulate how the hippocampus generates and refines concepts by comparing incoming episodes to existing memories—integrating when similar, separating when novel. This stands in contrast to classic neural networks, which typically suffer from catastrophic forgetting when trained incrementally (2002.12411).

2. Methodological Principles: Centroid-Based Concept Learning

A core methodological example is Centroid-Based Concept Learning (CBCL), which embodies cognitively inspired self-supervised learning for few-shot and incremental regimes. In CBCL:

• Class Representation: Each class is represented by a set of centroids in feature space, not by storing all training instances.
• Centroid Creation and Update: When a new data point's feature vector is sufficiently close to an existing centroid (within threshold $D$), the centroid is updated using a weighted mean:

$C_{new} = \frac{w_C \times C_{old} + x}{w_C + 1}$

where $C_{old}$ is the current centroid, $x$ is the new feature, and $w_C$ is the current centroid's sample count.

• Pattern Separation: If the feature is too distant from current centroids, a new centroid is created, capturing intra-class variability.
• Memory Independence: Each class’s centroids are updated independently; adding new classes does not affect previously learned centroids, which structurally avoids catastrophic forgetting.
• Self-supervised characteristics: Centroid adjustment and addition operate without external supervision (beyond class identity). The process clusters abstract features into concepts, analogously to how humans internalize categories, despite only minimal annotation.

This method departs from traditional deep neural networks that require all prior data to be available for retraining, and from memory replay mechanisms that store raw samples.

3. Handling Catastrophic Forgetting

A haLLMark of cognitively inspired schemes is resistance to catastrophic forgetting—the tendency of standard neural networks to overwrite prior knowledge when exposed to new classes sequentially. In centroid-based approaches:

• Centroids for previously learned classes are not altered when new classes are introduced.
• During inference, all previously established centroids participate in classification through a voting or nearest neighbor mechanism, combining information over all learned classes.
• Classification employs a distance-weighted vote across the $n$ nearest centroids:

$$\text{Pred}(y) = \sum_{j=1}^n \frac{1}{\mathrm{dist}(x, c_j)} \cdot [y_j = y]$$

To normalize for class imbalance, votes can be divided by the number of centroids for each class:

$$\text{Pred}(y)' = \frac{1}{N_y} \cdot \text{Pred}(y)$$

By compartmentalizing updates and allowing only local changes within the representation of each class, catastrophic forgetting is effectively mitigated (2002.12411).

4. Incremental and Few-Shot Learning: Evaluation and Results

CBCL was evaluated on benchmarks including Caltech-101, CUBS-200-2011, and CIFAR-100, using both standard incremental and few-shot incremental regimes. Key findings:

• Incremental batches: CBCL outperformed state-of-the-art regularization and rehearsal-based methods—including LWM, LWF-MC, iCaRL, EEIL, and BiC—especially when old data could not be stored.
• Few-Shot Incremental Learning (FSIL): With as few as 5 or 10 examples per class, CBCL achieved high incremental accuracy, surpassing baselines that retrained on complete data at each increment.
• Practical consequence: The method allows continual adaptation to novel class data, robust retention of earlier knowledge, and operational efficiency in data-sparse, real-world incremental settings.

The results indicate that cognitively inspired clustering-based learning schemes can be successfully scaled beyond toy problems, bridging laboratory cognitive theory and practical continual learning.

5. Mathematical Foundation and Implementation

CBCL and similar schemes are characterized by mathematically explicit local update and voting rules:

• Centroid Update:

$$C_{new} = \frac{w_C \times C_{old} + x_i^y}{w_C + 1}$$

• Classification (Voting) with $n$ nearest centroids:

$$\text{Pred}(y) = \sum_{j=1}^n \frac{1}{\mathrm{dist}(x, c_j)} \cdot [y_j = y]$$

• Class balance normalization:

$$\text{Pred}(y)' = \frac{1}{N_y} \cdot \text{Pred}(y)$$

• Centroid budget reduction (for memory constraints):

$$N_y^{*\text{(new)}} = N_y^* \cdot \left(1 - \frac{K_r}{K_t}\right)$$

where $K_r$ is the number of excess centroids and $K_t$ the total before reduction.

These formulations yield interpretable, modular implementations that are amenable to practical deployment and well-aligned with cognitive process analogues.

6. Context and Relationship to Broader Self-Supervised Learning

Although primarily “cognitively inspired,” centroid-based incremental learning demonstrates clear self-supervised traits:

• After feature extraction, instance grouping into centroids utilizes only intrinsic similarity structure—no explicit supervision is needed aside from class partition identity.
• Centroid creation and updating function as an internal “self-labeling” form of self-supervision, fostering efficient concept learning, especially in low-sample scenarios.
• This mechanism shares foundational goals with other self-supervised architectures: leveraging natural structure in learned representations (clustering nearby instances), local adaptation, and minimal reliance on externally provided labels.

These features position cognitively inspired self-supervised schemes as a distinctive subdomain connecting neuroscience theory, machine learning methodology, and practical advances in incremental and data-scarce learning.

7. Implications, Limitations, and Applications

The cognitively inspired self-supervised paradigm offers several real-world benefits:

• Data efficiency: Enables robust class incremental and few-shot learning without large labeled datasets.
• Resilience: Structure inherently guards against forgetting, reducing the need for replay buffers or continual retraining.
• Computational practicality: Centroid computations and updates are efficient and can be scaled; memory management is explicit (e.g., maximum centroid budget with reduction rules).
• Broad applicability: Can be adapted to streaming learning, resource-constrained or embedded learning environments, and continual robotics perception.

Potential limitations include:

• Feature extractor reliance: CBCL depends on the quality of the underlying feature representations (often requiring pre-trained convolutional networks).
• Centroid memory management: If intra-class variation is excessive and memory constraints are tight, reducing centroids could limit expressivity.
• Threshold sensitivity: The choice of clustering distance threshold $D$ can impact centroid proliferation and classification accuracy; tuning is necessary for new domains.

In sum, cognitively inspired self-supervised learning schemes provide an interpretable, efficient, and neurobiologically plausible foundation for continual, incremental, and few-shot learning, achieving practical advances in machine learning by mirroring established models of human concept formation (2002.12411).