Centroid-Based Memory Mechanism

• Centroid-based memory mechanisms are methods that use representative centroids to summarize high-dimensional data for efficient learning and decision-making.
• They are applied in tasks such as clustering, classification, and continual learning to improve scalability, interpretability, and robustness.
• Dynamic update strategies like momentum and multi-centroid selection enable these systems to adapt and maintain high performance even in noisy environments.

A centroid-based memory mechanism is a class of methods in machine learning and artificial intelligence that utilize statistical centroids—typically mean or representative points in a high-dimensional feature space—to encapsulate, abstract, or recall information relevant for learning, inference, or decision-making. Centroids function as compressed, interpretable, and computationally efficient summaries (or "memories") of data distributions, classes, episodes, or clusters. Across supervised and unsupervised tasks—including classification, clustering, continual learning, optimization, summarization, and ensemble methods—these mechanisms can improve efficiency, robustness, scalability, and interpretability.

1. Foundations of Centroid-Based Memory Mechanisms

Centroids operate as canonical representatives: for a set of vectors $\{x_i\}$ (e.g., samples from a class or cluster), a centroid is usually

$c = \frac{1}{N} \sum_{i=1}^N x_i$

This aggregate may be adapted during learning via online updates, momentum, or clustering (such as k-means), depending on the application. Centroid-based memory mechanisms are grounded in the theory that relevant information about a group can be encoded (and retrieved) via such representative statistics, supporting efficient similarity calculation, abstraction, and generalization.

Centroid memory is foundational to a spectrum of approaches:

• Prototype and prototype-inspired methods (e.g., prototypical networks, k-means clustering, multi-centroid associative memory)
• Contrastive and cluster-based representation learning (using centroids as anchors or negatives/positives)
• Ensemble decision structures (e.g., trees or forests with centroid-driven partitioning)
• Online/continual learning and unlearning (centroids as persistently updated memories or targets for erasure).

2. Algorithmic Strategies: Construction, Update, and Usage

Centroid Construction and Update

Most algorithms compute centroids as the mean of selected features or data vectors—either for the full data partition (as in clustering) or for selected sub-structures (e.g., class-specific, cluster-specific, or feature-selected subsets).

Centroid update strategies include:

• Batch mean calculation: Accumulating all current members of a class or cluster.
• Momentum updates:

$$c_t \leftarrow \omega c_{t-1} + (1-\omega) \bar{x}_t$$

with $\omega$ a smoothing hyperparameter and $\bar{x}_t$ a fresh mean, for adaptivity over time.

• Matching or assignment mechanisms: Matching queries to centroids via bipartite (Hungarian) algorithms or maximum similarity, to drive centroid refinement or selection.
• Multi-centroid representation: Allowing a sub-cluster structure within each group (e.g., multiple centroids per class) to capture multimodality or mitigate label noise.

Centroid-Based Inference and Memory Usage

Retrieval, classification, or decision processes typically use a distance metric (such as Euclidean or cosine distance) between a query (sample, feature vector) and one or more centroids:

• Nearest-centroid rule: Assign to the label of the closest centroid.
• Similarity-based averaging: Weight responses by similarities, especially in memory-augmented or attention-based models.
• Centroid-based distillation or regularization: Use geometry—intra- and inter-centroid distances—as an additional constraint for continual learning or feature space anchoring.

Some frameworks also perform dynamic selection or generation of negatives/positives for contrastive learning using centroid distributions (as in SONI negative construction or median positive selection).

3. Applications Across Learning Paradigms

Clustering, Deep Clustering, and Reclustering

Centroid-based clustering (e.g., k-means, DEC, DCN) underpins unsupervised class discovery, where the centroids are dynamically adjusted to fit latent space distributions. A documented limitation—the "reclustering barrier"—arises when embeddings overfit initial centroids and lose plasticity. The BRB algorithm addresses this by periodically applying soft weight resets and reclustering to ensure continued adaptation and prevent premature commitment, empirically overcoming the performance plateau characteristic of conventional centroid-based deep clustering (2411.02275).

Contrastive and Cluster Memory Learning

Cluster contrastive learning for tasks such as object re-identification often employs centroids as cluster memories. Individual-based updating can be sensitive to outliers; in contrast, centroid-based updating uses batch means, providing robust, stable cluster centers less affected by label noise or outlier fluctuation (2112.04662). Combining individual and centroid memories, and enforcing cross-view consistency constraints, further enhances discrimination and robustness.

The multi-centroid memory (MCM) mechanism extends this approach to allow multiple centroids per cluster, facilitating resilience to label noise (imperfect clustering) and better capturing intra-class diversity (2112.11689).

Continual Learning and Memory Distillation

Centroid-based mechanisms are central to several continual learning frameworks:

• Centroids Matching (2208.02048): Regularizes networks to preserve embedding-to-centroid distances across tasks, mitigating catastrophic forgetting without requiring rehearsal buffers.
• Centroid Distance Distillation (2303.02954): Maintains a compact memory by storing centroid distance matrices after each task and enforcing their preservation during subsequent tasks, anchoring the geometry of the latent space and reducing domain drift.
• Rehearsal-based sampling and memory cache: Dynamically constructs and maintains a centroid-aware cache of exemplars for future rehearsal, reducing sample bias.

Machine Unlearning

Centroid kinematics are leveraged in unlearning to erase specific knowledge by manipulating sample embeddings relative to class centroids. For a sample to be forgotten, its representation is pushed towards the centroid of another class and away from its original centroid, effectively removing class-specific memory (2312.02052). This operation can be made efficient and is measurable with metrics such as the Adaptive Unlearning Score (AUS).

Memory-Efficient Hardware Implementations

The MEMHD framework employs multi-centroid memory for Hyperdimensional Computing on In-Memory Computing arrays (2502.07834). By matching the number and structure of centroids to the array, it achieves 100% memory utilization, enables one-shot or few-shot associative recall, and improves both energy and inference latency. Centroids are initialized via k-means-like clustering, quantized, and updated iteratively in a quantization-aware manner for robustness and efficiency.

Tree-Based and Ensemble Learning

Centroid decision forests (CDF) (2503.19306) redefine decision boundaries in high-dimensional spaces by partitioning data via closest-class-centroid assignment, rather than axis-aligned splits. Feature selection is performed using a class separability score; splits are made by assigning points to the nearest mean of discriminative features in the current partition. This centroid-driven approach yields more geometry-aligned, robust, and interpretable ensembles, outperforming classical tree and forest methods across numerous high-dimensional datasets.

Transformer-Efficient Abstraction

Centroid attention generalizes self-attention by mapping $N$ inputs to $M$ centroids ($M \leq N$), compressing information flow via differentiable clustering (2102.08606). The result is reduced computational and memory cost ($O(NM)$ vs $O(N^2)$), while often preserving or improving downstream accuracy; empirical results demonstrate effectiveness in summarization, 3D vision, and image processing tasks.

4. Advantages and Limitations

Advantages

• Memory compression: Summarizes feature distributions via a handful of centroids, making storage and recall highly efficient.
• Robustness: Averaging mitigates the influence of outliers and label noise.
• Scalability: Multi-centroid representations can adapt allocation to class complexity; arrays and memory can be fully utilized in hardware implementations.
• Interpretability: Centroids often have an interpretable geometric or physical meaning (prototypical instance, mean feature vector).
• Speed: Enables constant or logarithmic-time lookup in various settings (e.g., associative memory, context trees), reducing reliance on linear scans.
• Flexibility: Methods are domain- and modality-agnostic, adaptable for supervised, unsupervised, or semi-supervised regimes.

Limitations

• Homogeneity assumption: Single centroids poorly represent classes/clusters with multi-modal or highly variant distributions.
• Over-commitment risk: Premature or excessive reliance on early centroid assignments can reduce adaptability, as seen in the reclustering barrier (2411.02275).
• Granularity tradeoffs: Too few centroids yield underfitting; too many approach instance-based learning with commensurate resource costs.
• Selection sensitivity: Matching strategies (e.g., moderate positive selection, k-means initialization) can significantly affect downstream accuracy and noise resilience.
• Update complexity: Maintenance of centroid memory (especially with multi-centroid or multi-head approaches) can add overhead unless properly managed or hardware-mapped.

5. Empirical Outcomes and Comparative Performance

Extensive empirical studies across domains substantiate the strengths of centroid-based memory mechanisms:

• Object Re-Identification: Centroid-based updating and multi-centroid memories show superior stability and accuracy (Market-1501 mAP up to 89.9%) compared to individual-based approaches (2112.04662, 2112.11689).
• Continual Learning: Centroid-anchored regularization and caching reduce forgetting and domain drift, with top accuracy and memory efficiency on benchmarks such as CIFAR10, CUB-200 (2208.02048, 2303.02954).
• Image and Text Tasks: In vision transformers, centroid attention reduces MACs by up to 50% while matching or exceeding baseline accuracy (2102.08606). In multi-document summarization, centroid-based pretraining improves zero/few-shot ROUGE scores (2208.01006).
• Ensemble Learning: CDF consistently outperforms random forests, SVMs, and k-NN in high-dimensional datasets, with higher accuracy and lower variance (2503.19306).
• Hardware Utilization: MEMHD achieves full AM utilization, $80 \times$ reduction in computation cycles, and up to $13.25 \times$ memory efficiency improvement over leading binary HDC approaches (2502.07834).

6. Interpretability, Adaptation, and Future Research Directions

Centroid-based mechanisms naturally align with instance-prototype theory in cognitive science and offer explicit, often visualizable, representations. t-SNE and PCA analyses across several studies reveal that centroid maintenance or manipulation produces well-separated, stable clusters and enables geometric reasoning about category memory, drift, or deletion effects.

Adaptation remains an active area:

• Continuous adaptation: Periodic centroid and weight resets (as in BRB) are essential for sustained exploration in deep clustering.
• Dynamic centroid allocation: Multi-centroid methods allow for adaptive representation based on class complexity or confusion.
• Efficient unlearning: Centroid-based kinematic manipulation enables gradient-driven erasure of selected memories, supporting privacy and bias mitigation requirements.

Potential extensions include domain-specific centroid selection criteria, hierarchical or multi-granular centroids, and integration with emerging neuromorphic or in-memory computing paradigms.

---

Centroid-based memory mechanisms thus represent a unifying and highly adaptable blueprint for memory, abstraction, and knowledge transfer in modern machine learning, supporting both theoretical insight and practical scalability across a diverse array of applications.