ImageHD: Energy-Efficient On-Device Continual Learning of Visual Representations via Hyperdimensional Computing

Abstract: On-device continual learning (CL) is critical for edge AI systems operating on non-stationary data streams, but most existing methods rely on backpropagation or exemplar-heavy classifiers, incurring substantial compute, memory, and latency overheads. Hyperdimensional computing (HDC) offers a lightweight alternative through fast, non-iterative online updates. Combined with a compact convolutional neural network (CNN) feature extractor, HDC enables efficient on-device adaptation with strong visual representations. However, prior HDC-based CL systems often depend on multi-tier memory hierarchies and complex cluster management, limiting deployability on resource-constrained hardware. We present ImageHD, an FPGA accelerator for on-device continual learning of visual data based on HDC. ImageHD targets streaming CL under strict latency and on-chip memory constraints, avoiding costly iterative optimization. At the algorithmic level, we introduce a hardware-aware CL method that bounds class exemplars through a unified exemplar memory and a hardware-efficient cluster merging strategy, while incorporating a quantized CNN front-end to reduce deployment overhead without sacrificing accuracy. At the system level, ImageHD is implemented as a streaming dataflow architecture on the AMD Zynq ZCU104 FPGA, integrating HDC encoding, similarity search, and bounded cluster management using word-packed binary hypervectors for massively parallel bitwise computation within tight on-chip resource budgets. On CORe50, ImageHD achieves up to 40.4x (4.84x) speedup and 383x (105.1x) energy efficiency over optimized CPU (GPU) baselines, demonstrating the practicality of HDC-enabled continual learning for real-time edge AI.

• The paper introduces a novel FPGA-accelerated continual learning system that integrates an INT8-quantized MobileNetV2 with hyperdimensional encoding to process visual data on-device.
• It employs a hardware-efficient kMeans++ clustering method for bounded memory management, achieving up to 40.4x lower latency and significant energy efficiency gains compared to conventional baselines.
• Experimental results validate that the aggressive quantization and simplified cluster management yield minimal accuracy loss while enabling real-time, edge-grade lifelong learning.

ImageHD: FPGA-Accelerated, Hyperdimensional On-Device Continual Learning for Visual Data

Introduction and Motivation

Continual learning (CL), the capacity for incremental adaptation to evolving data distributions without episodic retraining, is necessary for edge AI deployments that face streaming visual inputs with strict latency and resource constraints. Traditional approaches based on backpropagation or exemplar-based replay are hindered by high memory, compute, and latency overheads, significantly limiting their deployability on edge hardware. Hyperdimensional computing (HDC) offers an alternative due to its lightweight, non-iterative algebraic updates, but contemporary HDC-based continual learning systems still suffer from complex memory management and inefficient hardware footprints. The paper "ImageHD: Energy-Efficient On-Device Continual Learning of Visual Representations via Hyperdimensional Computing" (2604.21280) introduces a domain-specific FPGA accelerator to address these challenges, targeting fully streaming, quantized, and hardware-bounded on-device learning for visual workloads.

System and Algorithm Co-Design

Hybrid CNN-HDC Pipeline

ImageHD couples an INT8-quantized MobileNetV2 feature extractor with a streaming HDC encoding pipeline. The CNN front-end yields a compact feature embedding, which is then mapped to a binary hypervector using positional and level lookup tables with fixed binding/bundling operations. The downstream HDC learner conducts similarity search and prototype management, admitting new clusters or amalgamating with existing ones in a fully online fashion, eschewing any backward optimization or unbounded replay.

Bounded Memory with Unified Exemplar Management

Distinct from prior HDC continual learners (e.g., LifeHD), which utilize multi-tier (short-term/long-term) prototypes and spectral clustering for global memory consolidation, ImageHD implements a single, finite cluster memory realized entirely on chip. Memory growth is upper-bounded by a hardware-optimized cluster merge unit, which consolidates prototypes using a linear-time kMeans++ algorithm exploiting the quasi-orthogonality of hypervector representations. This design obviates the need for costly cubic-complexity spectral clustering (which involves eigen decompositions), drastically reducing control complexity and memory requirement.

Hardware-Efficient Clustering and Quantization

Instead of spectral methods, the kMeans++ merge in ImageHD uses Top-M sampling to initialize segregated cluster centroids, followed by a small number of HV-space refinement iterations (majority bundling and assignment based on Hamming distance). The resilience of HDC to quantization artifacts is leveraged to deploy a fully INT8 feature extractor, reducing the model size and memory bandwidth without compromising unsupervised clustering fidelity.

Accelerator Microarchitecture

Streaming Dataflow and Fine-Grained Parallelism

The implementation targets the AMD Zynq ZCU104 FPGA, mapping the complete learning pipeline onto a dataflow architecture. The design consists of dedicated units: Pointwise and Depthwise Convolution Units for CNN inference, a Hyperdimensional Encoding Unit (HEU) for stream-aligned feature-hypervector mapping, a Hyperdimensional Classifier Unit (HLU) for similarity search and cluster updates, and a hardware-optimized Cluster Merge Unit (CMU).

Figure 1: Overview of ImageHD accelerator, illustrating sequential dataflow from the quantized CNN feature extractor to hyperdimensional encoding, classification, and bounded cluster management.

Each submodule is optimized for word-packed binary operations, exposing parallelism both along feature and hypervector dimensions (using register banking and chunk-based streaming), with minimal global reduction and latency.

Streaming CNN Subsystem

The CNN front-end implements layer-wise pipelined, tile-based inverted residual blocks. Local row/sliding-window buffer strategies allow for efficient implementation of depthwise convolutions, keeping data on-chip and enabling high-throughput processing of feature maps.

Figure 2: Streaming architecture of the inverted residual block, enabling pipelined convolutional processing with low latency and minimal off-chip bandwidth.

Hyperdimensional Encoding and Cluster Management

The HEU leverages feature-level and hypervector-level parallelism to accelerate feature-to-hypervector mapping with concurrency in both table lookup and bitwise binding/bundling operations.

Figure 3: Hyperdimensional Encoding Unit (HEU), mapping quantized feature vectors to binary hypervectors through parallel, word-packed operations.

Cluster management, including similarity search and prototype update, is undertaken by the HLU, which uses parallel PE mesh architectures for bitwise distance computation and majority thresholding. When cluster count exceeds capacity, the CMU performs the hardware-optimized kMeans++ merge.

Figure 4: Cluster Merge Unit (CMU), the primary hardware engine for in-place cluster consolidation using a streaming, FPGA-efficient kMeans++ algorithm.

End-to-End Compute Flow

The entire learning pipeline is maintained in a streaming fashion, from convolutional feature extraction to hypervector encoding to cluster admission/merge, with minimal stalling and bounded memory accesses.

Figure 5: Compute flow of the on-device continual learning accelerator, integrating all specialized engines for seamless dataflow.

Experimental Results and Numerical Analysis

Accuracy and Quantization Impact

Post-training INT8 quantization of the CNN front-end did not degrade, and in certain cases slightly improved, unsupervised clustering accuracy (ACC, NMI) compared to full-precision baselines, confirming HDC’s inherent resilience to quantized—potentially noisy—feature inputs. Across CIFAR-10, CIFAR-100, and CORe50 datasets, the reduction in purity was minor and offset by a 3.7x reduction in model size.

Latency and Energy Efficiency

On the CORe50 dataset, ImageHD achieves up to 40.4x lower end-to-end latency and 383x higher energy efficiency compared to an AMD EPYC 7763 CPU baseline, and 4.84x and 105.1x improvement over an NVIDIA A4000 GPU, respectively. Similar gains are observed against edge-class Jetson Orin platforms. These metrics incorporate the entire online CL pipeline (feature extraction, encoding, class management, and merge).

Bounded, Hardware-Efficient Clustering

Replacing spectral clustering with kMeans++ in the prototype merge phase yields 8-9x speedup on GPU (algorithmic effect), and up to 145.7x total speedup on FPGA (algorithmic + architectural), with only negligible reduction in clustering accuracy, supporting the claim that distance-based consolidation suffices due to the linear separability of high-dimensional binary spaces in HDC.

Figure 6: Execution time comparison between spectral clustering (GPU) and ImageHD’s hardware-efficient kMeans++ on GPU and FPGA, highlighting the drastic reduction in consolidation runtime on FPGA.

The unified exemplar design reduces worst-case memory usage for cluster storage by 33% relative to prior multi-tier memories, critical for edge deployment.

Implications and Future Prospects

The explicit algorithm-architecture co-design demonstrated in ImageHD establishes a viable blueprint for high-throughput, low-energy, on-device CL in visual settings with strict hardware budgets. By conclusively bounding memory growth and eschewing iterative, power-hungry procedures (backpropagation, replay, eigen-decomposition), ImageHD broadens the class of applications suitable for deployment at the edge—including robotics, low-power vision systems, and privacy-aware inference on mobile hardware.

Theoretically, the findings support the hypothesis that binary HDC representations with simple cluster management suffice for continual adaptation in high-dimensional visual domains—questioning the necessity of deep, repeatedly-fine-tuned backbones for certain resource-constrained use cases. Practically, the streaming, pipelined hardware architecture and aggressive quantization strategies set a new standard for edge-grade, real-time lifelong learning.

Future developments could investigate adaptive quantization schedules, more sophisticated novelty detection/admittance criteria, or integration of modest supervised signals in mixed-labeled streaming environments. Extension to broader modalities (audio, text) and multi-modal settings, leveraging unified hyperdimensional representations, is another promising avenue.

Conclusion

ImageHD demonstrates the feasibility and efficiency of end-to-end FPGA acceleration for unsupervised continual visual learning using hyperdimensional computing. Through joint algorithmic and hardware innovations—bounded clustering, streaming quantized CNNs, and highly-parallel bitwise operations—the system achieves strong clustering fidelity alongside radical reductions in latency and power. These results endorse HDC as a leading paradigm for real-time, lifelong learning at the edge, with significant implications for the design of future adaptive AI systems in resource-limited contexts.