Fly-CL: Bio-Inspired Continual Learning

• Fly-CL is a bio-inspired framework for continual representation learning that recasts learning as similarity matching over frozen pretrained features.
• It employs sparse random expansion and top-k sparsification to mitigate prototype multicollinearity and lower computational and memory costs.
• A streaming ridge classifier with adaptive regularization enables efficient online updates and robust performance across various benchmarks.

Searching arXiv for the exact term and closely related works to ground the entry. arXiv search query: "Fly-CL" arXiv search query: "Learning on the Fly Replay-Based Continual Object Perception for Indoor Drones" Fly-CL is a fly-inspired framework for pre-trained model-based continual representation learning that targets the regime in which a backbone is nearly frozen or fully frozen and only a lightweight classifier or representation adaptation layer is updated. In this setting, continual learning is recast as a similarity-matching problem over pretrained features and class prototypes. The method is motivated by two difficulties identified in that regime: multicollinearity among class prototypes and the computational cost of stronger analytic or ridge-based baselines. Drawing on the fly olfactory circuit, Fly-CL combines sparse random expansion, top-$k$ sparsification, and streaming ridge classification with adaptive regularization in order to reduce training time while retaining strong continual-learning accuracy (Zou et al., 19 Oct 2025).

1. Problem formulation and intended setting

Fly-CL addresses continual representation learning (CRL) with pre-trained models, especially the case where the feature extractor is kept nearly fixed across tasks. The attraction of this setting is that it limits catastrophic forgetting associated with full-model updates, but its standard implementation can be brittle because classification depends on cosine similarity among prototypes derived from pretrained embeddings. The paper identifies the core issue as multicollinearity: prototypes for different classes can remain highly correlated, so their directions in embedding space are insufficiently separated for robust similarity matching (Zou et al., 19 Oct 2025).

The baseline formulation begins with an encoder $f_\theta$ that maps an input $\bm{x}_t^i$ to a feature vector

$\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$

For class $i$ in task $t$, the prototype is

$$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$

and prediction uses cosine similarity,

$$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$

The paper states that pretrained prototypes often satisfy

$$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$

which encodes the multicollinearity problem directly. In the paper’s framing, the central objective is therefore not only prototype matching but decorrelation of the representation used for matching (Zou et al., 19 Oct 2025).

A second motivation is computational. The paper situates Fly-CL against methods such as RanPAC and other analytic or ridge-based approaches that may be accurate but can remain expensive because they rely on dense random projections, repeated cross-validation over ridge parameters, matrix inversions, or iterative update procedures. Fly-CL is designed to lower both training time and memory cost while remaining compatible with a wide range of pretrained backbones (Zou et al., 19 Oct 2025).

2. Biological inspiration and architectural mapping

Fly-CL is explicitly inspired by the fly olfactory circuit, which the paper presents as a mechanism for pattern separation and decorrelation. The biological analogy has three computationally relevant components: a high-dimensional sparse expansion from projection neurons (PNs) to Kenyon cells (KCs), a competitive sparsification step akin to winner-take-all, and a downstream readout from KCs to MBONs (Zou et al., 19 Oct 2025).

The method translates this motif into a three-stage computational design. First, embeddings from the pretrained encoder are normalized. Second, they are sent through a sparse random projection into a much higher-dimensional space, followed by top-$k$ sparsification. Third, the resulting sparse codes are processed by a streaming ridge classifier with adaptive regularization. This architecture is intended to progressively reduce prototype correlation while maintaining low time complexity (Zou et al., 19 Oct 2025).

The projection stage is defined by

$f_\theta$0

where $f_\theta$1 with $f_\theta$2, each row of $f_\theta$3 has exactly $f_\theta$4 non-zero entries, and the non-zero entries are sampled from $f_\theta$5. The top-$f_\theta$6 operator keeps the $f_\theta$7 largest-magnitude components and sets the rest to zero: $f_\theta$8 In the paper’s interpretation, the random expansion improves linear separability, while top-$f_\theta$9 suppresses weak and redundant components that contribute to collinearity (Zou et al., 19 Oct 2025).

3. Streaming classifier, adaptive regularization, and inference

After sparse expansion, Fly-CL performs classification through a streaming ridge-regression readout. For task $\bm{x}_t^i$0, if $\bm{x}_t^i$1 denotes the matrix of sparse features and $\bm{x}_t^i$2 denotes one-hot labels, Fly-CL maintains the streaming statistics

$\bm{x}_t^i$3

The classifier matrix is then

$\bm{x}_t^i$4

This stage serves as the learned readout analogous to the KC $\bm{x}_t^i$5 MBON pathway and is used to stabilize classification under correlated features through $\bm{x}_t^i$6-regularization (Zou et al., 19 Oct 2025).

A key implementation issue is the choice of the ridge parameter $\bm{x}_t^i$7. Instead of exhaustive cross-validation, Fly-CL uses Generalized Cross-Validation (GCV). If

$\bm{x}_t^i$8

is the SVD of the task feature matrix, with $\bm{x}_t^i$9, then the shrinkage matrix and effective degrees of freedom are

$\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$0

The paper defines

$\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$1

and minimizes

$\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$2

over a candidate set $\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$3. The optimal parameter is

$\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$4

This choice avoids repeated large matrix inversions associated with naïve ridge-parameter selection (Zou et al., 19 Oct 2025).

For prototype calculation, the method uses Cholesky factorization rather than a generic inverse or LU decomposition: $\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$5 At test time, a sample is encoded, projected, sparsified, and classified by

$\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$6

with inference complexity reported as $\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$7 (Zou et al., 19 Oct 2025).

4. Theoretical properties and complexity profile

The paper supplies two principal theoretical arguments for the projection stage. First, it states that sparse random projection preserves rank with high probability: $\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$8 This is used to justify that sparse expansion does not collapse representational capacity despite its aggressive sparsity (Zou et al., 19 Oct 2025).

Second, top-$\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.$9 sparsification is analyzed under assumptions that the leading $i$0 coordinates capture most of the energy and that downstream loss is Lipschitz: $i$1

$i$2

The resulting bounds are

$i$3

and

$i$4

The paper concludes that if

$i$5

the error bound decays polynomially with increasing dimension (Zou et al., 19 Oct 2025).

The computational argument is equally central. A dense projection would cost

$i$6

whereas Fly-CL’s sparse projection costs

$i$7

with $i$8. Similarity computation is reduced from

$i$9

to

$t$0

because only the top-$t$1 active dimensions participate in later operations. For regularization, the paper describes the expensive step as the SVD of $t$2, with cost

$t$3

and contrasts this with a full grid-search cross-validation strategy of roughly

$t$4

The Cholesky-based solve further lowers factorization cost from

$t$5

to

$t$6

while improving numerical stability (Zou et al., 19 Oct 2025).

5. Experimental evaluation, datasets, and quantitative results

The empirical study evaluates Fly-CL with ViT-B/16 and ResNet-50 backbones on CIFAR-100, CUB-200-2011, VTAB, ImageNet-R, and ImageNet-A. The main task settings are $t$7 for CIFAR-100 with 10 classes per task, $t$8 for CUB-200-2011 with 20 classes per task, and $t$9 for VTAB with 10 classes per task. Longer-sequence experiments double the number of tasks, and severe domain-shift experiments use ImageNet-R and ImageNet-A with $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$0 and 20 classes per task. Baselines include L2P, DualPrompt, EASE, RanPAC, and F-OAL. The reported metrics are Average accuracy at stage $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$1,

$$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$2

Overall accuracy,

$$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$3

and the efficiency metrics $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$4 and $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$5, where $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$6 isolates post-extraction algorithmic cost (Zou et al., 19 Oct 2025).

On ViT-B/16, the paper reports the following representative results. On CIFAR-100, Fly-CL reaches $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$7 s with $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$8, while the best baseline accuracy is RanPAC at $$\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,$$9; the paper characterizes this as about 91% lower post-extraction time versus the most efficient baseline, with about 0.32% accuracy drop. On CUB-200-2011, Fly-CL achieves $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$0 s and $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$1, improving over RanPAC’s $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$2 by 1.17% while cutting post-extraction time by about 83%. On VTAB, Fly-CL reaches $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$3 s and $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$4, improving over F-OAL’s $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$5 by 2.38% and reducing post-extraction time by about 67% (Zou et al., 19 Oct 2025).

On ResNet-50, Fly-CL remains competitive. The paper reports $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$6 with $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$7 s on CIFAR-100, $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$8 with $$\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.$$9 s on CUB-200-2011, and $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$0 with $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$1 s on VTAB. The corresponding improvements over the best baseline are reported as 1.89%, 1.53%, and 1.20%, respectively. The paper specifically notes that F-OAL degrades on ResNet-50 because of iterative error accumulation, whereas Fly-CL does not (Zou et al., 19 Oct 2025).

The method is also evaluated in online CIL. On ViT-B/16, the paper reports $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$2 s and $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$3 on CIFAR-100, $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$4 s and $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$5 on CUB-200-2011, and $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$6 s and $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$7 on VTAB. Under longer task sequences, Fly-CL is reported to exceed prior state of the art by about 0.54%, 1.21%, and 1.58% on CIFAR-100, CUB-200-2011, and VTAB, while reducing post-extraction training time by about 89%, 74%, and 59% versus the most efficient baselines. Under severe domain shift, Fly-CL reaches $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$8 with $$\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,$$9 s on ImageNet-R and $k$0 with $k$1 s on ImageNet-A; the ImageNet-R comparison is explicitly contrasted with RanPAC’s $k$2 s post-extraction time (Zou et al., 19 Oct 2025).

The paper also isolates the sources of efficiency. On CUB-200-2011, the optimized random projection reduces runtime from 0.22 s to 0.08 s, ridge selection from 7.34 s to 0.14 s, prototype calculation from 0.20 s to 0.10 s, and similarity comparison from 0.21 s to 0.08 s. Default hyperparameters are $k$3, $k$4, and $k$5. The reported sensitivity study states that accuracy increases with $k$6 and saturates after about 10,000, improves with $k$7 unless $k$8 is too small, and exhibits a broad plateau for $k$9. Reported peak memory is 6.7 GB for CIFAR-100, 4.6 GB for CUB-200-2011, and 4.3 GB for VTAB (Zou et al., 19 Oct 2025).

6. Relation to adjacent work, naming ambiguity, and stated limitations

Fly-CL belongs to a broader family of fruit-fly-inspired machine-learning methods. A notable antecedent is FlyNN/FlyNNFL, which reprograms FlyHash and the Fly Bloom Filter (FBF) to emulate nearest-neighbor classification in federated learning with one communication round and optional $f_\theta$00-differential privacy (Ram et al., 2021). The relationship is conceptual rather than algorithmic: both lines of work treat the fly olfactory circuit as a source of sparse expansion and efficient readout, but Fly-CL is formulated for continual representation learning with frozen pretrained backbones, whereas FlyNN/FlyNNFL is formulated for federated nearest-neighbor classification (Ram et al., 2021).

The name Fly-CL is also used in a distinct context for replay-based continual object perception for indoor drones. In “Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones,” Fly-CL denotes a continual object perception benchmark built around the UAV-IndoorCL dataset and a class-incremental detection protocol for resource-constrained indoor UAVs. That work evaluates replay strategies such as ER, MIR, and FAR, with FAR performing best under 5% and 10% replay budgets (Nae et al., 13 Feb 2026). This overlap in naming means that the term is context-sensitive: in one usage it denotes a bio-inspired continual representation learning framework, and in another it denotes a continual object perception benchmark and study for indoor drones.

The paper on Fly-CL (Zou et al., 19 Oct 2025) also states clear limits on scope. It focuses on representation-based continual learning with frozen pretrained models, so it is not presented as a general solution to all continual-learning settings. It further notes that the random projection layer “may not be fully random biologically” and suggests that a learnable or constrained projection layer could improve adaptability. For implementation, the paper reports code availability, use of timm for pretrained backbones, architecture-specific normalization—standard normalization to $f_\theta$01 for ViT-B/16 and ImageNet normalization for ResNet-50—and experiments on Ubuntu 20.04.4 LTS, an Intel Xeon Platinum 8358P CPU, and 8 × NVIDIA RTX 3090 GPUs with CUDA 11.7 (Zou et al., 19 Oct 2025).

Taken in its principal sense, Fly-CL is therefore best understood as a progressive decorrelation pipeline for continual learning on top of pretrained models: frozen feature extraction, sparse random expansion, top-$f_\theta$02 suppression, and streaming ridge classification with efficient regularization. Its significance lies in showing that a biologically motivated design can be translated into a concrete algorithmic recipe with explicit complexity reductions, strong empirical performance across transformer and CNN backbones, and low post-extraction latency in both standard and online continual-learning regimes (Zou et al., 19 Oct 2025).