---
title: 'Fly-CL: Bio-Inspired Continual Learning'
url: https://www.emergentmind.com/topics/fly-cl
type: topic
---

# Fly-CL: Bio-Inspired Continual Learning

Searching arXiv for the exact term and closely related works to ground the entry.
arXiv search query: "Fly-CL"
arXiv search query: "Learning on the Fly Replay-Based Continual Object Perception for Indoor Drones"
Fly-CL is a fly-inspired framework for **pre-trained model-based continual representation learning** that targets the regime in which a backbone is **nearly frozen** or fully frozen and only a lightweight classifier or representation adaptation layer is updated. In this setting, continual learning is recast as a **similarity-matching** problem over pretrained features and class prototypes. The method is motivated by two difficulties identified in that regime: **multicollinearity among class prototypes** and the **computational cost** of stronger analytic or ridge-based baselines. Drawing on the fly olfactory circuit, Fly-CL combines sparse random expansion, top-\(k\) sparsification, and streaming ridge classification with adaptive regularization in order to reduce training time while retaining strong continual-learning accuracy [2510.16877].

## 1. Problem formulation and intended setting

Fly-CL addresses **continual representation learning (CRL)** with **pre-trained models**, especially the case where the feature extractor is kept nearly fixed across tasks. The attraction of this setting is that it limits catastrophic forgetting associated with full-model updates, but its standard implementation can be brittle because classification depends on cosine similarity among prototypes derived from pretrained embeddings. The paper identifies the core issue as **multicollinearity**: prototypes for different classes can remain highly correlated, so their directions in embedding space are insufficiently separated for robust similarity matching [2510.16877].

The baseline formulation begins with an encoder \(f_\theta\) that maps an input \(\bm{x}_t^i\) to a feature vector
\[
\bm{v}_t^i=f_\theta(\bm{x}_t^i)\in \mathbb{R}^d.
\]
For class \(i\) in task \(t\), the prototype is
\[
\bm{\mu}_t^i = \frac{1}{N_i} \sum_{j=1}^{|\mathcal{D}_t|}\mathbb{I}(y_t^j=i)f_\theta(\bm{x}_t^j)\in\mathbb{R}^d,
\]
and prediction uses cosine similarity,
\[
\hat{y} = \arg\mathop{\max}\limits_{t,i} \frac{\bm{v}^\top \bm{\mu}_t^i}{\lVert\bm{v}\rVert \cdot \lVert\bm{\mu}_t^i\rVert}.
\]
The paper states that pretrained prototypes often satisfy
\[
\mathbb{E}[\bm{\mu}_{t_i}^{m_i\top}\bm{\mu}_{t_j}^{m_j}] \gg 0,
\]
which encodes the multicollinearity problem directly. In the paper’s framing, the central objective is therefore not only prototype matching but **decorrelation** of the representation used for matching [2510.16877].

A second motivation is computational. The paper situates Fly-CL against methods such as **RanPAC** and other analytic or ridge-based approaches that may be accurate but can remain expensive because they rely on dense random projections, repeated cross-validation over ridge parameters, matrix inversions, or iterative update procedures. Fly-CL is designed to lower both **training time** and **memory cost** while remaining compatible with a wide range of pretrained backbones [2510.16877].

## 2. Biological inspiration and architectural mapping

Fly-CL is explicitly inspired by the **fly olfactory circuit**, which the paper presents as a mechanism for **pattern separation** and **decorrelation**. The biological analogy has three computationally relevant components: a high-dimensional sparse expansion from **projection neurons (PNs)** to **Kenyon cells (KCs)**, a competitive sparsification step akin to **winner-take-all**, and a downstream readout from **KCs to MBONs** [2510.16877].

The method translates this motif into a three-stage computational design. First, embeddings from the pretrained encoder are normalized. Second, they are sent through a **sparse random projection** into a much higher-dimensional space, followed by **top-\(k\)** sparsification. Third, the resulting sparse codes are processed by a **streaming ridge classifier** with adaptive regularization. This architecture is intended to progressively reduce prototype correlation while maintaining low time complexity [2510.16877].

The projection stage is defined by
\[
\bm{h}'=Z(\bm{v}) = \text{top-}k(\bm{h}) = \text{top-}k\left(\bm{W}\bm{v}\right),
\]
where \(\bm{W} \in \mathbb{R}^{m \times d}\) with \(m \gg d\), each row of \(\bm{W}\) has exactly \(p\) non-zero entries, and the non-zero entries are sampled from \(\mathcal{N}(0,1)\). The top-\(k\) operator keeps the \(k\) largest-magnitude components and sets the rest to zero:
\[
[\bm{h}']_i= \begin{cases} [\bm{h}]_i & \text{if the magnitude of } [\bm{h}]_i \text{ is among the top-$k$ values of } \bm{h}, \\ 0 & \text{otherwise.} \end{cases}
\]
In the paper’s interpretation, the random expansion improves linear separability, while top-\(k\) suppresses weak and redundant components that contribute to collinearity [2510.16877].

## 3. Streaming classifier, adaptive regularization, and inference

After sparse expansion, Fly-CL performs classification through a streaming ridge-regression readout. For task \(t\), if \(\bm{H}_t\in \mathbb{R}^{n_t\times m}\) denotes the matrix of sparse features and \(\bm{Y}_t \in \{0,1\}^{n_t\times c_t}\) denotes one-hot labels, Fly-CL maintains the streaming statistics
\[
\bm{G}_t\leftarrow \bm{G}_{t-1} + \bm{H}_t^\top \bm{H}_t, \qquad \bm{S}_t\leftarrow \bm{S}_{t-1} + \bm{H}_t^\top \bm{Y}_t.
\]
The classifier matrix is then
\[
\bm{C}_t = (\bm{G}_t + \lambda \bm{I}_m)^{-1} \bm{S}_t.
\]
This stage serves as the learned readout analogous to the **KC \(\to\) MBON** pathway and is used to stabilize classification under correlated features through \(\ell_2\)-regularization [2510.16877].

A key implementation issue is the choice of the ridge parameter \(\lambda\). Instead of exhaustive cross-validation, Fly-CL uses **Generalized Cross-Validation (GCV)**. If
\[
\bm{H}_t = \bm{U}_t \bm{\Sigma}_t \bm{V}_t^\top
\]
is the SVD of the task feature matrix, with \(\bm{\Sigma}_t=\mathrm{diag}(s_1,\dots,s_r)\), then the shrinkage matrix and effective degrees of freedom are
\[
\bm{D}_t = \frac{\bm{\Sigma}_t^2}{\bm{\Sigma}_t^2+\lambda \bm{I}_r}, \quad \mathrm{df}(\lambda) = \mathrm{tr}(\bm{D}_t)=\sum_{i=1}^r \frac{s_i^2}{s_i^2 + \lambda}.
\]
The paper defines
\[
\hat{\bm{Y}_t = \bm{U}_t(\mathrm{vecdiag}(\bm{D}_t)\otimes\bm1_c^\top)\odot\bm{U}_t^\top\bm{Y}_t
\]
and minimizes
\[
\mathrm{GCV}(\lambda)=\frac{\lVert\bm{Y}_t - \bm{\hat{Y}_t(\lambda)\rVert_F^2}{n_t \left(1 - \frac{\mathrm{df}(\lambda)}{n_t}\right)^2}
\]
over a candidate set \(\Lambda\). The optimal parameter is
\[
\lambda_t^* = \arg\min_{\lambda \in \Lambda} \mathrm{GCV}(\lambda).
\]
This choice avoids repeated large matrix inversions associated with naïve ridge-parameter selection [2510.16877].

For prototype calculation, the method uses **Cholesky factorization** rather than a generic inverse or LU decomposition:
\[
\bm{L}_t\bm{L}_t^\top = \bm{G}_t + \lambda_t^*\bm{I}_m, \qquad \bm{C}_t = \bm{L}_t^{-\top}(\bm{L}_t^{-1}\bm{S}_t).
\]
At test time, a sample is encoded, projected, sparsified, and classified by
\[
\hat{y}=Z(\bm{v})^\top\bm{C}_t,
\]
with inference complexity reported as \(\mathcal{O}(kc_t)\) [2510.16877].

## 4. Theoretical properties and complexity profile

The paper supplies two principal theoretical arguments for the projection stage. First, it states that sparse random projection preserves rank with high probability:
\[
\mathbb{P}\left(|\det(\mathcal{W})| \geq \left(\frac{p}{d}\right)^{d/2} \sqrt{d!} \exp({-d^{1/2+\epsilon})\right)=1-o(1).
\]
This is used to justify that sparse expansion does not collapse representational capacity despite its aggressive sparsity [2510.16877].

Second, top-\(k\) sparsification is analyzed under assumptions that the leading \(k\) coordinates capture most of the energy and that downstream loss is Lipschitz:
\[
\mathbb{E}\left[\frac{\sum_{i=1}^k\bm{h}_i^2}{\lVert\bm{h}\rVert_2^2}\right]\geq1-\frac{C}{k},
\]
\[
|L(\bm{h}, y)-L(\bm{h}', y)|\le M\cdot\lVert\bm{h}-\bm{h}'\rVert.
\]
The resulting bounds are
\[
\mathbb{E}\left[\lVert\bm{h}-\bm{h}'\rVert_2^2\right]\le\frac{C}{k}\cdot\mathbb{E}\left[\lVert\bm{h}\rVert_2^2\right]
\]
and
\[
\mathbb{E}\left[|L(\bm{h}, y)-L(\bm{h}', y)|\right]\le M \cdot\sqrt{\frac{C}{k}\cdot\mathbb{E}[\lVert\bm{h}\rVert_2^2]}.
\]
The paper concludes that if
\[
k = \Omega(m^\alpha), \quad 0 < \alpha < 1,
\]
the error bound decays polynomially with increasing dimension [2510.16877].

The computational argument is equally central. A dense projection would cost
\[
\mathcal{O}(mn_td),
\]
whereas Fly-CL’s sparse projection costs
\[
\mathcal{O}(mn_tp),
\]
with \(p \ll d\). Similarity computation is reduced from
\[
\mathcal{O}(mn_tc_t)
\]
to
\[
\mathcal{O}(kn_tc_t),
\]
because only the top-\(k\) active dimensions participate in later operations. For regularization, the paper describes the expensive step as the SVD of \(\bm{H}_t\), with cost
\[
\mathcal{O}(n_trm),
\]
and contrasts this with a full grid-search cross-validation strategy of roughly
\[
\mathcal{O}(lm^3).
\]
The Cholesky-based solve further lowers factorization cost from
\[
\mathcal{O}\left(\frac{2}{3}m^3\right)
\]
to
\[
\mathcal{O}\left(\frac{1}{3}m^3\right)
\]
while improving numerical stability [2510.16877].

## 5. Experimental evaluation, datasets, and quantitative results

The empirical study evaluates Fly-CL with **ViT-B/16** and **ResNet-50** backbones on **CIFAR-100**, **CUB-200-2011**, **VTAB**, **ImageNet-R**, and **ImageNet-A**. The main task settings are \(T=10\) for CIFAR-100 with 10 classes per task, \(T=10\) for CUB-200-2011 with 20 classes per task, and \(T=5\) for VTAB with 10 classes per task. Longer-sequence experiments double the number of tasks, and severe domain-shift experiments use ImageNet-R and ImageNet-A with \(T=10\) and 20 classes per task. Baselines include **L2P**, **DualPrompt**, **EASE**, **RanPAC**, and **F-OAL**. The reported metrics are **Average accuracy** at stage \(t\),
\[
A_t=\frac{1}{t}\sum_{i=1}^ta_{t,i},
\]
**Overall accuracy**,
\[
\bar{A}=\frac{1}{T}\sum_{i=1}^T A_t,
\]
and the efficiency metrics \(\tau_{\text{train}}\) and \(\tau_{\text{post}}\), where \(\tau_{\text{post}}\) isolates post-extraction algorithmic cost [2510.16877].

On **ViT-B/16**, the paper reports the following representative results. On CIFAR-100, Fly-CL reaches \(\tau_{\text{post}}=5.38\) s with \(\bar{A}=93.89\%\), while the best baseline accuracy is RanPAC at \(94.21\%\); the paper characterizes this as about **91%** lower post-extraction time versus the most efficient baseline, with about **0.32%** accuracy drop. On CUB-200-2011, Fly-CL achieves \(\tau_{\text{post}}=0.35\) s and \(\bar{A}=93.84\%\), improving over RanPAC’s \(92.67\%\) by **1.17%** while cutting post-extraction time by about **83%**. On VTAB, Fly-CL reaches \(\tau_{\text{post}}=0.34\) s and \(\bar{A}=96.54\%\), improving over F-OAL’s \(94.68\%\) by **2.38%** and reducing post-extraction time by about **67%** [2510.16877].

On **ResNet-50**, Fly-CL remains competitive. The paper reports \(\bar{A}=84.61\%\) with \(\tau_{\text{post}}=5.25\) s on CIFAR-100, \(\bar{A}=80.25\%\) with \(\tau_{\text{post}}=0.44\) s on CUB-200-2011, and \(\bar{A}=94.00\%\) with \(\tau_{\text{post}}=0.34\) s on VTAB. The corresponding improvements over the best baseline are reported as **1.89%**, **1.53%**, and **1.20%**, respectively. The paper specifically notes that **F-OAL** degrades on ResNet-50 because of iterative error accumulation, whereas Fly-CL does not [2510.16877].

The method is also evaluated in **online CIL**. On ViT-B/16, the paper reports \(\tau_{\text{post}}=12.57\) s and \(\bar{A}=92.96\%\) on CIFAR-100, \(\tau_{\text{post}}=2.33\) s and \(\bar{A}=92.59\%\) on CUB-200-2011, and \(\tau_{\text{post}}=1.09\) s and \(\bar{A}=96.38\%\) on VTAB. Under longer task sequences, Fly-CL is reported to exceed prior state of the art by about **0.54%**, **1.21%**, and **1.58%** on CIFAR-100, CUB-200-2011, and VTAB, while reducing post-extraction training time by about **89%**, **74%**, and **59%** versus the most efficient baselines. Under severe domain shift, Fly-CL reaches \(\bar{A}=83.19\%\) with \(\tau_{\text{post}}=0.21\) s on ImageNet-R and \(\bar{A}=67.98\%\) with \(\tau_{\text{post}}=0.15\) s on ImageNet-A; the ImageNet-R comparison is explicitly contrasted with RanPAC’s \(67.71\) s post-extraction time [2510.16877].

The paper also isolates the sources of efficiency. On CUB-200-2011, the optimized random projection reduces runtime from **0.22 s** to **0.08 s**, ridge selection from **7.34 s** to **0.14 s**, prototype calculation from **0.20 s** to **0.10 s**, and similarity comparison from **0.21 s** to **0.08 s**. Default hyperparameters are \(m=10{,}000\), \(p=300\), and \(k=3{,}000\). The reported sensitivity study states that accuracy increases with \(m\) and saturates after about **10,000**, improves with \(p\) unless \(p\) is too small, and exhibits a broad plateau for \(k\). Reported peak memory is **6.7 GB** for CIFAR-100, **4.6 GB** for CUB-200-2011, and **4.3 GB** for VTAB [2510.16877].

## 6. Relation to adjacent work, naming ambiguity, and stated limitations

Fly-CL belongs to a broader family of fruit-fly-inspired machine-learning methods. A notable antecedent is **FlyNN/FlyNNFL**, which reprograms **FlyHash** and the **Fly Bloom Filter (FBF)** to emulate nearest-neighbor classification in federated learning with one communication round and optional \((\epsilon,0)\)-differential privacy [2112.07157]. The relationship is conceptual rather than algorithmic: both lines of work treat the fly olfactory circuit as a source of sparse expansion and efficient readout, but Fly-CL is formulated for **continual representation learning with frozen pretrained backbones**, whereas FlyNN/FlyNNFL is formulated for **federated nearest-neighbor classification** [2112.07157].

The name **Fly-CL** is also used in a distinct context for **replay-based continual object perception for indoor drones**. In “Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones,” Fly-CL denotes a continual object perception benchmark built around the **UAV-IndoorCL** dataset and a class-incremental detection protocol for resource-constrained indoor UAVs. That work evaluates replay strategies such as **ER**, **MIR**, and **FAR**, with **FAR** performing best under **5%** and **10%** replay budgets [2602.13440]. This overlap in naming means that the term is context-sensitive: in one usage it denotes a **bio-inspired continual representation learning framework**, and in another it denotes a **continual object perception benchmark and study for indoor drones**.

The paper on Fly-CL [2510.16877] also states clear limits on scope. It focuses on **representation-based continual learning with frozen pretrained models**, so it is not presented as a general solution to all continual-learning settings. It further notes that the random projection layer “may not be fully random biologically” and suggests that a **learnable or constrained projection layer** could improve adaptability. For implementation, the paper reports code availability, use of **timm** for pretrained backbones, architecture-specific normalization—standard normalization to \([-1,1]\) for **ViT-B/16** and **ImageNet normalization** for **ResNet-50**—and experiments on **Ubuntu 20.04.4 LTS**, an **Intel Xeon Platinum 8358P CPU**, and **8 × NVIDIA RTX 3090 GPUs** with **CUDA 11.7** [2510.16877].

Taken in its principal sense, Fly-CL is therefore best understood as a **progressive decorrelation pipeline** for continual learning on top of pretrained models: frozen feature extraction, sparse random expansion, top-\(k\) suppression, and streaming ridge classification with efficient regularization. Its significance lies in showing that a biologically motivated design can be translated into a concrete algorithmic recipe with explicit complexity reductions, strong empirical performance across transformer and CNN backbones, and low post-extraction latency in both standard and online continual-learning regimes [2510.16877].

Source: https://www.emergentmind.com/topics/fly-cl