--- title: Dynamic Few-Shot Learning url: https://www.emergentmind.com/topics/dynamic-few-shot-learning type: topic --- # Dynamic Few-Shot Learning Dynamic few-shot learning refers to a class of techniques in machine learning designed to enable models to rapidly generalize to novel classes with only a small number of labeled examples, while adapting model parameters, metrics, or representations on-the-fly as new data or classes arrive. The "dynamic" aspect encompasses both real-time adaptation at inference and internal mechanisms that flexibly adjust learned representations, metrics, and classifiers according to the support data provided per task or episode. The primary goals include efficient learning of new categories, retention of previously acquired knowledge (i.e., avoiding catastrophic forgetting), and scaling to a wide range of few-shot scenarios beyond fixed-shot, fixed-way settings. ## 1. Architectural Principles of Dynamic Few-Shot Learning Dynamic few-shot learning frameworks typically comprise three central components: a feature extractor, a mechanism for dynamically generating classifier or metric parameters, and a means for rapidly incorporating new classes without extensive retraining. - **Feature extractor**: Models such as convolutional neural networks (CNNs) or transformers are pre-trained on a large base dataset to encode inputs into feature vectors. - **Dynamic classifier or metric**: - In "Dynamic Few-Shot Visual Learning without Forgetting," the classifier is a cosine similarity module parameterized by a set of class weight vectors $W^*$, with test-time expansion by dynamically generated weights for novel classes [1804.09458]. - In TADAM, the metric is made task-dependent via conditioning feature activations through Feature-wise Linear Modulation and scaling distances by a learnable, dynamically-tuned parameter [1805.10123]. - **Dynamic parameter/generator modules**: Examples include attention-based few-shot classification weight generators, dynamic meta-filters for channel- and spatial-specific adaptation, and memory routing modules in text models [1804.09458, 2103.13582, 2005.05727]. This dynamic architecture distinguishes itself from static metric learning or few-shot models, which require retraining or are fixed at test time. ## 2. Dynamic Adaptation Mechanisms A central attribute of dynamic few-shot learning is the capacity to construct, adapt, or fuse model modules in response to new tasks or data. - **Dynamic classifier weight generation**: The attention-based generator $G$ in [1804.09458] creates per-class weights for new categories by aggregating support features and attending to base-class weights. This allows the immediate extension of the classifier with novel weight vectors at inference, unifying the recognition of both base and novel categories. - **Task-/instance-conditioned representations**: - TADAM modifies feature extraction via task embeddings computed from the current support set and modulates convolutional activations dynamically for each episode [1805.10123]. - Instance- and task-aware dynamic convolutional kernels (INSTA) adapt each convolutional layer for new tasks and individual instances, decomposing filters into spatial and multi-spectral channel branches [2112.03494]. - **Dynamic alignment and adaptation**: Meta-filters in [2103.13582] align query and support features via position- and channel-specific filters, where filter parameters are predicted per position and per episode, and alignment depth is adaptively controlled by a Neural ODE solver. - **Memory and routing**: Dynamic memory induction and routing networks update latent memory capsules on-the-fly using routing-by-agreement algorithms, conditioning the adaptation of class prototypes on both the support set and queries [2005.05727]. These dynamic mechanisms facilitate per-task or per-instance flexibility, enabling the model to accommodate rapidly changing support data and classes. ## 3. Optimization and Training Regimes Dynamic few-shot learners are typically meta-trained using episodic optimization and extensions of meta-learning principles. - **Episodic meta-training**: Sampling N-way K-shot episodes, support and query sets are constructed, and adaptation or dynamic parameter generation is performed within each episode. - **Multi-stage training**: For example, [1804.09458] uses a two-stage process, first training a feature extractor and base-class weights, then meta-training a generator to produce classifier weights for “fake” novel classes, using only base data. - **Auxiliary objectives**: Co-training with standard classification tasks regularizes dynamic modules, as in TADAM which introduces an auxiliary classification objective to stabilize the learning of the Task Embedding Network [1805.10123]. - **Regularization for stability-plasticity**: Techniques such as the Static-Dynamic Collaboration (SDC) [2601.08448] and Mamba-FSCIL [2407.06136] decouple the learning of base and novel classes using static and dynamic projector branches, with appropriate regularization to avoid forgetting. These meta-training procedures enable the system to simulate the test-time dynamics of few-shot adaptation and optimize the relevant modules for fast and robust task adaptation. ## 4. Evaluation Protocols and Empirical Results Dynamic few-shot learning methods are evaluated by their ability to generalize to novel classes (few-shot) without degrading performance on base classes (no forgetting), across standard benchmarks. - **Benchmark datasets**: Mini-ImageNet, tiered-ImageNet, FC100, CUB, and domain-specific datasets such as the modified Jester gesture dataset [1804.09458, 1805.10123, 2212.08363]. - **Metrics**: 1-shot and 5-shot recognition accuracy (usually 5-way), base class retention, and mean average precision for detection tasks. - **Empirical results**: The dynamic few-shot learning system of [1804.09458] achieves 74.9% (5-shot) and 58.6% (1-shot) accuracy on Mini-ImageNet, surpassing contemporaneous metric and matching networks—while retaining ~70.9% base accuracy with zero degradation. TADAM achieves 76.7% (5-shot, 5-way) on Mini-ImageNet, up to a 14% improvement over unscaled baseline metrics [1805.10123]. Approaches such as SDC-FSCIL and Mamba-FSCIL outperform static and prior dynamic approaches in class-incremental settings [2601.08448, 2407.06136]. Notably, dynamic few-shot mechanisms have also yielded notable sample savings in time-series tasks, e.g., dynamic hand gesture recognition achieves up to 88.8% accuracy (5-way, 5-shot) with reduction of up to 1200 labeled sequences compared to conventional retrained models [2212.08363]. ## 5. Stability, Plasticity, and Forgetting A principal challenge for dynamic few-shot learners is balancing the retention of prior (base) class knowledge (stability) with adaptability to novel classes (plasticity). - **No-forgetting property**: The model in [1804.09458], by employing a cosine-similarity classifier and dynamic weight generator, enables the addition of new classes at test time without harming base-class accuracy. - **Static-dynamic architectures**: Methods like SDC and Mamba-FSCIL explicitly separate static (frozen) and dynamic (adaptable) modules, tuning the mixing coefficient between static and dynamic projections (α) to achieve state-of-the-art retention–adaptation tradeoff [2601.08448, 2407.06136]. - **Class-sensitive adaptation**: In Mamba-FSCIL, dual selective-SSM projectors are used—one frozen for base classes, and one dynamically updated for novel classes—with extra losses ensuring that dynamic adaptation minimally impacts base-class representations [2407.06136]. Empirically, these architectures demonstrate reduced catastrophic forgetting relative to static networks or purely plastic dynamic networks. ## 6. Representative Dynamic Learning Strategies across Modalities Dynamic few-shot learning has been instantiated in a range of architectures and modalities: - **Metric scaling and task conditioning**: E.g., global and local modulation of the feature extractor via episode-specific embeddings and scaling of similarity/distance functions [1805.10123]. - **Dynamic alignment/filtering**: Position- and channel-specific convolutional meta-filters modulate spatial and channel-wise features per query-support pair, enabling fine-grained adaptation [2103.13582]. - **Dynamic graph inference**: In few-shot object detection, dynamic GCNs built from correlations among support and query features propagate relevance signals for improved class representation [2108.02235]. - **Dynamic memory/routing**: Capsule-style dynamic routing modules guide the aggregation and induction of class prototypes for new tasks in text classification [2005.05727]. - **Dynamic prompting**: In PromptAL, dynamic prompts are generated for each unlabeled input, adjusting the decision boundary of the model before computing acquisition metrics for few-shot active learning [2507.16424]. - **Dynamic input assembly**: Models that assemble the network structure dynamically per episode, e.g., dynamically constructing the number of pairwise relational computations for variable shot counts [1708.06819]. These strategies are unified by their goal of providing on-demand, data-driven model flexibility at test time. ## 7. Limitations and Open Problems Despite significant empirical advances, several limitations remain: - **Computational complexity**: Some dynamic modules (e.g., capsule routing, dynamic filter generation) can introduce significant per-episode overhead. - **Hyperparameter sensitivity**: Model performance can depend critically on the tuning of meta-learning rates, mixing coefficients, or the architecture of dynamic generator modules. - **Sample efficiency in ultra-light settings**: Highly parameter-efficient modes (e.g., AgileNet's ultra-light adaptation) may underfit if novel classes are dissimilar from the training distribution [1805.08311]. - **Scalability to large-way, large-shot, or continual open-set environments**: While extensions exist (e.g., SDC, Mamba-FSCIL for FSCIL), dynamic few-shot learning in large-scale, dynamic real-world environments remains a challenge. - **Out-of-distribution adaptation and retrieval quality**: In dynamic few-shot prompting (e.g., DFSL for KGQA), retrieval strategies strongly impact cross-domain generalization [2407.01409]. Continued research is required to extend dynamic few-shot adaptation to new domains, optimize resource use, and understand principles of stability, plasticity, and dynamic modularity. --- **References**: - Dynamic Few-Shot Visual Learning without Forgetting [1804.09458] - Task dependent adaptive metric for improved few-shot learning [1805.10123] - Fast Learning of Dynamic Hand Gesture Recognition with Few-Shot Learning Models [2212.08363] - TGDM: Target Guided Dynamic Mixup for Cross-Domain Few-Shot Learning [2210.05392] - Learning Dynamic Alignment via Meta-filter for Few-shot Learning [2103.13582] - Divide and Conquer: Static-Dynamic Collaboration for Few-Shot Class-Incremental Learning [2601.08448] - Dynamic Few-Shot Learning for Knowledge Graph Question Answering [2407.01409] - Dynamic Relevance Learning for Few-Shot Object Detection [2108.02235] - PromptAL: Sample-Aware Dynamic Soft Prompts for Few-Shot Active Learning [2507.16424] - Learning Instance and Task-Aware Dynamic Kernels for Few Shot Learning [2112.03494] - Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning [2407.06136] - Dynamic Memory Induction Networks for Few-Shot Text Classification [2005.05727] - Unsupervised Meta-Learning via Dynamic Head and Heterogeneous Task Construction for Few-Shot Classification [2410.02267] - Dynamic Input Structure and Network Assembly for Few-Shot Learning [1708.06819] - Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning [2104.08808] - AgileNet: Lightweight Dictionary-based Few-shot Learning [1805.08311] Source: https://www.emergentmind.com/topics/dynamic-few-shot-learning