Meta Cyclical Annealing Schedule

• Meta Cyclical Annealing Schedule is a meta-regularization strategy that cyclically modulates distribution-matching strength to mitigate meta-amortization error in few-shot learning.
• It employs a linear within-cycle annealing coefficient with an MMD penalty to preserve latent information and prevent collapse compared to static KL regularization.
• Empirical results on Omniglot and mini-ImageNet demonstrate that MCA enhances classification accuracy by alternating free latent exploration with soft regularization.

Meta Cyclical Annealing Schedule (MCA) is a meta-regularization strategy introduced to alleviate meta-amortization error in amortized Bayesian meta-learning frameworks. Designed primarily for few-shot learning scenarios, MCA cyclically modulates the strength of a distribution-matching regularization term—typically between query- and support-conditioned task posteriors—using a linear within-cycle annealing coefficient. By annealing the regularizer over multiple cycles and leveraging the Maximum Mean Discrepancy (MMD) statistic as a divergence surrogate, MCA preserves informative latent code usage and prevents the collapse endemic to traditional Kullback–Leibler (KL) regularization. Empirical results on Omniglot and mini-ImageNet demonstrate substantial gains in classification accuracy and latent information preservation when MCA and MMD are instantiated jointly with Neural Processes (NPs) or variants such as VERSA (Hayashi et al., 2020).

1. Motivation: Information Preference and Posterior Collapse

Amortized Bayesian meta-learners—exemplified by Neural Processes and VERSA—perform task adaptation via an amortization network $q_{\lambda}(\phi|\cdot)$ that infers task-specific latent variables $\phi$ from few-shot task support sets. To regularize inference across tasks, standard practice is to penalize the discrepancy between the inferred $\phi$-distribution conditioned on new (query) inputs and the one conditioned solely on the support set, usually via a KL divergence. However, this regularization often induces the "information preference problem," in which the learned posterior collapses to a single point or the prior, resulting in low mutual information between latent codes and targets. When KL-regularization is constant and strong, the decoder ceases to use the latent variable for predictive purposes. This effect is especially harmful in few-shot learning, where the ability to encode diverse task adaptations is essential.

2. Cyclical Annealing Schedule: Formalism

Inspired by cyclical annealing strategies in Variational Autoencoders, MCA replaces a static regularization coefficient with a dynamically modulated annealing factor. The total meta-training duration $E$ is partitioned into $C$ cycles of $\ell=E/C$ epochs each. Within each cycle, the annealing coefficient $\alpha_e\in[0,1)$ at epoch $e$ is defined as: $\alpha_e = \frac{e \bmod \ell}{\ell}$ where $e \bmod \ell \in [0,\ell)$. Every cycle, $\phi$0 increases linearly from 0 to just below 1, then resets, implementing repeated phases of "free" latent exploration ($\phi$1) and "soft" regularization ($\phi$2) rather than one monotonic ramp-up.

3. Meta-Regularization Objective with MMD

MCA employs a meta-regularized objective in which the primary predictive loss over meta-tasks is combined with an MMD regularization term weighted by the cycle-dependent annealing factor: $\phi$3 where

$\phi$4

with kernel $\phi$5. Substituting MMD for the intractable KL-divergence enables principled regularization in high-dimensional latent spaces characteristic of meta-learning setups. The support-conditioned and query-conditioned posteriors, $\phi$6, are amortized across tasks.

4. MCA Training Procedure

The MCA training loop interleaves cyclical annealing with meta-batch sampling and stochastic optimization. At each epoch $\phi$7, the annealing weight $\phi$8 is computed, a mini-batch of meta-tasks is processed, and the total loss combines predictive and MMD regularization. Encoder $\phi$9 and amortizer $\phi$0 parameters are updated using Adam, with task-specific latents sampled for both support and query distributions. The procedural outline is as follows: $\phi$5 This cyclical weighting ensures the latent codes remain non-degenerate and information-rich throughout training.

5. Mechanisms for Preventing Collapse and Meta-Amortization Error

The repeated reset of the annealing coefficient in MCA allows the meta-learner's latent space to avoid local minima associated with latent collapse. During low-$\phi$1 (early cycle) phases, the amortizer and decoder are unconstrained and the task-specific latent $\phi$2 can acquire high mutual information with targets. As $\phi$3 increases, the regularizer softly aligns posteriors across support and query. Oscillating this process over many cycles encourages robustness to collapse, prevents trivial solutions, and mitigates meta-amortization error. Theoretical decomposition shows the regularization introduces both a mutual-information penalty and a marginal-matching component, with cycles balancing these objectives and preserving non-trivial task-specific representations.

6. Experimental Results and Empirical Analysis

Experiments on Omniglot (1623 classes, 20-way one-/five-shot partition, 600 test episodes) and mini-ImageNet (100 classes, 5-way protocol) establish the effectiveness of MCA. Standard architectures employ a 4-block convolutional encoder and Adam optimizer (learning rate $\phi$4). MCA is evaluated alone and in combination with MMD and NPs or VERSA. Results show:

Dataset/Protocol	VERSA	MCA + NPs	MCA + MMD + NPs
Omniglot 20-way 1-shot	97.66 ± 0.29	99.73 ± 0.12	99.81 ± 0.14
Omniglot 20-way 5-shot	98.77 ± 0.18	99.90 ± 0.21	99.90 ± 0.17
mini-ImageNet 5-way 1s	53.40 ± 1.82	77.37 ± 1.67	76.67 ± 1.46
mini-ImageNet 5w 5s	67.37 ± 0.86	90.83 ± 0.88	91.78 ± 0.89

Ablation studies confirm the necessity of both cyclical annealing and MMD regularization. Removing either, or both, results in degenerate latents, as visualized in 2D projections. This suggests that the complementary effects of cyclical relaxation and closed-form distribution matching are required for robust meta-inference.

7. Summary and Implications

Meta Cyclical Annealing Schedule alternates periods of "free" latent learning and "soft" distributional regularization, enabled by cyclical annealing and the adoption of a tractable MMD penalty in place of KL-divergence (Hayashi et al., 2020). This approach substantially reduces the meta-amortization gap, stabilizes latent code usage, and advances performance on benchmark few-shot classification tasks. A plausible implication is that cyclical regularization schedules and MMD-based divergence surrogates could generalize to other domains where amortization error and latent code degeneracy threaten representation learning efficacy.