CRINN: RL Optimization & Continual Invariant Learning

• CRINN is a reinforcement learning framework that optimizes approximate nearest neighbor search code via module-by-module contrastive RL and GRPO using execution-derived rewards.
• Empirical evaluations demonstrate significant gains, such as up to +85.25% on MNIST-784, while also revealing limitations on certain angular benchmarks.
• Informally, CRINN also denotes a continual invariant risk minimization approach that employs variational Bayes and ADMM to mitigate catastrophic forgetting in sequential settings.

CRINN most directly denotes "CRINN: Contrastive Reinforcement Learning for Approximate Nearest Neighbor Search", a framework for automatically optimizing approximate nearest neighbor search algorithms using a reinforcement learning-augmented LLM (Li et al., 4 Aug 2025). In some informal usage, however, “CRINN” is also used to refer to the continual invariant learning framework developed in "Continual Invariant Risk Minimization", although that paper does not introduce CRINN as an official acronym and instead names its main methods C-BVIRM, C-VIRMv1, and C-VIRMG (Alesiani et al., 2023). The term therefore has both an official systems-oriented meaning in ANNS and a non-official shorthand meaning in continual invariant learning.

1. Referential scope and nomenclature

The term has two distinct research uses, only one of which is official.

Referent	Status	Paper
Contrastive Reinforcement Learning for Approximate Nearest Neighbor Search	Official acronym	"CRINN: Contrastive Reinforcement Learning for Approximate Nearest Neighbor Search" (Li et al., 4 Aug 2025)
Continual Invariant Risk Minimization framework	Informal usage only	"Continual Invariant Risk Minimization" (Alesiani et al., 2023)

In the ANNS literature, CRINN is the paper’s explicit method name and title. In the continual invariant learning literature, the relevant paper is titled Continual Invariant Risk Minimization, and the main named method is C-BVIRM rather than CRINN. This distinction matters because the two works address different technical problems: one concerns automated optimization of HNSW-based approximate nearest-neighbor search implementations, and the other concerns out-of-distribution generalization when environments arrive sequentially.

A common source of confusion is therefore terminological rather than conceptual. In the first case, CRINN is a named framework centered on contrastive RL, GRPO, and execution-derived reward. In the second, “CRINN” is best understood as an informal label for a broader continual IRM framework built around variational Bayes, bilevel invariance constraints, and an ADMM-based solver.

2. CRINN as reinforcement learning for ANNS optimization

In its official usage, CRINN is a framework for automatically optimizing approximate nearest neighbor search (ANNS) algorithms using a reinforcement learning (RL)-augmented LLM (Li et al., 4 Aug 2025). The paper motivates this setting by noting that ANNS has become “increasingly critical” for retrieval-augmented generation (RAG), agent-based LLM applications, and massive vector databases. The optimization target is not a wholly new ANN family; rather, CRINN starts from an existing implementation—GLASS in the reported experiments—and improves it progressively.

The system treats ANNS code optimization itself as an RL problem. The agent is the LLM policy $\pi_\theta$, which generates candidate code implementations for an ANNS module. The environment is the execution-and-evaluation pipeline in which generated code is integrated, run, and benchmarked. The action space is effectively the space of program edits / synthesized code implementations under fixed interface constraints, and the prompt acts as the observation by providing Task Description, Previous Implementations with Speed, Generation Protocol, and Critical Requirements.

The reward is based on execution speed, but the paper does not use raw QPS in isolation. Because ANNS performance depends jointly on QPS and recall, CRINN sweeps the search control parameter $ef$, obtains multiple $(\text{QPS}, \text{recall})$ points, filters to recall range $[0.85, 0.95]$, and computes the area under the QPS-recall curve over that range. A faithful summary of the reward definition is

$$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$

The prompt also imposes hard constraints: search quality must match the reference implementation exactly, the same interface must be preserved via build_index() and search(), and results must be deterministic and reproducible. According to the prompt constraints, failure to maintain search accuracy yields a score of $0$.

The paper optimizes three HNSW-related components module by module sequentially: graph construction, search, and refinement. It explicitly states: “We treat each module in ANSS as independent and optimize module by module sequentially using contrastive RL.” This modularization is central to the framework’s practical scope.

3. Contrastive RL mechanism, prompting, and training dynamics

In CRINN, “contrastive” does not refer to a standard embedding-based contrastive loss such as InfoNCE (Li et al., 4 Aug 2025). Instead, the term denotes a comparative prompting and learning setup in which the model receives multiple previous code implementations plus their performance scores, performs comparative analysis of fast and slow variants, and is then updated using execution-derived RL reward.

The framework maintains a performance-indexed database of successful code samples. Exemplars are sampled from this database using a temperature-scaled softmax over scores:

$$P(B_i) = \frac{\exp\left( ({s_i} - \mu)/\tau \right)}{\sum_{j} \exp\left(({s_j} - \mu)/\tau\right)}.$$

Here, $s_i$ is the score of code sample $i$, $\mu$ is the mean score across all codes in the database, and $ef$0 is the temperature. This biases sampling toward stronger prior implementations while preserving exploration.

Prompt construction is rigidly structured. For the current module, the LLM is required to return sections titled ## Performance Analysis, ## Algorithm Design, and ## Code. The Performance Analysis section must discuss which implementations are faster, why they are faster, what bottlenecks harm slower ones, and what optimizations remain unexplored. This comparative reasoning is part of the method’s stated contrastive mechanism.

Policy optimization uses Group Relative Policy Optimization (GRPO). For each prompt $ef$1, the current policy generates a group of completions $ef$2; rewards are then normalized within the group according to

$ef$3

where $ef$4. The paper also gives a clipped GRPO objective with KL regularization to a reference policy. The intended structure is clear even though the notation in the source is described as slightly malformed. What is explicit is that the update increases the probability of code completions with better normalized rewards while clipping updates and regularizing against a reference policy.

The end-to-end workflow proceeds sequentially. CRINN begins from GLASS, builds a database of previous implementations, samples exemplars, constructs a structured prompt, generates candidate implementations, evaluates them by sweeping $ef$5, computes reward from the restricted QPS-recall AUC, updates the policy via GRPO, adds successful new code samples to the database, and then moves to the next module. The paper does not specify an explicit stopping rule, such as a fixed number of RL iterations or a convergence threshold. It also does not define a formal grammar or mutation operator set for the search space; this suggests that the effective search space is the set of code variants reachable by the LLM under module-specific prompts and interface constraints.

4. Benchmarks, empirical behavior, and limitations of the ANNS framework

CRINN is evaluated on six ANN-Benchmarks-style datasets: SIFT-128-Euclidean, GIST-960-Euclidean, MNIST-784-Euclidean, GloVe-25-Angular, GloVe-100-Angular, and NYTimes-256-Angular (Li et al., 4 Aug 2025). The RL model is trained exclusively on SIFT-128 rewards, even though final evaluation is conducted on both Euclidean and angular datasets without modification. This is presented as a generalization test.

The reported headline claim is that CRINN achieves best performance on three datasets and ties for first place on two datasets, although the supplied text identifies an editorial inconsistency: GloVe-25-angular is listed in both categories in the abstract/introduction summary. The quantitative tables are therefore the more concrete evidence. At fixed recall levels, CRINN exceeds the best baseline by +25.57\% on SIFT-128 at recall 0.900, +24.59\% on GIST-960 at recall 0.950, and +85.25\% on MNIST-784 at recall 0.999. On GloVe-25, it improves over the best listed baseline by +32.01\% at recall 0.950. By contrast, GloVe-100 is mixed, including -5.84\% at recall 0.950, and NYTimes-256 is a clear failure case, with 1,623 QPS for CRINN versus 9,459 for the best baseline at recall 0.900, corresponding to -82.85\%.

The strongest reported gains therefore occur on MNIST-784-Euclidean, GIST-960-Euclidean, SIFT-128-Euclidean, and GloVe-25-Angular. The paper highlights the +85.25\% on MNIST-784 at 0.999 recall as especially notable because very high recall is often where performance gaps are hardest to widen. At the same time, the poor behavior on NYTimes-256-Angular shows that the claimed generalization is not universal.

A module-by-module progressive analysis indicates that graph construction contributes the largest average single-stage gain, followed by search, then refinement. The reported Overall Average gains are +22.11\% for graph construction, +18.30\% for search, +9.69\% for refinement, and +50.10\% final cumulative. The paper interprets this as evidence that graph construction is more foundational and that earlier stages offer more “low-hanging fruit.”

The paper also analyzes the kinds of transformations discovered by the RL-guided LLM. In graph construction, it reports Adaptive Search with Dynamic EF Scaling, Zero-Overhead Multi-Level Prefetching, and Multi-Entry Point Search Architecture. In search, it reports Multi-Tier Entry Point Selection, Batch Processing with Adaptive Prefetching, and Intelligent Early Termination with Convergence Detection. In refinement, it reports Adaptive Memory Prefetching and Pre-computed Edge Metadata with Pattern Recognition. These examples are presented as evidence that the method performs systems-level implementation optimization, not merely scalar hyperparameter tuning.

Several limitations are explicit. Training uses only Euclidean rewards from SIFT-128, which may not capture patterns needed for angular similarity; this is the stated explanation for poorer performance on some angular datasets, especially NYTimes-256-Angular. The provided text also notes missing low-level experimental details, including model family, parameter count, learning rate, optimizer, number of RL steps, group size $ef$6, temperature $ef$7, clipping $ef$8, KL coefficient $ef$9, and timing methodology details such as warm-up or run averaging. Code availability is explicitly claimed via the GitHub repository https://github.com/deepreinforce-ai/CRINN.

5. Informal “CRINN” usage in continual invariant learning

In the second, non-official usage, “CRINN” refers to the framework introduced in "Continual Invariant Risk Minimization", which generalizes IRM to settings where environments are observed sequentially rather than simultaneously (Alesiani et al., 2023). The target problem is out-of-distribution generalization when environments arrive sequentially, with motivating settings including federated learning, streaming domains, hospitals arriving one at a time, and evolving contexts.

The paper studies predictors of the form

$(\text{QPS}, \text{recall})$0

where $(\text{QPS}, \text{recall})$1 is a feature extractor / representation and $(\text{QPS}, \text{recall})$2 is a classifier or predictor. The central objective is to learn a mechanism that relies on environment-invariant (ideally causal) features, rather than on spurious correlations that change across environments.

The motivation begins with the failure of ERM in multi-environment settings. For environment $(\text{QPS}, \text{recall})$3 with dataset $(\text{QPS}, \text{recall})$4, the risk is

$(\text{QPS}, \text{recall})$5

ERM minimizes pooled loss,

$(\text{QPS}, \text{recall})$6

and therefore ignores environment structure. In the colored-digit experiments described in the paper, ERM learns color instead of shape, so test accuracy collapses when the color-label correlation reverses.

Classical IRM addresses this by requiring that the same classifier $(\text{QPS}, \text{recall})$7 be optimal in every environment:

$(\text{QPS}, \text{recall})$8

However, the invariance constraint compares optimality across environments and therefore requires simultaneous access to multiple environments. This is exactly what fails in the continual setting, where only the current environment is visible. The same issue affects IRMv1 and IRMG, which also assume that all environments jointly participate in the optimization. Sequentially training these methods naively is therefore not enough: the model either overfits current spurious features or forgets previously discovered invariances.

The paper’s core contribution is a continual/sequential extension of IRM using variational Bayes and a bilevel optimization view, together with an ADMM-based solver. The key idea is not to replay all past environments directly, but to retain a posterior/variational distribution over model parameters and propagate it across environments so that prior invariant structure acts as memory.

6. C-BVIRM, bilevel variational structure, and empirical profile

The continual framework begins with a variational Bayesian backbone. If $(\text{QPS}, \text{recall})$9 is data from environment $[0.85, 0.95]$0 and $[0.85, 0.95]$1 denotes cumulative data up to step $[0.85, 0.95]$2, Bayes’ rule gives

$[0.85, 0.95]$3

Approximating the posterior with $[0.85, 0.95]$4 yields the continual variational objective

$[0.85, 0.95]$5

The KL term regularizes against catastrophic forgetting.

The paper then reformulates IRM as a bilevel constrained problem, called BIRM, by replacing the inner argmin conditions with first-order optimality constraints:

$[0.85, 0.95]$6

This is described as mathematically equivalent to IRM under the stated assumptions and more convenient for optimization.

To make this compatible with continual Bayes, the paper lifts both $[0.85, 0.95]$7 and $[0.85, 0.95]$8 from point estimates to variational distributions, giving BVIRM. In the sequential setting, the previous posterior becomes the next prior, producing C-BVIRM:

$[0.85, 0.95]$9

$$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$0

The paper’s interpretation is that the risk term learns the current environment, the KL-to-previous-posterior term preserves previously learned structure and reduces forgetting, and the invariance constraints encourage retention of the intersection of mechanisms supported across environments rather than arbitrary predictive shortcuts.

An information-theoretic motivation is also given through KL asymmetry. For

$$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$1

the support obeys

$$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$2

Sequential KL projections therefore tend to shrink support toward the intersection of invariant model families. The paper presents this as intuition for why continual variational learning can help discover invariant mechanisms.

Optimization is handled by an ADMM-style decomposition. For BVIRM, environment-specific classifier-distribution parameters $$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$3, a consensus classifier parameter $$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$4, dual variables $$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$5, and stationarity-enforcing variables $$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$6 are updated through alternating steps, with the augmented Lagrangian

$$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$7

The implementation uses a mean-field Gaussian approximation with reparameterization,

$$r = \operatorname{AUC}\left(\{(\text{QPS}, \text{recall}) : \text{recall} \in [0.85, 0.95]\}\right).$$8

so expected risks are estimated by Monte Carlo and Gaussian KL terms are available in closed form.

Empirically, the framework is evaluated on colored versions of MNIST, Fashion-MNIST, KMNIST, and EMNIST, with two color schemes: b01, where digit color is correlated with label, and b11, where background color is correlated with label. Training environments use mild correlation noise and the test environment strongly inverts the correlation. The baselines include ERM, IRMv1, IRMG, EWC, GEM, MER, VCL/VCLC, and the continual invariant variants C-VIRMv1, C-VIRMG, and C-BVIRM; EIIL is also evaluated for cases without environment labels.

The reported pattern is that standard ERM, IRM, and IRMG fail when environments arrive sequentially, and generic continual learning baselines also fail because they do not explicitly enforce invariance across environments. The continual invariant variants improve OOD generalization; in many reported settings, C-VIRMv1 obtains the best test accuracy, while C-BVIRM is competitive and clearly better than non-invariant baselines. In the 2-environment MNIST b01 setup, the reported test accuracies are approximately 12.7\% for ERM, 9.9\% for IRMv1, 24.9\% for VCL, 29.6\% for C-BVIRM, and 46.0\% for C-VIRMv1. The paper interprets this as evidence that the continual invariant methods do not rely exclusively on spurious color features. It also reports that EIIL can allow the continual invariant method to work when environments are not explicitly given, and that in some settings inferred environments improve both train and test performance.

In this non-official sense, then, “CRINN” denotes a continual extension of invariant risk minimization in which sequential variational posteriors, bilevel invariance constraints, and ADMM optimization are combined to preserve previously learned invariant structure while improving generalization under sequentially observed environments.