CLOC: Continual Localization in Dynamic Environments

Updated 26 June 2026

CLOC is a continual learning framework for localization tasks that adapts to non-stationary data while preventing catastrophic forgetting.
CLOC solutions leverage replay buffers, self-supervised predictive learning, and gradient-based parameter updates to dynamically maintain spatial accuracy.
Evaluation metrics such as Information Retention and average online accuracy demonstrate CLOC's effectiveness across benchmarks like geo-localization and action detection.

Continual LOCalization (CLOC) designates a family of continual learning methods and benchmarks for localization tasks under non-stationary data streams. CLOC methodologies have been developed for video action localization, camera and object geo-localization, world-knowledge tracking in LLMs, and multiple instance learning for histopathological analysis. CLOC solutions are unified by their emphasis on sustaining spatial or structural localization performance as data distributions drift over time, mitigating catastrophic forgetting through replay, self-supervision, memory-efficient adaptation, or targeted parameter updates.

1. Formal Problem Definition and Major Benchmarks

In CLOC settings, a learner is exposed to a long sequence of data $(x_t, y_t)$ , in which $x_t$ may be a video frame, an image requiring camera pose or geographic label, a text segment reflecting temporal world knowledge, or a collection of instances within a bag. The goal is to maintain accurate, up-to-date localization or classification abilities while adapting quickly to new concepts and avoiding backward transfer loss on previous regions, instances, or entities.

Key benchmarks include:

CLOC Image Geo-localization: A chronologically ordered, 39M-image, 712-class subset of YFCC-100M, with non-stationary class popularity and shifting visual distributions. The learner must predict place labels for each incoming image (Prabhu et al., 2023, Cai et al., 2022, Bornschein et al., 2024).
Continual Camera Localization: Sequential learning over multiple scenes with pose annotated images, necessitating direct or hierarchical regression of 6-DOF camera poses (Wang et al., 2021).
Continual Predictive Action Localization: Real-time detection of spatial and temporal regions of interest in streaming video without manual bounding box annotation (Aakur et al., 2020).
Lifelong LLM Pretraining: Continual ingestion and updating of entity and relational knowledge through layer-localized, gradient-driven updates (Fernandez et al., 2024).
Continual Multiple Instance Localization in Histopathology: Instance (cell/patch/tumor) and bag-level adaptation on disjoint, unrehearsed tasks under strict continual MIL constraints (Lee et al., 3 Jul 2025).

Evaluation protocols generally enforce single-pass operation, strict online prediction before label revelation, and metrics for both rapid adaptation and information retention.

2. Core CLOC Algorithms and Sampling Strategies

Solutions to continual localization share several conceptual and algorithmic elements:

Replay and Memory-Based Adaptation: Standard approaches maintain a buffer for replay, as in fixed-size episodic memory for image-based camera localization (Wang et al., 2021), or unrestricted history for large-scale classification (Prabhu et al., 2023).
Coverage-Score Buffering (Buff-CS): In camera localization, buffer sampling is prioritized by maximizing spatial coverage over scene subregions to prevent regional forgetting. Pseudocode for Buff-CS is:

function update_buffer(x, y, class c):
  if buffer not full:
    add (x, y) to B
    return
  if c != largest_class_in_B:
    replace_random_from(largest_class_in_B) ← (x, y)
    return
  if cs_1(x) ≠ ∅:
    replace_random_from_class(c) ← (x, y)
  else:
    with prob = m_c/n_c:
      replace_random_from_class(c) ← (x, y)
    else:
      ignore x

where

cs_1(x)

denotes new coverage at the coarsest cluster level.

Self-Supervised Predictive Learning: CLOC for action localization (Aakur et al., 2020) dispenses with manual labels by stacking a CNN encoder and hierarchical LSTMs to predict the next-frame feature maps. Prediction errors drive (i) continual weight updates, and (ii) attention maps for “surprise” localization:

$e_t(i,j) = \| x_t(i,j) - \hat y_t(i,j) \|^2, \quad S_t(i,j) = \frac{\exp(e_t(i,j))}{\sum_{m,n} \exp(e_t(m,n))}$

Nonparametric kNN Memory: In planetary-scale online geo-localization, Adaptive Continual Memory (ACM) uses a fixed ResNet-50+MLP encoder and inserts every new feature into a log-time searchable approximate kNN structure, achieving immediate adaptation and perfect consistency (Prabhu et al., 2023).
Gradient Localization for LLM Pretraining: CLOC for LLMs precomputes per-layer gradient norms on probe tasks, then restricts plasticity to those layers where new or updated world knowledge is empirically localized. CLOC-Freeze and CLOC-Scale implement this by masking updates or applying proportional learning rates (Fernandez et al., 2024).

3. Architectures and Continual Training Protocols

CLOC solutions employ varied architectures adapted to their tasks, but share certain principles optimized for online learning and localization:

Predictive Action Localization: A VGG-16 CNN encoder (up to conv5_3) feeds $14\times14\times512$ features to three stacked LSTMs, generating high-level feature predictions. A spatial softmax over squared feature errors localizes regions of novelty in streaming video, optionally updating via online gradient descent (Aakur et al., 2020).
Hierarchical Scene-Coordinate Regression: In camera localization, HSCNet uses a multi-head CNN backbone producing cluster and coordinate regressions, with stagewise replay-buffered SGD (Wang et al., 2021).
Transformer-based Online Learning: For image geo-localization, pi-transformers embed image–label pairs as sequential tokens. The privileged label is injected into keys/values of self-attention. Transformer-XL-style streaming with memory and in-order replay streams maintains both in-context adaptation and weight-based long-term memory. Losses are applied only on next-step label predictions (Bornschein et al., 2024).
Low-Rank Orthogonal Adaptation: In continual MIL, Orthogonal Weighted Low-Rank Adaptation (OWLoRA) maintains an expandable multi-task low-rank parameterization with intra- and inter-task orthogonality constraints for buffer-free continual localization (Lee et al., 3 Jul 2025).

Learning is performed strictly online—i.e., each example is processed once, sometimes in small minibatches (e.g., batch size 256 in OCL (Cai et al., 2022)), with updates applied to the main weights, memory, or both.

4. Evaluation Metrics, Results, and Comparative Analysis

CLOC methods are evaluated with protocols specific to the continual, non-stationary, and localization-relevant regime:

Information Retention (IR): Fraction of held-out test samples from prior time windows correctly labeled after long training. CLOC-ACM achieves 32% IR versus 12% for replay-based SGD (Prabhu et al., 2023).
Rapid Adaptation/Average Online Accuracy ( $a_T$ ): Running average accuracy over the online stream, e.g., 32.0% for ACM, outperforming ER/MIR/ACE by 5 absolute points (Prabhu et al., 2023).
Frame-level mAP & IoU-overlap: For streaming action localization, frame-level mAP at multiple IoU thresholds and AUC of recall/overlap (Aakur et al., 2020).
Camera Pose Accuracy: Percentage of test images with translation error below 5 cm and angular error under 5° (Wang et al., 2021).
Instance and Bag-level Metrics: For MIL, metrics include instance-level ACC, IoU, Dice, and bag-level classification accuracy; CoMEL achieves IoU=41.87 and ACC_inst=72.64, outperforming prior LoRA and prompt-based baselines (Lee et al., 3 Jul 2025).
LLM Perplexity on Probe Tasks: For continual LLM pretraining, perplexity scores on ECBD and TempLAMA probes; CLOC-variants lower perplexity uniformly for static, updated, and novel entity knowledge (Fernandez et al., 2024).

A summary comparison from (Prabhu et al., 2023):

Method	Avg. Online Accuracy ( $a_T$ )	Information Retention (IR_h)
ER	27.0%	12.5%
MIR	25.5%	11.0%
ACM (CLOC)	32.0%	32.0%

5. Principal Insights, Limitations, and Future Directions

Optimization, Not Storage, is the Bottleneck: Even unlimited-replay SGD fails for online information retention unless enhanced optimizers (AMA+MALR) and anti-overannealing learning-rate schedules are employed (Cai et al., 2022).
Buffer Coverage is Critical in Spatial Domains: Buff-CS sampling achieves 2–6 point mAP improvements over standard reservoir sampling through explicit coverage maximization, effectively reducing regional catastrophic forgetting (Wang et al., 2021).
Synergy of Fast and Slow Learning Mechanisms: Inegrating in-context attention (within replay KV-cache for transformers) and long-term SGD/Adam learning ensures both rapid adaptation to abrupt local drifts and slow global improvement in representation and memory (Bornschein et al., 2024).
Gradient Localization Reduces Forgetting in LLMs: Restricting continual updates to empirically identified "salient" layers for knowledge revisions both accelerates uptake of novel entities and enhances retention of static world knowledge (Fernandez et al., 2024).

Limitations include dependence on buffer size and update frequency, reliance on quality of coverage proxies (e.g., cluster-level annotations), computational costs for large-scale replay, and, for LLMs, sensitivity to probe-task selection for mask computation.

6. Task-Specific Extensions and Applications

Action Localization via Self-Supervised Prediction: CLOC allows competitive streaming video action localization with no bounding box annotation, outputting real-time attention maps for both action tubes and unsupervised egocentric gaze prediction (AUC=0.861, AAE=13.6°) (Aakur et al., 2020).
Continual Multiple Instance Localization in Histology: CoMEL with GDAT and BPPL introduces tractable attention over $10^5$ instances per bag and robust instance pseudo-labelling, preserving localization accuracy over long task sequences without rehearsal (Lee et al., 3 Jul 2025).
Lifelong Knowledge Editing in LLMs: Applying continual localization enables efficient, minimally destructive knowledge updating during multi-epoch web-scale LLM pretraining (Fernandez et al., 2024).

Practical recommendations include calibrating buffer size for coverage, combining in-context and replay-based memory, and adaptively profiling parameter-saliency for knowledge-rich domains with frequent concept drift.

7. Summary Table: CLOC Variants and Their Key Attributes

CLOC Variant	Domain	Core Mechanism	Notable Metric / Result
Buff-CS (Wang et al., 2021)	Camera localization	Coverage-score buffer sampling	+2–6pp mAP over class-balance
ACM (Prabhu et al., 2023)	Geo-localization	kNN memory, no storage limit	IR=32% vs. ER=12.5%
CLOC-Freeze/Scale (Fernandez et al., 2024)	LLMs	Layer-wise gradient localization	3–6 perplexity point drop
Pi-transformer + replay (Bornschein et al., 2024)	Image localization	Transformer with memory + replay	59–70% avg. acc. (MAE ViT-L)
CoMEL (Lee et al., 3 Jul 2025)	Histology MIL	GDAT/BPPL + OWLoRA	ACC_inst=72.64, IoU=41.87

A plausible implication is that CLOC frameworks, via memory- and adaptation-efficient design, provide a generic backbone for addressing catastrophic forgetting and localization drift in high-dimensional, non-stationary continual learning regimes.