Cue-Resistant Memorization Framework

• Cue-Resistant Memorization (CRM) is a framework that rigorously controls surface-form cues to reliably measure genuine memorization in large language models.
• It employs specific metrics like HR(τ) and Recon(τ) to distinguish true memorization from pattern completion by filtering high cue-overlap instances.
• Empirical results show that high memorization rates often stem from cue-driven completions, emphasizing CRM’s role in accurate memory and privacy assessments.

Cue-Resistant Memorization (CRM) is a principled, cue-controlled evaluation framework for measuring genuine memorization capabilities in LLMs, specifically under conditions where trivial prompt–target overlap is eliminated. Designed as a necessary condition for reliable memorization assessment, CRM explicitly conditions on the degree of surface-form cues in the prompt, ensuring that successes can be attributed to true memorization rather than to pattern completion or direct copying. This framework has gained prominence across both privacy-sensitive domains such as personally identifiable information (PII) leakage (Luo et al., 7 Jan 2026), and cognitive memory benchmarking under latent behavioral constraints (Li et al., 11 Feb 2026), providing a unified approach to disentangling trivial overlap-driven retrieval from memory phenomena of genuine research interest.

1. Core Definitions and Formalism of CRM

CRM establishes rigorous definitions for memorization by controlling the lexical overlap between model input (prompt) and target output. Let $p \in \Sigma^*$ denote a prompt prefix and $s \in \Sigma^*$ the target suffix (such as an email or behavioral action).

Overlap cue: CRM quantifies surface-form prompt–target cues as

$c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$

where $\nu(\cdot)$ denotes normalization (NFKC, downcasing, non-alphanumerics removed) and $LCS(\cdot, \cdot)$ is the length of the longest common substring. For structured data types (e.g., emails $s = \ell@d$), overlap is defined component-wise.

Cue-resistant metrics: Conditioning on $c(s, p) < \tau$ (where $\tau$ is a cue-threshold, usually $\leq 0.5$ for strict isolation), CRM defines:

• $$HR(\tau) = E[ I[s \subseteq \hat{s}(p)] \mid c(s, p) < \tau ]$$, the proportion of exact reconstructions under low-cue prompts.
• $s \in \Sigma^*$0, the expected suffix log-likelihood under the same condition.

This formalism generalizes to behavioral constraints in dialogue (as in cognitive memory tests), where "cue–trigger semantic disconnect" ensures $s \in \Sigma^*$1 for cue $s \in \Sigma^*$2 and query $s \in \Sigma^*$3 under some low similarity metric $s \in \Sigma^*$4.

2. Lexical Cue Control and Experimental Design

CRM operationalizes cue control by filtering evaluation samples such that the prompt contains minimal contiguous substring overlap with the target. This process involves:

• Calculating $s \in \Sigma^*$5 for each evaluation pair.
• Keeping only those prompts where $s \in \Sigma^*$6, ensuring direct copying or completion is not possible.
• For data types with structure (e.g., emails), applying component-level normalization and overlap (e.g., $s \in \Sigma^*$7 for emails).

In cognitive memory settings (as implemented in LoCoMo-Plus), triggers are generated that are lexically and semantically distant from the original cue. Automated similarity filtering (e.g., BM25, MPNet) enforces $s \in \Sigma^*$8, and manual validation confirms that response success requires true long-term memory, not local lexical priming (Li et al., 11 Feb 2026).

CRM benchmarks thus comprise hand-curated collections of cue–trigger pairs—balanced across cognitive phenomena for beyond-factual tests, or across 32 languages for PII leakage studies.

3. Evaluation Protocols and Task Types

A CRM assessment measures whether, under strict low-cue prompting, a model can:

• Reconstruct verbatim training suffixes (exact match).
• Associate structured or relational information deprived of signature string cues.
• Exhibit spontaneous "extractable memorization" in cue-free generations.
• Survive membership-inference attacks (MIAs) under cue-exclusion.

Table: CRM Task Variants and Metrics

Task Type	Cue Control Mechanism	Key Metric
Verbatim prefix–suffix	LCS-based filtering	$s \in \Sigma^*$9, $c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$0
Associative reconstruction	Custom prompt templates	$c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$1 for various $c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$2
Cue-free generation	Uninformative prompts	$c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$3, $c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$4
Membership-inference	Cue-filtered context window	AUROC, TPR at FPR thresholds

In LoCoMo-Plus, the evaluation framework for latent-constraint queries uses a consistency score:

$c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$5

with $c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$6 denoting the model response, $c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$7 the set of responses consistent with constraint $c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$8, and label assignment automated via LLM-as-judge (demonstrating $c(s, p) = \frac{LCS(\nu(s), \nu(p))}{|\nu(s)|}$9 agreement with humans) (Li et al., 11 Feb 2026).

4. Empirical Findings and Key Results

CRM-based studies reveal systematic over-estimation of LLM memorization in standard protocols. Under high-cue prompts ($\nu(\cdot)$0), reported hit rates can be $\nu(\cdot)$1 for PII, but at strict low-cue thresholds ($\nu(\cdot)$2 or $\nu(\cdot)$3), these collapse to near-zero across all models and languages tested (Luo et al., 7 Jan 2026). For mGPT3-13B, $\nu(\cdot)$4 for email/phone; nearly all positive hits at $\nu(\cdot)$5 arise from cue-driven completion, not genuine memory.

In relativistic dialogue settings, LoCoMo-Plus demonstrates a marked drop from factual recall performance ($\nu(\cdot)$6–$\nu(\cdot)$7) to cognitive memory under cue–trigger disconnect ($\nu(\cdot)$8–$\nu(\cdot)$9), with all evaluated architectures exhibiting a $LCS(\cdot, \cdot)$0–$LCS(\cdot, \cdot)$1 point absolute drop. Retrieval-augmented generation and memory-system baselines fail to bridge this gap; even top-tier closed-source LLMs exhibit severe performance collapse. Length-sensitivity analyses show that cue-resistant cognitive memory is especially vulnerable, degrading sharply after 20+ dialogue turns (Li et al., 11 Feb 2026).

5. Implications for Privacy, Memory Research, and Model Auditing

Empirical results using CRM demonstrate that apparent memorization, such as PII leakage or long-horizon constraint consistency, is almost entirely benign under low-cue evaluation. Most existing claims regarding LLM privacy risk arise from evaluation on examples with significant prompt–target overlap. Once cue-resistant metrics are enforced, practical privacy risk is orders of magnitude lower than previously reported.

A plausible implication is that memory phenomena associated with loose association, latent constraint, or privacy-sensitive data recovery must be reframed as contrastive, cue-controlled tests. Only recovery of ground-truth content in the absence of substantial surface cues can be fairly attributed to true model memorization.

CRM provides a unified protocol for privacy audits, robustness evaluations, and memory-system benchmarking across mono- and multilingual settings.

6. Recommendations for Research and Future Directions

CRM studies recommend:

• Always reporting $LCS(\cdot, \cdot)$2 and $LCS(\cdot, \cdot)$3 as explicit functions of the cue-threshold $LCS(\cdot, \cdot)$4, not just global recall rates.
• Adopting $LCS(\cdot, \cdot)$5 as a necessary cutoff for true memorization claims.
• Designing multilingual suites and using component-specific overlap metrics for structured data.
• Extending CRM to white-box settings, fine-tuned LLMs, and broader classes of sensitive or behavioral information.
• Employing CRM in both privacy mitigation and memory-system development to ensure that apparent improvements do not merely mask cue exploitation.

CRM thus establishes reproducible, cross-domain guardrails for research on memorization, privacy, and cognitive memory in LLMs (Li et al., 11 Feb 2026, Luo et al., 7 Jan 2026).