LLM-Metrics: Measuring Research Impact Through Large Language Model Memory

Abstract: Citation counts remain the dominant metric for assessing research impact, yet they suffer from well-documented limitations: temporal lag, disciplinary bias, and Matthew effects. Here we propose LLM-Metrics, a research-impact assessment metric derived from the parametric memory of LLMs. The central hypothesis is that high-impact papers receive greater exposure in the academic community, that this exposure enters LLM training data in textual form, and that models consequently form stronger parametric memory of these papers. We designed four types of multiple-choice probes, covering title recognition, author recognition, method recognition, and venue recognition, and evaluated 549 computer science papers published in 2023-2024 across 17 LLMs spanning 0.5B to 72B parameters from six vendors. Of the 17 models, 15 produced positive predictions, 9 of which were significant at p less than 0.05, with an overall Spearman correlation of rho = 0.1495 and p = 0.0004 against citation counts. Three additional findings support the proposed mechanism. First, the predictive signal was stronger for 2024 papers, rho = 0.1880, whose citation counts were near zero at model-training time, reducing the plausibility of a simple reverse-causality explanation. Second, author-recognition probes showed the strongest discriminative power, consistent with an exposure-driven memory mechanism. Third, model scale and predictive power were non-monotonic: a 3B-parameter model, Llama-3.2-3B-Instruct, with rho = 0.1829, outperformed most larger models, supporting a selective-memory hypothesis in which the limited capacity of smaller models can serve as an effective information filter. LLM-Metrics offers a real-time, cross-disciplinary, citation-independent paradigm for research assessment.

• The paper introduces LLM-Metrics as a novel citation-independent indicator using LLM parametric memory to predict future citations.
• It employs metadata probes—title, author, venue recognition—scored on a 5-point scale, showing strong empirical correlations with citation counts.
• The study reveals non-monotonic effects across model sizes and vendor differences, highlighting LLM memory's promise as a real-time impact measure.

LLM-Metrics: Assessing Research Impact via Parametric Memory in LLMs

Introduction

"LLM-Metrics: Measuring Research Impact Through LLM Memory" (2605.22176) introduces LLM-Metrics, a novel citation-independent scientometric indicator leveraging the parametric memory of LLMs. The central hypothesis is that the selective memory of LLMs reflects scholarly exposure of academic papers—high-exposure publications more robustly imprinted in model parameters are likely to accrue greater citation impact over time. This work empirically evaluates LLM-Metrics across multiple recent LLM architectures, presenting comprehensive evidence that model-memory-based signals are predictive of future citations, with unique statistical and mechanistic characteristics.

Figure 1: Overview of LLM-Metrics core empirical findings—(left) Stronger correlation with later citations for 2024 versus 2023 papers; (middle) Highest mean Spearman correlation in intermediate model sizes; (right) Author recognition yields the strongest discriminative signal.

LLM-Metrics: Definition, Measurement, and Probing Paradigm

LLM-Metrics is operationalized through multi-faceted, metadata-focused probe questions—comprising title recognition, author recognition, method recognition, and venue identification—with each probe scored on a 5-point ordinal scale later binarized for statistical uniformity. The memory score for a given paper and model is the mean binarized probe score, interpretable as the fraction of remembered probe dimensions. Model evaluation is performed deterministically (temperature=0), parsed and scored automatically, and averaged over large-scale paper-model matrices for robust signal extraction.

The methodology is designed to minimize confounds such as test-set leakage, superficial pattern recognition, and task artifacts by ensuring all distractors are semantically and lexically comparable and no direct citation features are present in the inputs. The probe set spans 549 computer science papers (2023–2024) systematically sampled to capture full coverage of citation-volume stratifications, author prominence, and venue diversity.

Empirical Results: Correlational Validity and Mechanistic Diagnostics

Cross-Model Predictive Signal

In correlational analysis, 15/17 models yield positive Spearman $\rho$ between LLM-Metrics and citation counts; 9 are significant at $p < 0.05$ and 4 at $p < 0.001$. The ensemble LLM-Metrics achieves $\rho = 0.1495$ ($p = 0.0004$), consistent across parameterizations and vendors. Meta's LLaMA-3 family shows the most robust signal.

Figure 2: Spearman $\rho$ between LLM-Metrics and citation counts for all 17 models, with significance and organizational groupings.

Moreover, sign consistency across models is 88.2%, significantly exceeding chance, and mean pairwise inter-model memory correlation is $r = 0.44$, indicating a non-trivial consensus signal reflective of shared exposure patterns.

Temporal and Mechanistic Evidence

A critical causal diagnostic leverages the training cutoff/publication year mismatch: for 2024 papers (published post-training or with negligible citation data), the predictive signal is stronger ($\rho = 0.1880$) than for 2023 papers ($\rho = 0.0559$), directly countering explanations based on simple memorization of citation counts.

Figure 3: Year-based split analysis: 8/10 models exhibit stronger prediction for 2024 papers than for 2023, excluding access to citation counts at training time.

Citation-binned memory score trajectories using Llama-3.2-3B-Instruct demonstrate monotonic score increases aligned with real citation accumulation, while error and refusal rates inversely correlate with impact.

Figure 4: (a) Llama-3.2-3B-Instruct shows memory scores rising with citation bin; (b) Temporal split confirms signal is not artifactually driven by citation-count familiarity.

Probe-Type Discrimination and Non-Monotonic Capacity Scaling

Among probe classes, author recognition emerges as the most discriminative for impact, consistent with known sociometric Matthew effects; titles and venues offer moderate discrimination, and method/technique probes, while informative for knowledge probing, are weaker discriminators for citation impact. This pattern suggests LLM-Metrics partially encodes author-visibility signals.

LLM-Metrics' performance is non-monotonic in model size—contradicting simple scale-performance assumptions. The 3B Llama-3.2-3B-Instruct model achieves the top $\rho=0.1829$, outperforming several 8–70B models. Within-vendor variation also exceeds naive scaling predictions.

Figure 5: (a) Meta's LLaMA-3 family leads in both mean and maximal correlation; (b) The size-correlation relationship is non-monotonic, maximal at intermediate scales.

This nonlinearity supports a “selective-memory window” hypothesis wherein moderate-capacity models act as efficient information filters, preferentially encoding frequent high-exposure content while limited in low-frequency/long-tail memorization.

Vendor and Model Family Heterogeneity

Vendor-level analysis finds Meta's LLaMA-3 models yield the highest and most consistent memory-impact signal, likely due to both larger-scale and higher-coverage academic corpora in their training pipelines. Alibaba's Qwen2.5 family exhibits high internal variance, with some models heavily refusing to answer. Google's Gemma-2-27B unexpectedly reverses the trend, yielding negative correlation. This underscores that training mix, alignment, and data management are primary determinants of memory-based scientometric signal, not model scale alone.

Positioning LLM-Metrics Among Scientometric Paradigms

LLM-Metrics fundamentally departs from classical, citation- or altmetric-based impact predictors [costas2015altmetrics, vital2024predicting, hull2025forecite], which typically require multi-year lags, field normalization, and access to explicit citation metadata. Unlike supervised citation-prediction architectures, LLM-Metrics is fully citation-agnostic and operates zero-shot, operating purely on latent exposure-coupled parametric knowledge. Its empirical effect size (ensemble $p < 0.05$0) is smaller than that of dedicated prediction models, but this is offset by its unique real-time, cross-field, and training-data-centric perspective.

Limitations and Prospects

The study is constrained to computer science papers over a two-year window, and trained LLMs’ data provenance is only partially known, leaving open questions about cross-field generalizability and absolute causal interpretation. The metric is also susceptible to biases in model training artifacts—e.g., over-representation of authors/institutions, altmetric echo effects—and underrepresents the “intrinsic merit” dimension in favor of scholarly exposure. Future research should extend probing to other domains, leverage richer probe types (e.g., open-ended recall), systematically integrate with feature-based and early-citation forecasters, and track longitudinal trends in LLM-mediated scholarly memory.

Conclusion

LLM-Metrics establishes LLM parametric memory as a measurable, reproducible, and citation-independent indicator of research exposure and downstream citation impact. The metric’s predictive potency is supported by multiple lines of evidence: cross-model consistency, selective author/title channel discrimination, temporal exclusion of citation-count leakage, and non-monotonic scale effects. The practical implication is that LLMs can be repurposed as scientific measurement instruments, offering immediate, field-agnostic, and training-data-representative assessments as complements—not substitutes—to classical scientometric indicators. As LLMs become further integrated into academic communication cycles, memory-based metrics of this form will serve as valuable early-exposure barometers and diagnostic feedback channels within the research ecosystem.