Reliability > Novelty: Trustworthy Methods

• Reliability > Novelty is a principle where novel outputs are accepted only when supported by robust calibration, low false positives, and structural consistency.
• It spans domains such as open-set recognition, time series forecasting, and scientific evaluation, utilizing metrics like AUROC, MDL, and modularity to balance risk and innovation.
• Practical implementations include conservative rejection schemes and reliability-first gates that ensure new insights meet stringent validation and accuracy criteria.

Searching arXiv for the specified papers and closely related work to ground the article in current records. arxiv_search("(Borlino et al., 2022)") “Reliability > Novelty” denotes a design and evaluation principle in which systems that detect, generate, or assess novelty are constrained to privilege dependable behavior over aggressive novelty seeking. In the literature, this principle appears in several distinct but convergent forms: low-false-positive open-set recognition, reliability-first rejection under forecasting uncertainty, MDL-based structural constraints on novelty generation, evidence-linked and self-validated novelty reporting, and axiomatic stress tests for scientific novelty metrics (Borlino et al., 2022, Feng et al., 25 Mar 2025, Nakagawa et al., 18 Jun 2026, Hou et al., 21 Mar 2026, Liu et al., 16 Apr 2026). Novelty is therefore not eliminated; it is admitted, generated, or asserted only when supported by robustness, calibration, faithfulness, or structural consistency.

1. Reliability as a cross-domain constraint

The cited works do not use a single universal definition of reliability. In classical test theory, reliability is “the fraction of observed-score variance that was not error,” formalized as $$\rho_{XX} = \sigma_T^2/\sigma_X^2 = 1 - \sigma_E^2/\sigma_X^2$$; the same paper then derives factor-analytic and structured-covariance variants for multidimensional settings (Diao, 12 Nov 2025). In semantic novelty detection and open-set recognition, reliability is operationalized through AUROC, FPR@95, and, in open-set domain generalization, H-score, with particular emphasis on low false positives in safety-critical settings (Borlino et al., 2022). In graph novelty generation, reliability is defined as preservation of “global structural consistency,” measured through small DNML changes and empirically through conductance and modularity variation (Nakagawa et al., 18 Jun 2026). In novelty reporting, reliability is faithfulness to the target paper and cited literature, measured through Target Faithfulness, Cited Faithfulness, and Citation Accuracy (Hou et al., 21 Mar 2026). In recommender systems, reliability is the confidence value $l_{ui}$ attached to a prediction or recommendation, and its quality is judged by whether high reliability aligns with correct predictions and relevant recommendations (Bobadilla et al., 2024). In LLM creativity evaluation, novelty itself is made reliability-aware by combining originality and task-specific quality via the harmonic mean $H = 2OQ/(O+Q)$ (Padmakumar et al., 13 Apr 2025).

Setting	Reliability object	Novelty object
Open-set recognition	AUROC, FPR@95, H-score	Unknown classes at test time
Graph novelty generation	DNML stability, CD, MOD	Generated latent samples distinct from existing components
Novelty reporting	TF, CF, CA	Claimed manuscript contributions
Recommender systems	$l_{ui}$, RPI, RRI	Potentially relevant recommendations
LLM generation	Quality $Q$ combined with originality $O$	Unseen $n$-grams and creative outputs

This suggests that “Reliability > Novelty” is best understood not as a single metric but as a family of constraints. The common pattern is that novelty is treated as admissible only when the system can maintain some domain-specific notion of trustworthy behavior.

2. Reliable novelty detection under unknown classes and continual shift

In visual recognition, the principle is explicit in “Semantic Novelty Detection via Relational Reasoning” (Borlino et al., 2022). The task is formalized through a labeled support set $S$ of known classes and an unlabeled test set $T$ that may contain unknown classes, with a novelty score $S(x^t) \in [0,1]$ thresholded by $l_{ui}$0. Rather than fine-tuning on task-specific known classes, ReSeND pre-trains a feature extractor $l_{ui}$1, a transformer-based relational module $l_{ui}$2, and a similarity head $l_{ui}$3 to estimate whether two examples belong to the same class. At deployment, it computes class prototypes

$l_{ui}$4

then evaluates relational similarities $l_{ui}$5 and uses the maximum-softmax probability

$l_{ui}$6

as a normality score. The decision rule is conservative: predict “unknown” if $l_{ui}$7.

The reliability claim is empirical as well as architectural. ReSeND avoids task-time parameter updates, requires only prototype computation, and is designed for settings with “no access to private task data beyond computing class prototypes,” “no fine-tuning,” and strict memory and compute budgets (Borlino et al., 2022). On intra-domain tests it consistently improved AUROC and reduced FPR@95 relative to supervised and contrastive baselines: on Texture, ReSeND achieved $l_{ui}$8 versus Cross Entropy (ResNet) at $l_{ui}$9; on DomainNet Real, $H = 2OQ/(O+Q)$0 versus CutMix at $H = 2OQ/(O+Q)$1; on Painting, $H = 2OQ/(O+Q)$2 versus ViT at $H = 2OQ/(O+Q)$3 (Borlino et al., 2022). Under a strict PACS multi-source budget of approximately $H = 2OQ/(O+Q)$4 seconds on $H = 2OQ/(O+Q)$5 GPU, ReSeND reached $H = 2OQ/(O+Q)$6, outperforming MSP, ODIN, Energy, GradNorm, OODFormer, Mahalanobis, and Gram under the same budget (Borlino et al., 2022).

Continual Novelty Detection extends the same priority to continual learning (Aljundi et al., 2021). After each stage $H = 2OQ/(O+Q)$7, the framework distinguishes $H = 2OQ/(O+Q)$8, $H = 2OQ/(O+Q)$9, and the reliability-critical set $l_{ui}$0 of forgotten samples. It emphasizes selective prediction through time-indexed scores $l_{ui}$1 and thresholds $l_{ui}$2, with risk–coverage tradeoffs

$l_{ui}$3

The paper reports that novelty detection degrades as more stages are learned, that forgotten samples cluster near decision boundaries, and that Softmax MSP and ODIN remain the strongest baselines, while feature-space and generative detectors degrade more severely under feature drift (Aljundi et al., 2021). The prototype-aware baselines B1 and B2 improve mean AUC over MSP in several regimes; for example, in TinyImageNet multi-head with LwF, mAUC increases from $l_{ui}$4 to $l_{ui}$5 with B2 (Aljundi et al., 2021).

Taken together, these results establish a reliability-first interpretation of novelty detection: the central objective is not to maximize the quantity of flagged novelties, but to minimize risky acceptance under class shift, domain shift, and continual forgetting.

3. Reliability-first rejection and structurally constrained novelty

A second line of work treats novelty as acceptable only after explicit rejection mechanisms or structural consistency tests. In time series forecasting, the framework “Towards Reliable Time Series Forecasting under Future Uncertainty” separates two failure modes: ambiguity within the training distribution and novelty outside it (Feng et al., 25 Mar 2025). Ambiguity rejection uses historical forecast error variance, estimated by a rolling window or EMA, to abstain on low-confidence in-distribution cases. Novelty rejection uses a VAE and Mahalanobis distance in latent space to abstain on OOD inputs. The fusion rule is deliberately conservative:

$l_{ui}$6

where $l_{ui}$7 means reject. The paper states that “the system first checks for novelty, then for ambiguity; either trigger causes a rejection,” and reports that at approximately $l_{ui}$8 rejection, MAE and MSE “consistently improve across ETTm2, ETTh1/2, Weather, and Exchange for TimeXer, PAttn, and Autoformer” (Feng et al., 25 Mar 2025).

In graph novelty generation, reliability is imposed even more explicitly through an MDL criterion (Nakagawa et al., 18 Jun 2026). Latent graph representations are modeled by a finite mixture of a Gaussian radial component and a von Mises-Fisher directional component, and a candidate novelty batch $l_{ui}$9 is accepted only if it satisfies both a novelty condition and a reliability condition. The reliability condition is

$Q$0

The paper recommends a “reliability-first gate”: first require $Q$1, then require $Q$2 (Nakagawa et al., 18 Jun 2026). Theoretical analysis gives an upper bound on the false reliability event,

$Q$3

and shows exponential decay under the stated separation condition (Nakagawa et al., 18 Jun 2026). Empirically, on Amazon Computers, the top-$Q$4 strict-$Q$5 regime yields $Q$6 and $Q$7 on the reported scale, outperforming LL and KL in preserving modularity (Nakagawa et al., 18 Jun 2026).

Both papers make the same substantive move. Novelty is not rewarded on its own terms; it is subordinated to rejection, abstention, or structural preservation. A plausible implication is that reliability-first systems replace open-ended extrapolation with bounded novelty budgets.

4. Scientific novelty assessment: faithfulness before originality

The same principle appears in systems that assess scholarly novelty. “NoveltyAgent” is explicitly framed as a reliability-centered alternative to general AI reviewers and web deep-research systems (Hou et al., 21 Mar 2026). Its three-stage pipeline—Literature Database Construction, Point-Wise Report Generation, and Faithfulness-Enhanced Self-Validation—requires only the paper title, builds a localized full-text repository, decomposes the manuscript into discrete novelty points, performs RAG-based comparison for each point, and then validates every externally grounded sentence against source text before final polishing. The paper emphasizes conservative behaviors: full-text evidence for similarities and unique differences, abstention when evidence is thin, strict citation insertion, and programmatic correction of invalid claims (Hou et al., 21 Mar 2026).

The reported gains are predominantly reliability gains. On the checklist-based evaluation, NoveltyAgent achieves overall $Q$8, with Completeness $Q$9, Depth $O$0, and Faithfulness $O$1; the overall score exceeds GPT-5 DeepResearch by $O$2 (Hou et al., 21 Mar 2026). Relative to GPT-5 DeepResearch, it reports TF $O$3 versus $O$4, CF $O$5 versus $O$6, and CA $O$7 versus $O$8; removing self-validation lowers CF by $O$9 points and CA by $n$0 points (Hou et al., 21 Mar 2026). The system therefore treats novelty reporting as credible only when externally verified.

“On the Limits of LLM-as-Judge for Scientific Novelty Assessment” arrives at the same conclusion from the opposite direction (Sinhahajari et al., 10 Jun 2026). Using RQ-Bench, it shows that LLM judges display a “novelty mirage”: they rate model-generated research questions as highly novel, especially in comparative settings, while domain experts prefer the author-anchored reference questions. For $n$1 samples in comparative non-obviousness judgments, Expert-1 preferred the ground-truth question in $n$2 cases and the model-generated one in $n$3, whereas the Gemini judge preferred the model-generated one in $n$4 cases and the ground-truth one in $n$5 (Sinhahajari et al., 10 Jun 2026). The paper further shows that model RQs are often source-bound, and that LLM judges miss this dimension unless it is scored explicitly. On source-boundedness, the combined two-judge means are $n$6 for ground truth and $n$7 for GPT-5.5, with lower being broader (Sinhahajari et al., 10 Jun 2026).

The two papers are complementary. One constructs a pipeline that suppresses hallucination before making novelty claims; the other shows that novelty judgments without such safeguards systematically overrate superficially distinctive but narrow outputs. This suggests that, in scientific assessment, reliability is not merely an auxiliary property of novelty reports but a precondition for their legitimacy.

5. Metrics, axioms, and reliability-aware evaluation

A reliability-first stance also changes how novelty is measured. In language-model generation, “Beyond Memorization: Mapping the Originality-Quality Frontier of LLMs” rejects the identification of novelty with originality alone (Padmakumar et al., 13 Apr 2025). Its novelty score is

$n$8

where $n$9 is the fraction of unseen $S$0-grams and $S$1 is a task-specific quality score from $S$2 to $S$3, normalized to $S$4 (Padmakumar et al., 13 Apr 2025). Because the harmonic mean is conservative, high originality with low quality, or high quality with low originality, cannot produce a high score. The paper reports that inference-time methods such as higher temperature often move along an originality–quality trade-off, whereas scaling and post-training more reliably shift the Pareto frontier outward. On CoPoet at $S$5, OLMo-7B-Instruct reaches $S$6, $S$7, and $S$8, compared with $S$9 for OLMo-7B base and $T$0 for the Dolma baseline (Padmakumar et al., 13 Apr 2025).

At the level of paper-level novelty metrics, “An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics” argues that correlations with citation counts, review scores, or human preference are unreliable because those proxies conflate novelty with impact, quality, and bias (Liu et al., 16 Apr 2026). It introduces axioms such as self-recognition, paraphrase invariance, distributed coverage, unrelatedness, citation relevance, citation primacy, and temporal accumulation, each formulated as inequalities over manipulated literature pools. No single metric satisfies all axioms consistently. The best individual metric, RND, achieves an average pass rate of $T$1, while a per-axiom weighted ensemble reaches $T$2, with per-axiom rates of $T$3 on Ax1, Ax2, and Ax8, $T$4 on Ax4 and Ax6, $T$5 on Ax7, $T$6 on Ax3<base, and $T$7 on Ax5; Ax3grad remains difficult at $T$8 (Liu et al., 16 Apr 2026). The central point is that novelty metrics become more reliable when stress-tested against necessary behavioral constraints rather than validated against confounded proxies.

In recommender systems, the same move appears in the design of RPI and RRI (Bobadilla et al., 2024). RPI evaluates whether high reliability values align with low prediction error, while RRI evaluates whether relevant recommendations are associated with above-average reliability. The paper reports that, on MovieLens with $T$9 and KNN variability, reliability application yields “about $S(x^t) \in [0,1]$0 improvement in MAE,” and that RRI gains are larger for small $S(x^t) \in [0,1]$1 (Bobadilla et al., 2024). Here again, novelty-like beyond-accuracy objectives are subordinated to trustworthy prediction.

This measurement literature converges with the factor-analytic account of reliability in psychometrics. The EFA-based paper argues that shortcut metrics such as alpha and KR-20 can misestimate reliability under multidimensionality, and that reliability should be computed directly from the modeled common variance, for example through McDonald’s $S(x^t) \in [0,1]$2 (Diao, 12 Nov 2025). A plausible implication is that “Reliability > Novelty” depends not only on conservative decision rules but also on measurement systems that do not confuse novelty with other desirable but distinct properties.

6. Limits, tensions, and future directions

The literature also shows that prioritizing reliability does not remove trade-offs. In ReSeND, extreme domain shift remains difficult: in PACS single-source, the Sketch target yields FPR@95 of $S(x^t) \in [0,1]$3, and the paper identifies sparse support data and semantically close unknowns as failure modes (Borlino et al., 2022). In graph novelty generation, stricter reliability thresholds reduce false acceptance risk but also reduce acceptance probability, which scales as $S(x^t) \in [0,1]$4; the paper explicitly notes the curse of dimensionality and the possibility that overly large $S(x^t) \in [0,1]$5 can severely curtail novelty (Nakagawa et al., 18 Jun 2026). In scientific novelty judging, comparative LLM evaluation amplifies the novelty mirage rather than correcting it, and explicit breadth or source-boundedness checks are required to recover part of the human signal (Sinhahajari et al., 10 Jun 2026). In language-model generation, novelty-seeking prompts can reduce quality, especially on MacGyver, where “Asking for novelty” and Denial prompting both lower $S(x^t) \in [0,1]$6 at $S(x^t) \in [0,1]$7 (Padmakumar et al., 13 Apr 2025). NoveltyAgent, despite strong results, still inherits limitations from text-only analysis, citation-network dependence, retrieval quality, and the absence of reported inter-rater agreement statistics (Hou et al., 21 Mar 2026).

The future directions named in these works preserve the same ordering. ReSeND proposes richer relational architectures, adaptive thresholds, multi-prototype per class, and explicit calibration or risk-coverage analyses for safety-critical settings (Borlino et al., 2022). The graph framework points to multi-prototype or more expressive relational structures, dynamic MDL criteria for richer graph types, and improved threshold calibration (Nakagawa et al., 18 Jun 2026). Novelty assessment papers argue for breadth or source-independence as first-class dimensions, expert-calibrated judging, retrieval-augmented evaluation, and architecturally diverse ensembles (Sinhahajari et al., 10 Jun 2026, Liu et al., 16 Apr 2026). Across domains, the recurring lesson is that systems should not be rewarded for producing, detecting, or declaring novelty unless they also satisfy the domain’s strongest available tests of trustworthiness.

In that sense, “Reliability > Novelty” is not an anti-novelty doctrine. It is a regime of admissibility. Novelty is valuable, but only after false positives, hallucinations, structural damage, source-boundedness, calibration error, or unsafe extrapolation have been constrained to acceptable levels.