NOrmed Index of Retention (NOIR)

• NOrmed Index of Retention (NOIR) is an automated scalar metric that evaluates summaries by balancing semantic retention with token-length compression.
• It leverages cosine similarity of pre-trained sentence embeddings and a deterministic token ratio to produce an interpretable, continuous quality score.
• Empirical validation demonstrates NOIR’s robustness in ranking LLM-generated summaries, matching human assessments and outperforming traditional metrics.

The NOrmed Index of Retention (NOIR) is an automated, reference-free scalar metric for assessing the quality of text summarization, developed to address the simultaneous demands of semantic retention and compression in summary generation. NOIR operationalizes the recall–compression trade-off through a mathematically principled formula, leveraging pre-trained sentence-embedding models to compute semantic similarity, and is validated against both synthetic and human-annotated summarization datasets. This metric is designed for robust ranking and filtering of summary candidates in diverse, high-throughput, and domain-agnostic settings, providing a continuous, interpretable scalar that encodes both the amount of original meaning retained and the degree of conciseness achieved by the summary (Foland, 10 Jul 2025).

1. Mathematical Formulation and Underlying Principles

NOIR assesses a summary in relation to its source document by quantifying two coupled dimensions: the degree of semantic preservation and the extent of token-length reduction. Given an original document of token length $T_0$ and a candidate summary of length $T_{\text{summ}}$, embeddings for both are computed via a sentence-embedding model, yielding unit-norm vectors $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$. Semantic retention is estimated by their cosine similarity: $$D = \hat{v}_{\text{text}} \cdot \hat{v}_{\text{summ}}$$ with $0 \le D \le 1$, where $D=1$ represents perfect semantic overlap.

Length compression is captured by the ratio

$k = \frac{T_{\text{summ}}}{T_0},\quad 0 < k \le 1.$

NOIR's canonical form is: $M_{\rm NOIR} = \frac{\ln k}{\ln D}$ which is positive when the summary is shorter than the source and retains meaningful content ($k < 1, D < 1$). Negative values indicate over-verbosity ($k > 1$), and values near zero correspond to summaries semantically unrelated to the source ($T_{\text{summ}}$0). In the limit as $T_{\text{summ}}$1, the NOIR score diverges to infinity, reflecting the unattainable ideal of perfect semantic retention at extreme compression. The scoring mechanism thus enshrines the recall–compression trade-off as a single interpretable scalar, where brevity and fidelity are quantitatively balanced (Foland, 10 Jul 2025).

NOIR may be generalized to

$T_{\text{summ}}$2

where $T_{\text{summ}}$3 modulates the relative weighting of length, though empirical and theoretical analyses identify $T_{\text{summ}}$4 as optimal.

2. Quantification of Semantic Retention and Token Compression

Semantic retention is operationalized entirely via sentence-embedding model similarity, sidestepping the need for reference summaries or manual annotation. In foundational experiments, all-MiniLM-L6-v2, a 384-dimensional sentence transformer, is employed; all-mpnet-base-v2 serves as a confirmatory embedding model. Once embeddings $T_{\text{summ}}$5 are obtained, their dot product directly yields $T_{\text{summ}}$6. In high dimensions with typical semantic-embedding training, cosine similarity exhibits approximately multiplicative behavior over successive compressions, theoretically justifying its use in the logarithmic relation central to NOIR.

Token lengths are computed via a deterministic tokenizer such as tiktoken. Robustness to modest (up to ±20%) discrepancies between tokenizer and embedder segmentation is ensured by the ratio structure of $T_{\text{summ}}$7, which cancels systematic biases in length accounting (Foland, 10 Jul 2025). The approach assumes that embedding cosine similarity between random text pairs is near zero and paraphrases approach unity—a criterion satisfied empirically for the tested sentence-transformer models.

3. Computation Protocol and Implementation Details

NOIR computation for any document–summary pair follows a standardized, parameter-free pipeline:

1. Tokenization: Apply a consistent tokenizer (e.g., tiktoken) to both document and summary to yield $T_{\text{summ}}$8 and $T_{\text{summ}}$9.
2. Embedding: Compute embeddings for both using the chosen sentence-model.
3. Cosine Similarity: Calculate semantic similarity as $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$0.
4. Compression Ratio: Compute $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$1.
5. NOIR Score Calculation: Evaluate $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$2.

This process is reference-free, does not require gold summaries or tunable weights, and scales computationally with embedding cost (typically $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$3). Maximum supported input length is dictated by the embedding model (e.g., 512 tokens for all-MiniLM-L6-v2); larger texts may necessitate chunking strategies, with careful validation of unbiased scoring (Foland, 10 Jul 2025).

4. Experimental Validation and Benchmarking

Empirical validation of NOIR encompasses a multi-tiered experimental design:

• Summarization Core Dataset: 456 MultiRC2 paragraphs from diverse genres are LLM-summarized into three length bins (200-word, 75-word, single-sentence).
• Paraphrase Dataset: 490 LLM-generated paragraphs paraphrased into 'concise' and 'verbose' sets, controlling for length-similarity confounding.
• Random Summary Pairs: Each summary paired with unrelated text to test baseline behavior ($$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$4, NOIR near zero).

Main findings:

• True summaries yield mean $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$5 ($$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$6), with random pairs clustering near zero ($$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$7).
• Embedding choice impacts absolute values (all-mpnet-base-v2 increases mean to $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$8, $$\hat{v}_{\text{text}}, \hat{v}_{\text{summ}} \in \mathbb{R}^d$$9), with cross-model rank correlation $$D = \hat{v}_{\text{text}} \cdot \hat{v}_{\text{summ}}$$0.
• Human annotation of 15 summaries (spanning five compression bins) yields Spearman's rank correlation $$D = \hat{v}_{\text{text}} \cdot \hat{v}_{\text{summ}}$$1 between NOIR ordering and human assessment, exceeding ROUGE point estimates and matching top recent learned metrics.

Interpretation of results suggests, for example, that a summary halving token length with NOIR = 4.55 corresponds to semantic retention per halving of $$D = \hat{v}_{\text{text}} \cdot \hat{v}_{\text{summ}}$$2, consistent with embedding-similarity decay observed in practice (Foland, 10 Jul 2025).

5. Advantages, Limitations, and Use Cases

The NOIR metric offers several operational advantages:

• Fully automated, reference-free, and tunable-parameter free.
• Explicit trade-off balancing of semantic retention and brevity.
• High-throughput, CPU-feasible computation (e.g., ≈1,800 embeddings/min on standard hardware).
• Continuous, unbounded-above scoring, smoothly penalizing over-verbosity.

Principal limitations stem from embedding model quality: random document pairs must yield cosine near zero, paraphrases near unity. For very long texts (≫512 tokens), embedding degradation or length constraints require methodological accommodations (e.g., sliding windows or hierarchical pooling) with empirical validation to avoid bias. Cosine similarity is unreliable for $$D = \hat{v}_{\text{text}} \cdot \hat{v}_{\text{summ}}$$3, though well-constructed summaries rarely breach this floor (Foland, 10 Jul 2025).

Principal use cases include:

• Filtering and ranking LLM-generated summaries for downstream selection or quality control.
• Automated prompt tuning, employing NOIR as an optimization feedback signal on held-out corpora.
• Evaluation in settings lacking human-authored references or gold summaries.
• Adaptive online systems that refine or retrain compressive summarization models via NOIR optimization.

6. Best Practices and Practical Considerations

Effective application of NOIR requires:

• Consistent tokenization regime, while tolerating modest absolute differences so long as proportionality is preserved.
• Initial validation of the embedding model's performance characteristics: random-pair cosine ≈0, paraphrase ≈1.
• Rigorous adherence to input-length constraints of embedding models; for overlength inputs, use of validated chunking/pooling, with caution regarding possible bias.
• Production thresholding via empirical calibration (e.g., accepting summaries scoring above domain-calibrated cutoffs such as one standard deviation below mean NOIR on a held-out set).
• Periodic verification of embedding model drift, ensuring preconditions for cosine-based semantic retention remain satisfied.

A plausible implication is that NOIR additionally serves as a monitoring signal for model or pipeline changes that affect semantic encoding fidelity, and as a robust diagnostic for low-quality or off-distribution summaries. Its explicit, parameter-free design supports immediate deployment across summarization domains and embedding model variants, with interpretability grounded in simple, transparent formulae (Foland, 10 Jul 2025).