Opinion Fidelity Rate (OFR)

• OFR is a metric that quantifies the semantic consistency between extracted opinions and the original review text using cosine similarity.
• It computes a corpus-level score by averaging triple-level semantic grounding scores across reviews, detecting opinion-level hallucination.
• Empirical results show that higher OFR values reflect stronger source fidelity, while balancing useful paraphrasing for recommendation systems.

Opinion Fidelity Rate (OFR) is introduced in HADSF as one of the paper’s two core hallucination diagnostics for LLM-based review information extraction. It is designed to measure semantic grounding of extracted opinions in the original review text. HADSF already constrains extracted aspects through a corpus-level vocabulary $A^*$, but that alone cannot tell whether the accompanying opinion text is faithful. A model may choose a valid aspect such as service or price yet still invent or over-interpret the opinion phrase attached to it. OFR is therefore designed to measure semantic grounding of extracted opinions in the original review text. This makes it complementary to Aspect Drift Rate (ADR) rather than redundant with it: ADR asks whether the extracted aspect leaves the controlled aspect schema, while OFR asks whether the extracted opinion remains semantically supported by the source review. In the paper’s framing, ADR captures schema-level drift, while OFR captures content-level faithfulness (Nie et al., 30 Oct 2025).

1. Conceptual role and motivation

The motivation for OFR is explicit: prior review-extraction pipelines for explainable recommendation lacked “rigorous, quantitative assessment of the factual fidelity of extracted content,” especially at the level of whether an extracted opinion phrase is actually grounded in the source review. HADSF addresses this gap after Stage II extraction of structured aspect-opinion-sentiment triples. The paper’s wording is concise: “Opinion Fidelity Rate (OFR): Evaluates the semantic consistency between extracted opinions and original review content.” It later summarizes the operational interpretation as helping “verify grounding in source spans” and diagnose “opinion-level speculation” (Nie et al., 30 Oct 2025).

This positioning is important because OFR is not a measure of aspect-schema adherence. It is best understood as a measure of semantic faithfulness / grounding of extracted opinion phrases, not of aspect-schema adherence. It is closer to factual grounding within the source review than to downstream usefulness per se, though the paper studies its relationship to recommendation accuracy. The kinds of extraction behaviors that reduce OFR are those where the model produces opinion content not semantically recoverable from any local review span: unsupported adjectives, exaggerated reformulations, inferred but unstated product properties, or contamination from few-shot exemplars. By contrast, OFR remains high when the model either copies opinion phrases directly or paraphrases them in a way that still aligns strongly with a contiguous review span (Nie et al., 30 Oct 2025).

2. Formal definition and notation

The formal definition appears in the Hallucination Quantification Framework, under “Novel Hallucination Metrics.” The paper defines OFR as

$$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$

The notation is slightly overloaded because $s$ denotes both the extracted triple set for a review and the sentiment field inside each triple. A cleaner equivalent restatement given in the source is

$$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$

Here $S$ is the set of evaluated extraction instances; each instance is associated with a user $u$, item $i$, timestamp $\tau$, source review $r$, and extracted triple set $s$ or $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$0. A triple $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$1 contains an extracted aspect $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$2, extracted opinion text $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$3, and sentiment polarity $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$4. The quantity $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$5 is the semantic similarity score between extracted opinion $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$6 and its best-matching grounded span in review $$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$7. OFR is computed first at the triple level, then averaged within each review over all extracted triples, and then averaged across the corpus of evaluated review extractions. It is therefore a corpus-level metric built from triple-level grounding scores with review-level normalization. In experiments, the authors report OFR for each model on each dataset, so practically it is also a model-level summary statistic over that model’s full extraction output on a dataset (Nie et al., 30 Oct 2025).

The paper defines the underlying semantic grounding function exactly as

$$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$8

with

$$\text{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,s) \in S} \frac{1}{|s|} \sum_{(a,o,s) \in s} \text{SemSim}(o, r). \tag{15}$$9

The special rule is explicit: “If the opinion text $s$0 appears verbatim in $s$1 (case-insensitive), we set $s$2.” In Eq. (16)–(17), $s$3 is a review of $s$4 tokens; $s$5 is the token length of the opinion phrase; $s$6 is a span-length tolerance hyperparameter; $s$7 is the set of all contiguous spans $s$8 in the review whose token length falls in $s$9; $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$0 is the embedding of token $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$1; and $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$2 is the mean embedding of a token sequence. The normalized quotient is cosine similarity between the mean embedding of the extracted opinion and the mean embedding of a candidate review span. This means that OFR is not based on exact string matching alone. It is a span-level semantic grounding metric (Nie et al., 30 Oct 2025).

3. Operational computation in HADSF

Operationally, OFR is measured after Stage II extraction. The pipeline is specified as follows (Nie et al., 30 Oct 2025):

1. Start from chronological interactions $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$3.
2. Use Stage I to induce the consolidated aspect vocabulary $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$4.
3. Use Stage II prompts to extract structured triples

$$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$5

from each review $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$6.

1. For each extracted opinion $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$7, tokenize $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$8 and the review $$\mathrm{OFR} = \frac{1}{|S|} \sum_{(u,i,\tau,\mathbf{s},r)\in S} \frac{1}{|\mathbf{s}|} \sum_{(a,o,\sigma)\in \mathbf{s}} \mathrm{SemSim}(o,r).$$9.
2. If $S$0 appears verbatim in $S$1, ignoring case, assign $S$2.
3. Otherwise, enumerate all contiguous review spans whose token lengths are within $S$3 of $S$4.
4. Embed each token with $S$5, mean-pool each candidate span and the opinion phrase using Eq. (17), and compute cosine similarity.
5. Take the maximum span similarity as $S$6.
6. Average these scores over all triples in the review. 10. Average the review-level scores over all extracted reviews in $S$7 to obtain OFR.

Table 6 notes $S$8, so in reported empirical results the allowable span length differed from the opinion length by at most two tokens. No human annotations, external reference spans, or LLM-as-a-judge are used in the formal OFR definition. Inputs are purely the source reviews $S$9, the extracted triples $u$0, a token embedding function $u$1, a tokenizer, the span tolerance $u$2, and the exact-match override rule. The paper does not specify the exact embedding model used for $u$3, does not mention any thresholding to binarize OFR into faithful or unfaithful, and does not define filtering rules for invalid triples beyond averaging over the extracted set $u$4. OFR is used as a continuous score (Nie et al., 30 Oct 2025).

4. Empirical values and observed patterns

OFR is reported in Table $u$5, captioned “Hallucination metrics across datasets,” with $u$6. The reported values are as follows (Nie et al., 30 Oct 2025):

Model	OFR values
Llama3.x 3B	Musical 0.7256; Industrial 0.7201; Yelp 0.9449
Llama3.x 8B	Musical 0.9121; Industrial 0.8770; Yelp 0.9892
Llama3.x 70B	Musical 0.9852; Industrial 0.9828; Yelp 0.9896
Qwen2.5 1.5B	Musical 0.9007; Industrial 0.9284; Yelp 0.9276
Qwen2.5 3B	Musical 0.7273; Industrial 0.7727; Yelp 0.9450
Qwen2.5 7B	Musical 0.9199; Industrial 0.9161; Yelp 0.9515
Qwen2.5 14B	Musical 0.9153; Industrial 0.9361; Yelp 0.9555
Qwen2.5 32B	Musical 0.9667; Industrial 0.9782; Yelp 0.9904
Distill 8B	Musical 0.8480; Industrial 0.8485; Yelp 0.8743
Distill 14B	Musical 0.9289; Industrial 0.9366; Yelp 0.9305

Several patterns emerge. First, OFR generally rises with larger base-model scale, especially within Llama3.x and Qwen2.5, though the recommendation gains are not monotonic. Llama 70B and Qwen 32B have among the highest OFR values, but they do not consistently produce the best downstream MSE. Second, Yelp consistently shows higher OFR than the two Amazon datasets for many models, suggesting opinions in Yelp reviews may be easier to ground semantically, possibly because the reviews are longer: average review length 90.03 versus 42.52 and 34.82. Third, the distilled DeepSeek-R1 variants do not improve OFR reliably and often underperform strong non-distilled base models, aligning with the broader conclusion that chain-of-thought optimization can inject unnecessary complexity into extraction (Nie et al., 30 Oct 2025).

Interpretively, OFR is a similarity-based score that, given the cosine form and the exact-match override, should lie in practice in $u$7 if embeddings are such that candidate similarities are nonnegative in this application; mathematically cosine similarity is in $u$8, but all reported values are positive and high, ranging roughly from 0.72 to 0.99. In the paper’s usage, higher OFR means stronger semantic consistency between extracted opinions and the source review, while lower OFR indicates more opinion-level hallucination, paraphrastic overreach, unsupported inference, or speculative elaboration (Nie et al., 30 Oct 2025).

5. Diagnostic significance, prompt effects, and downstream utility

The key finding linking OFR to recommendation is the paper’s claim of a nonmonotonic hallucination–performance relationship. In the RQ3 discussion, the authors state: “Lower hallucination does not uniformly yield better accuracy: mild hallucination often coincides with competitive or improved MSE—likely reflecting useful abstraction/paraphrasing that broadens aspect coverage—whereas extremely low rates are frequently associated with worse MSE, suggesting under-generation or overly conservative outputs... Overall, the hallucination–performance relationship is non-monotonic: moderate levels can be tolerated or beneficial, but both excessive hallucination and overly strict suppression degrade the extracted semantics and their utility for recommendation.” Applied specifically to OFR, this means that maximizing opinion grounding to the point of near-literal extraction is not always best for rating prediction. Some degree of paraphrastic abstraction can still preserve semantic faithfulness while producing more useful generalized signals for the recommender. Very low OFR indicates harmful hallucination; very high OFR may correlate with conservative extraction that misses semantically relevant but lexically diverse summaries. OFR is therefore presented as a diagnostic to balance with predictive utility rather than as a single objective to optimize blindly (Nie et al., 30 Oct 2025).

The paper also studies OFR under prompt design in Figure $u$9. Adding CoT tends to reduce ADR but often also reduces OFR, because “CoT often increases opinion-level speculation,” with decreases especially evident on Musical and Industrial, and also on Yelp for the 70B model. The summarized trade-off is explicit: “The few-shot without CoT setting consistently yields the highest ADR... By contrast, adding CoT reduces ADR... This gain is accompanied by reduced OFR... Model capacity moderates this trade-off: under few-shot+CoT, the 70B variant attains the lowest ADR while keeping OFR high, yielding the most balanced configuration overall.” This is one of the clearest interpretations of OFR in the paper: aspect selection can become more disciplined while opinion phrasing becomes more speculative, so OFR is needed precisely because ADR alone would miss that degradation (Nie et al., 30 Oct 2025).

6. Related formulations and limitations

Relative to existing hallucination or faithfulness metrics, HADSF does not conduct an explicit benchmark against ROUGE, BERTScore, QAGS, or entailment-based faithfulness scores. Its main comparison is conceptual: prior recommender-system hallucination practices such as “counting out-of-catalog items in generated lists” are said to be inadequate for review extraction. OFR’s significance is that it is local, extraction-specific, and source-grounded. For explainable recommendation systems, this is valuable because explanations should be traceable to what users actually wrote (Nie et al., 30 Oct 2025).

Several adjacent papers do not define OFR by name, but they make clear that “opinion fidelity” is not a single standardized object. In opinion summarization, the metric in “Automatically Evaluating Opinion Prevalence in Opinion Summarization” is a reference-free metric for opinion prevalence that measures whether summary statements are supported by many reviews rather than by one or a few; the paper explicitly notes that it never uses the term OFR, but that its metric is a form of population-level opinion fidelity and “a strong candidate for an Opinion Fidelity Rate if OFR is meant to quantify representativeness of stated opinions” (Malon, 2023). In opinion-aware RAG, “Beyond Factual Grounding: The Case for Opinion-Aware Retrieval-Augmented Generation” argues that factual RAG should minimize posterior entropy, whereas opinion-aware RAG must preserve it, and frames the task as one of distributional fidelity, retrieval coverage, and fairness rather than pointwise factual convergence (Agrawal et al., 13 Apr 2026). In rate-fidelity theory, “Subjective Information Measure and Rate Fidelity Theory” does not mention OFR by name, but provides a general mathematical template in which fidelity is defined through generalized or subjective mutual information rather than distortion (0705.3644). This suggests that OFR can refer to at least three distinct regimes: source-grounded opinion extraction, prevalence-weighted representativeness across review populations, and distributional fidelity in retrieval and generation.

The limitations of OFR in HADSF are also explicit. Because OFR depends on mean token embeddings and local contiguous span matching, it may undervalue opinions that are faithfully synthesized from noncontiguous evidence spread across a review. It may also reward semantically similar but not strictly entailed phrases if cosine similarity is high enough. The exact behavior depends on the unspecified embedding function $i$0, making reproducibility sensitive to implementation choices unless the repository fixes them. Because OFR averages over extracted triples equally, it does not weight by opinion importance or by sentiment strength. Finally, the notation in Eq. (15) is somewhat ambiguous because $i$1 is reused both for triple sets and sentiment labels; any implementation should disambiguate this. These caveats do not alter the central role of OFR in HADSF: it is the paper’s opinion-grounding metric, defined as a corpus-averaged, review-normalized mean of maximum semantic similarities between each extracted opinion phrase and the best contiguous source-review span of similar length (Nie et al., 30 Oct 2025).