---
title: Factuality-Weighted Score (FWS) Evaluation
url: https://www.emergentmind.com/topics/factuality-weighted-score-fws
type: topic
---

# Factuality-Weighted Score (FWS) Evaluation

Factuality-Weighted Score (FWS) is a metric name used in recent large language model evaluation for schemes that explicitly prioritize factual reliability over surface fluency. In the literature summarized here, the term refers to two distinct constructs introduced in 2025. In "MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs," FWS is a weighted \(F_1\) measure over atomic claims in long-form answers, where claim weights are derived from a fact-importance hierarchy and claim labels are produced by a multi-agent debate process [2510.22967]. In "Factuality Beyond Coherence: Evaluating LLM Watermarking Methods for Medical Texts," FWS is a scalar composite score that combines relevance or completeness, factual accuracy, and coherence, with factuality receiving greater weight than coherence [2509.07755].

## 1. Terminological scope and disambiguation

A central point of clarification is that FWS is not a single standardized metric across the two papers. The same acronym denotes two non-equivalent evaluation designs: one is a claim-level weighted factuality measure for long-form generation, and the other is a criterion-weighted quality score for medical watermarking assessment [2510.22967] [2509.07755].

| Paper | Evaluation setting | FWS form |
|---|---|---|
| "MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs" [2510.22967] | Long-form factuality evaluation | Weighted \(F_1\) over atomic claims |
| "Factuality Beyond Coherence: Evaluating LLM Watermarking Methods for Medical Texts" [2509.07755] | Medical watermarking evaluation | Linear combination of relevance, factual accuracy, and coherence |

This terminological overlap matters because the two formulations answer different questions. The MAD-Fact formulation evaluates whether a long-form answer preserves important facts and suppresses false claims. The medical watermarking formulation evaluates whether watermarking preserves medically reliable content while also tracking coherence and completeness. A plausible implication is that comparisons of “FWS” values across these papers are not meaningful unless the underlying definition is specified.

## 2. Weighted factuality as claim-level \(F_1\) in MAD-Fact

In MAD-Fact, a single test instance consists of a question \(q_i\) and a long-form answer \(a_i\). The Clerk decomposes \(a_i\) into \(T\) atomic claims \(\{c_{i,1},\dots,c_{i,T}\}\). After debate and voting, each claim receives a binary factuality label
\[
p_{i,j}=
\begin{cases}
1,& \text{if judged TRUE}\\
0,& \text{if judged FALSE.}
\end{cases}
\]
Each claim also receives an importance weight \(\omega_{i,j}\) [2510.22967].

Let \(S=\{j:p_{i,j}=1\}\) be the set of judged-true claims. The weighted precision is
\[
\mathrm{Prec}_w(a_i)=
\frac{\sum_{j\in S}\omega_{i,j}}
{\sum_{j=1}^{T}\omega_{i,j}}.
\]
The weighted recall at recall cap \(\gamma\le 1\) is
\[
R_w@_\gamma(a_i)=
\min\!\left(
\frac{1}{\gamma}
\frac{\sum_{j\in S}\omega_{i,j}}
{\sum_{k=1}^{K_{\mathrm{gold}}}\omega^{\mathrm{gold}}_{i,k}},
1
\right).
\]
The per-example Factuality-Weighted Score is then the weighted \(F_1\):
\[
\mathrm{FWS}(a_i)=F_1@\gamma(a_i)=
\begin{cases}
\dfrac{2\,\mathrm{Prec}_w(a_i)\,R_w@_\gamma(a_i)}
{\mathrm{Prec}_w(a_i)+R_w@_\gamma(a_i)},
& \text{if }\sum_{j\in S}\omega_{i,j}>0,\\[6pt]
0,& \text{otherwise.}
\end{cases}
\]
Across a test set \(D\), the overall score is the mean:
\[
\mathrm{FWS}=\frac{1}{|D|}\sum_{i=1}^{|D|}F_1@\gamma(a_i).
\]

The construction is designed for long-form factuality, where errors accumulate across “complex reasoning chains, intertwined perspectives, and cumulative information.” The weighting scheme encodes the paper’s claim that not all factual errors are equally consequential. This suggests that MAD-Fact treats factuality as a structured property of answer content rather than as a sentence-level stylistic judgment.

## 3. Fact-importance hierarchy and multi-agent adjudication

The weights in MAD-Fact are generated through a fact-importance hierarchy, described as a \(G\)-level pyramid model. A set of \(G\) expert closed-source models \(\{\mathrm{Model}_1,\dots,\mathrm{Model}_G\}\) produces reference long-form answers \(r_{i,j}\) for each question \(q_i\). The Clerk decomposes each reference into atomic claims. Semantically equivalent claims are merged into a gold set \(\{g_{i,1},\dots,g_{i,K_{\mathrm{gold}}}\}\). For each gold claim \(g_{i,k}\), its frequency \(f_{i,k}\) is counted across the \(G\) reference answers, producing a pyramid in which claims mentioned by all \(G\) models occupy level 1, claims mentioned by \(G-1\) models occupy level 2, and so on, down to claims mentioned by one model at level \(G\) [2510.22967].

Level \(m\) receives a weight \(\omega_m\) satisfying \(\omega_1>\omega_2>\cdots>\omega_G\). In practice, the paper sets \(\omega_m=G-m+1\), or \(\omega=5-k\) when \(G=4\). During evaluation, each claim in the answer under test is matched to a gold claim and inherits the corresponding layer weight. Factually central claims therefore contribute more to the final score than peripheral claims.

Claim labels are produced by a multi-agent debate mechanism with three components. The Clerk Agent extracts fact-checkable atomic claims from \((q_i,a_i)\). A Jury of \(N\) Evaluator Agents then evaluates each claim through up to two debate rounds under one of three rules: Rule 1 “Autonomous Retrieval & Free Debate,” Rule 2 “Mandatory Retrieval & Evidence-Based Debate,” and Rule 3 “Dynamic Retrieval & Adaptive Debate.” In each round, agents take turns, consult shared knowledge \(K_{i,j,t-1}\) and previous statements \(M_{i,j,t-1}\), decide a direct, retrieval-based, or conditional response, optionally invoke \(\texttt{Search(query)} \rightarrow k_{i,j,t}\), and output \([p_{i,j,t},e_{i,j,t}]\), where \(p_{i,j,t}\in\{0,1\}\) is the judgment and \(e_{i,j,t}\) is the textual explanation. A Judge Agent applies majority vote to the final round labels \(\{p^n_{i,j}\}_{n=1}^{N}\), and ties are broken by choosing the last speaker’s vote.

This workflow makes factuality assessment an explicit adjudication pipeline. The paper’s full pseudocode operationalizes preprocessing, debate, weight assignment, and score aggregation as a deterministic evaluation routine. A plausible implication is that the framework is intended to reduce the brittleness of single-pass automated factuality judgments on long-form outputs.

## 4. Benchmarks, rankings, and human correlation in MAD-Fact

MAD-Fact reports representative results on two benchmarks. On the English benchmark LongFact, using 250 sampled items, GPT-4-Turbo achieved the highest FWS under both \(\gamma=1.0\) and \(\gamma=0.8\), with scores of \(0.569\) and \(0.681\), respectively. The second-best model was Doubao-1.5-Pro at \(0.552/0.669\). On the Chinese LongHalluQA benchmark, also with 250 items, QwQ-32B led with FWS \(=0.592/0.694\), followed by Doubao-1.5-Pro at \(0.560/0.666\). GPT-4-Turbo dropped to fifth place with \(0.452/0.563\), which the paper describes as highlighting a cultural bias [2510.22967].

The human-correlation analysis used 50 LongFact items manually rated by two undergraduates. The Pearson correlation between average human score and \(\mathrm{FWS}=F_1@_{0.8}\) was \(r=0.701\) with \(p=0.036\). The paper characterizes this as a statistically significant result confirming that FWS aligns closely with human judgments of long-form factuality.

The reported rankings support two empirical observations stated in the paper: larger LLMs generally maintain higher factual consistency, and domestic models excel on Chinese content. These findings are framed as evidence that long-form factuality evaluation must account for both model scale and language or cultural context.

## 5. Factuality-heavy composite scoring in medical watermarking

In the medical watermarking paper, FWS has a different mathematical form and a different evaluative role. The motivating claim is that, in high-stakes domains such as medicine, the primary failure mode is not that outputs sound “ungrammatical” but that they hallucinate or corrupt critical facts such as drug names and diagnoses. The paper further argues that watermarking methods often reweight low-entropy tokens, and that in medical text these tokens are disproportionately medical entities. Even minor sampling shifts can therefore introduce or substitute medical terms, undermining factual integrity [2509.07755].

The metric is defined from three normalized subscores:
\[
s_{\mathrm{coh}} \in [0,1], \quad
s_{\mathrm{rel}} \in [0,1], \quad
s_{\mathrm{fact}} \in [0,1].
\]
With weighting coefficients \(\alpha,\beta>0\) such that \(2\alpha+\beta=1\), the Factuality-Weighted Score is
\[
\mathrm{FWS}
=
\alpha\,(s_{\mathrm{rel}}+s_{\mathrm{fact}})
+
\beta\,s_{\mathrm{coh}}.
\]
The default configuration is \(\alpha=0.4\) and \(\beta=0.2\), so relevance and factual accuracy together receive \(0.8\) of the total weight and coherence receives \(0.2\).

The subscores can be obtained either through GPT-Judger or automatic metrics. For coherence, the paper uses GPT-Judger on a 1–5 Likert scale normalized to \([0,1]\), or SimCSE. For relevance or completeness, it uses GPT-Judger’s “Relevance/Completeness” score or ROUGE-L. For factual accuracy, it uses GPT-Judger’s “Factual Accuracy” score or AlignScore. The computation workflow takes prompt \(x\), generated text \(y\), and a watermark flag; computes optional automatic metrics; prompts GPT-4o to produce \(s_{\mathrm{coh}}, s_{\mathrm{rel}}, s_{\mathrm{fact}}\in\{1,\dots,5\}\); normalizes the Likert scores; optionally samples \(10\%\) of QA instances for review by 3 medical experts per item; and returns both \(\mathrm{FWS}_{\mathrm{auto}}\) and \(\mathrm{FWS}_{\mathrm{GPTJ}}\).

The paper states that \(\alpha=0.4,\beta=0.2\) was chosen via a sensitivity analysis measuring Pearson correlations between FWS and human judgments under different weightings. It further states that this split balanced the intuition that factuality is twice as important as coherence with empirical correlation to expert ratings.

## 6. Experimental behavior, practical use, and limitations

The medical watermarking study evaluates three tasks: HealthQA text completion, HealthQA question answering, and MeQSum summarization. The watermarking methods are grouped as post-hoc methods LogRank and DetectGPT, logit-based methods KGW and SWEET, the distribution-preserving method DiPmark, and the sampling-based method EXP-edit. The primary model is Meditron-7B, with BioMistral-7B and MedLlama-3-8B used for ablations [2509.07755].

For \(\mathrm{FWS}_{\mathrm{GPTJ}}\), the reported scores are task-dependent. In text completion, SWEET is highest at \(0.556\), with KGW at \(0.540\), DiPmark at \(0.552\), and EXP-edit at \(0.369\). In question answering, KGW is top at \(0.411\), followed by DiPmark at \(0.408\), SWEET at \(0.393\), and EXP-edit at \(0.369\). In summarization, KGW scores \(0.419\), SWEET \(0.451\), DiPmark \(0.432\), and EXP-edit \(0.405\). Relative to un-watermarked generation, GPT-Judger average drops are reported as \(-11.58\%\) for coherence, \(-12.42\%\) for relevance or completeness, and \(-11.09\%\) for factual accuracy.

Human correlation is substantially stronger for GPT-Judger-based FWS than for the automatic variant. On the QA task, \(\mathrm{FWS}_{\mathrm{GPTJ}}\) versus human judgments yields Pearson \(r=0.84\), whereas \(\mathrm{FWS}_{\mathrm{auto}}\) yields \(r=0.25\). The sensitivity analysis shows that heavier factuality weightings, from \(\alpha/\beta=0.4/0.2\) up to \(0.46/0.08\), further improve GPTJ-human correlation up to \(r\approx 0.841\).

The paper’s practical guidelines are explicit. It recommends always including a factuality-heavy composite metric such as FWS when evaluating watermarking for medical LLMs; using GPT-Judger or similarly capable LLM evaluators with periodic expert validation; selecting watermark parameters such as \(\gamma\) and \(\delta\) in KGW or SWEET via joint optimization that penalizes FWS drop as well as detection ability; preferring schemes that preserve low-entropy tokens or selectively skip watermarking on critical medical entities; and enforcing an FWS threshold, for example \(>0.4\) on the GPTJ scale, below which watermark deployment is disabled. It also notes that current human validation was QA-only and sample-limited, and suggests future extensions to entity-level or evidence-trace subscores, multi-reference factuality checks, and dynamic weighting by medical subdomain.

Taken together, the two 2025 usages of FWS share a common design principle: factuality is explicitly up-weighted rather than treated as a side effect of coherence. They diverge, however, in operationalization. MAD-Fact makes factuality a claim-centric, debate-verified, hierarchy-weighted \(F_1\) problem for long-form generation, whereas the medical watermarking work makes it a criterion-weighted scalar for deployment-oriented quality control. A common misconception is therefore that FWS denotes a single canonical metric. The literature here instead supports the narrower conclusion that “FWS” is best understood as a family name for evaluation schemes that deliberately privilege factual correctness in settings where fluent errors are safety-critical.

Source: https://www.emergentmind.com/topics/factuality-weighted-score-fws