Keyword-Prioritized Correction Score (KPCS)

• KPCS is a family of evaluation metrics that assigns higher weight to answer‐ or domain-critical keywords, ensuring that key tokens drive the overall score.
• It comprises dual implementations—KPQA for generative question answering using learned, question-conditioned token weights, and MedGuards for medical error correction via annotated clinical keywords.
• Empirical studies show that KPCS variants correlate more strongly with human judgments than standard metrics, particularly in contexts requiring numeric precision and clinical accuracy.

Keyword-Prioritized Correction Score (KPCS) denotes a family of evaluation metrics that assign disproportionate importance to answer- or domain-critical terms when judging generated or corrected text. In the cited literature, the name is used for two related but formally distinct constructions. In generative question answering, KPCS is instantiated from KPQA as a KPQA-weighted precision/recall/$F_1$ over aligned tokens or spans, where question-conditioned keyphrase weights determine which parts of an answer matter most for correctness. In medical error correction, KPCS is defined as a convex combination of exact-match recall over annotated clinical keywords and the average of ROUGE-1, BERTScore, and BLEURT, with a tunable balance parameter $\alpha$ (Lee et al., 2020, Ma et al., 24 Jun 2026).

1. Terminological scope and conceptual basis

Both uses of KPCS are motivated by the same evaluation problem: conventional lexical and semantic similarity metrics can assign high scores to outputs that preserve broad surface overlap while missing the terms that determine factual or clinical correctness. The central idea is therefore keyword prioritization rather than uniform token treatment.

Source	Core score	Keyword mechanism
"KPQA: A Metric for Generative Question Answering Using Keyphrase Weights" (Lee et al., 2020)	Weighted precision/recall/$F_1$ over aligned tokens or spans	Question-conditioned KeyPhrase Weights (KPW) per token
"MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction" (Ma et al., 24 Jun 2026)	$\alpha \cdot K + (1-\alpha)\cdot M$	Exact-match recall over annotated clinical keywords

In the KPQA setting, the evaluation target is correctness of free-form generated answers. The metric reweights overlap so that matches on high-importance tokens dominate the score, while function words and other low-importance tokens contribute little. In the MedGuards setting, the evaluation target is reliability of medical correction, especially whether the corrected sentence reproduces clinically critical concepts such as a diagnosis, medication, therapeutic action, or causal organism.

A common source of confusion is to treat KPCS as a single canonical equation. The literature here instead shows a shared design principle—prioritize critical keywords—implemented in two different ways. This suggests that KPCS is better understood as a keyword-sensitive evaluation paradigm than as a uniquely fixed formula.

2. KPQA-derived KPCS in generative question answering

The KPQA formulation begins from a question $Q = [q_1, \ldots, q_\ell]$, a reference answer $R = [r_1, \ldots, r_n]$, and a generated answer $A = [a_1, \ldots, a_m]$. Inputs are lowercased and tokenized with BERT’s WordPiece tokenizer using bert-base-uncased. No explicit stemming, lemmatization, stopword removal, or punctuation stripping is performed; instead, punctuation and function words are assigned low importance by the learned keyphrase predictor (Lee et al., 2020).

KPQA predicts token-level importance conditioned on the question. For generated tokens, $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$, where $w_i \in [0,1]$ is the importance of $a_i$ for answering $\alpha$0. For reference tokens, $\alpha$1, where $\alpha$2 is the importance of $\alpha$3 for answering $\alpha$4. These weights are produced as output probabilities of a softmax layer, so they are normalized across the answer tokens, and the metric denominators include the sums of weights to keep the score well-defined regardless of normalization.

The paper states that a Keyword-Prioritized Correction Score can be instantiated directly from KPQA by defining a weighted precision/recall/$\alpha$5 over token or span matches using the KPQA keyphrase weights. Let $\alpha$6 be indices of generated tokens aligned to some reference token, and let $\alpha$7 be reference tokens that have at least one aligned generated token. Then

$\alpha$8

and

$\alpha$9

This definition abstracts away the alignment choice. Exact match over WordPiece tokens yields a strict factual criterion. LCS-based alignment yields order-sensitive correctness. Embedding similarity, including a max-cosine threshold for embeddings, allows paraphrase-sensitive matching. The paper presents this abstraction as a direct generalization of the weighted precision/recall/$F_1$0 structure already used in BLEU-1-KPQA, ROUGE-L-KPQA, and BERTScore-KPQA.

The underlying KPQA scoring family makes the same principle concrete. BLEU-1-KPQA uses weighted unigram precision with exact token equality and no clipping. ROUGE-L-KPQA computes weighted LCS-based precision, recall, and $F_1$1-measure using binary LCS membership. BERTScore-KPQA computes weighted cosine-similarity matching using contextual embeddings from bert-large-uncased-whole-word-masking-finetuned-squad, first hidden layer. In each case, the score is driven by KPW rather than uniform or IDF-style token weighting.

3. MedGuards KPCS in medical error correction

The MedGuards formulation defines KPCS over a reference corrected sentence $F_1$2, a model-corrected sentence $F_1$3, and a set of required clinical keywords extracted from $F_1$4. The metric proceeds in three steps (Ma et al., 24 Jun 2026).

The first step is a keyword check. Let $F_1$5 be the number of required keywords in $F_1$6, and let $F_1$7 be the number of those required keywords that appear in $F_1$8. The paper defines

$F_1$9

It also states that if no keyword is present in $\alpha \cdot K + (1-\alpha)\cdot M$0, the score is set to zero. By construction, $\alpha \cdot K + (1-\alpha)\cdot M$1.

The second step is semantic and lexical similarity:

$\alpha \cdot K + (1-\alpha)\cdot M$2

The third step is the weighted combination:

$\alpha \cdot K + (1-\alpha)\cdot M$3

with $\alpha \cdot K + (1-\alpha)\cdot M$4. The paper sets $\alpha \cdot K + (1-\alpha)\cdot M$5 for all main experiments and studies $\alpha \cdot K + (1-\alpha)\cdot M$6 in sensitivity analysis.

This version of KPCS uses exact string matching for the required keywords. The paper explicitly notes that this can miss clinically equivalent synonyms and suggests future integration of medical ontologies. It does not specify handling of multiple occurrences, partial matches, lemmatization, abbreviations, numeric tolerances and units, or negation polarity. Consequently, keyword robustness arises primarily from the $\alpha \cdot K + (1-\alpha)\cdot M$7 component, especially BERTScore and BLEURT, rather than from the keyword check itself.

Keyword selection is annotation-driven rather than automatically extracted. In MEDEC, categories include Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism, and the keywords correspond to the entities constituting the error. Non-error factual information is excluded from keywords. The paper does not specify automatic NER, UMLS concept extraction, or multilingual lexicons.

4. Computation, alignment, and implementation

In the KPQA setting, the practical pipeline begins with raw $\alpha \cdot K + (1-\alpha)\cdot M$8, tokenizes all three inputs with the bert-base-uncased WordPiece tokenizer, computes $\alpha \cdot K + (1-\alpha)\cdot M$9 and $Q = [q_1, \ldots, q_\ell]$0, selects an alignment rule, and then computes weighted precision, recall, and $Q = [q_1, \ldots, q_\ell]$1. The alignment rule can be unigram exact match for BLEU-1-KPQA, LCS dynamic-programming alignment for ROUGE-L-KPQA, or cosine similarity over contextual embeddings for BERTScore-KPQA. The paper also gives high-level pseudocode for these variants and notes the corresponding complexities: naive weighted unigram precision is $Q = [q_1, \ldots, q_\ell]$2 but reducible to $Q = [q_1, \ldots, q_\ell]$3 with a hash map over reference tokens; ROUGE-L-KPQA uses LCS dynamic programming with $Q = [q_1, \ldots, q_\ell]$4; BERTScore-KPQA computes embeddings in $Q = [q_1, \ldots, q_\ell]$5 and pairwise cosine maxima in $Q = [q_1, \ldots, q_\ell]$6 (Lee et al., 2020).

The KPQA weight predictor is a BERT-based classifier using bert-base-uncased with an additional fully-connected layer of 768 units and tanh, followed by a softmax output over answer tokens. The input is the concatenation [question, [SEP], answer-sentences]. The task is to classify whether each token in the answer sentence belongs to the gold answer span for that question. Training uses cross-entropy over tokens against binary in-span versus out-of-span labels. The training data combine SQuAD v1.1, HotpotQA excluding yes/no questions, and the MS-MARCO QA subset, split 9:1 for train and dev. Hyperparameters are max input length 256, AdamW with learning rate $Q = [q_1, \ldots, q_\ell]$7, batch size 16, and 5 training epochs, selecting the checkpoint with minimum dev loss and training five times to pick the best-performing model. Reported token-level dev $Q = [q_1, \ldots, q_\ell]$8 is 55.81 on SQuAD and 59.26 on MS-MARCO QA. Training takes approximately 150 minutes per epoch on a single 1080 Ti GPU, evaluation about 5 minutes, and model size is about 109.4M parameters.

In the MedGuards setting, computation is simpler. One takes $Q = [q_1, \ldots, q_\ell]$9 and the annotated keyword set $R = [r_1, \ldots, r_n]$0, counts exact string matches of each required keyword in $R = [r_1, \ldots, r_n]$1, computes $R = [r_1, \ldots, r_n]$2, computes ROUGE-1, BERTScore, and BLEURT using standard implementations, averages them to form $R = [r_1, \ldots, r_n]$3, and then combines $R = [r_1, \ldots, r_n]$4 and $R = [r_1, \ldots, r_n]$5 with $R = [r_1, \ldots, r_n]$6. The only tunable hyperparameter is $R = [r_1, \ldots, r_n]$7. No case normalization, edit-distance thresholds, synonym mapping, unit normalization, or negation handling is specified for the keyword check. The paper states that the authors release code at https://github.com/congboma/MedErrBench; the KPQA paper releases code at https://github.com/hwanheelee1993/KPQA.

These implementation differences are substantive. KPQA-derived KPCS requires a learned question-conditioned salience model and an explicit alignment operator. MedGuards KPCS instead assumes annotated keywords and uses exact-match recall as a lightweight keyword fidelity signal. This suggests that the two formulations occupy different points in the design space: one learns token salience, the other relies on curated domain-critical keyword sets.

5. Empirical behavior and validation

The KPQA study evaluates correctness on two newly annotated generative QA datasets: the MS-MARCO NLG subset, with average answer length 16.6 tokens, and AVSD, with average answer length 9.4 tokens. For context, it also reports results on NarrativeQA with average answer length 4.7 and SemEval-2018 Task 11 with average answer length 2.5. Human judgments are collected by training two GenQA models per dataset—UniLM and MHPGM for MS-MARCO, MTN and AMF for AVSD—sampling 500 questions per model, discarding exact matches to the ground truth, and obtaining 10 MTurk ratings per sample on a 5-point Likert scale. Up to five noisy ratings per sample are removed by z-score filtering, and the remaining ratings are averaged. Krippendorff’s alpha is $R = [r_1, \ldots, r_n]$8 for MS-MARCO and $R = [r_1, \ldots, r_n]$9 for AVSD, with average retained annotators per sample $A = [a_1, \ldots, a_m]$0 and $A = [a_1, \ldots, a_m]$1 respectively (Lee et al., 2020).

Against these human judgments, KPQA-integrated metrics achieve markedly higher correlation than standard metrics. On MS-MARCO, BLEU-1 has $A = [a_1, \ldots, a_m]$2 and $A = [a_1, \ldots, a_m]$3, ROUGE-L has $A = [a_1, \ldots, a_m]$4 and $A = [a_1, \ldots, a_m]$5, METEOR has $A = [a_1, \ldots, a_m]$6 and $A = [a_1, \ldots, a_m]$7, CIDEr has $A = [a_1, \ldots, a_m]$8 and $A = [a_1, \ldots, a_m]$9, and BERTScore has $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$0 and $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$1. The KPQA variants improve to BLEU-1-KPQA $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$2, $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$3; ROUGE-L-KPQA $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$4, $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$5; and BERTScore-KPQA $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$6, $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$7. On AVSD, baseline BLEU-1 reaches $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$8, $\mathrm{KPW}(Q,A) = (w_1,\ldots,w_m)$9; ROUGE-L $w_i \in [0,1]$0, $w_i \in [0,1]$1; and BERTScore $w_i \in [0,1]$2, $w_i \in [0,1]$3, while KPQA variants reach BLEU-1-KPQA $w_i \in [0,1]$4, $w_i \in [0,1]$5; ROUGE-L-KPQA $w_i \in [0,1]$6, $w_i \in [0,1]$7; and BERTScore-KPQA $w_i \in [0,1]$8, $w_i \in [0,1]$9. All reported correlations have $a_i$0.

The same paper reports several robustness studies. Training KPQA without MS-MARCO data lowers MS-MARCO correlations, but KPQA variants still outperform baselines: BLEU-1-KPQA $a_i$1, $a_i$2; ROUGE-L-KPQA $a_i$3, $a_i$4; and BERTScore-KPQA $a_i$5, $a_i$6. Removing question context from KPQA training also degrades performance. On MS-MARCO, BLEU-1-KP reaches $a_i$7, $a_i$8; ROUGE-L-KP reaches $a_i$9, $\alpha$00; and BERTScore-KP reaches $\alpha$01, $\alpha$02, all below the full KPQA variants. For question types on MS-MARCO, gains are reported across PERSON, NUMERIC, DESCRIPTION, LOCATION, and ENTITY, with the largest gains for NUMERIC. On a 100-example multiple-sentence subset, BERTScore-KPQA reaches $\alpha$03 and $\alpha$04, compared with BERTScore at $\alpha$05 and $\alpha$06. In model-ranking “win-lose” comparisons where the human score gap is at least 2, KPQA variants match human judgments on 74–76% of pairs in MS-MARCO versus 61–68% for baselines, and on 77–81% of pairs in AVSD versus 71–78% for baselines.

A qualitative example illustrates the intended behavior. For the question “How many steps are involved in a hypothesis test?”, the reference is “Four steps are involved in a hypothesis test.” and the generated answer is “There are seven steps …”. Human judgment is 0.063, BLEU-1 is 0.778, ROUGE-L is 0.713, BLEU-1-KPQA is 0.057, and ROUGE-L-KPQA is 0.127. The metric assigns high weight to the numeric token, so the weighted score approaches zero for the incorrect answer. In another example on “How long do I need to walk in order to lose a pound?”, BERTScore with IDF gives 0.72, BERTScore-KPQA gives 0.93, and human judgment is 0.94.

The MedGuards paper evaluates KPCS on MEDEC and on MedErrBench, a multilingual benchmark spanning English, Arabic, and Chinese. On MEDEC, KPCS improves substantially when MedGuards is used with several LLM backbones: Gemini 2.0 Flash improves from 0.370 to 0.472, GPT-4o-mini from 0.210 to 0.461, Doubao-1.5-thinking-pro from 0.219 to 0.554, Deepseek-V3 from 0.257 to 0.556, Gemini 2.5 Flash Lite from 0.127 to 0.385, and Llama-3.3-70B-Instruct from 0.360 to 0.487 (Ma et al., 24 Jun 2026). The paper states that all reported improvements for MedGuards are statistically significant with $\alpha$07 on the MEDEC results where p-values are shown.

Sensitivity analysis over $\alpha$08 yields the following KPCS values for MedGuards variants: for $\alpha$09, $\alpha$10; for $\alpha$11, $\alpha$12; for $\alpha$13, $\alpha$14. The paper concludes that $\alpha$15 yielded KPCS scores most consistent with human Mean Opinion Score trends.

6. Interpretation, limitations, and methodological significance

The principal strength shared by both formulations is sensitivity to content-bearing words that determine correctness. In the KPQA setting, the token weights are question-conditioned and learned to highlight answer-span tokens. This focuses the score on keywords and keyphrases that determine whether a generated answer is correct. In the medical setting, the keyword recall term directly penalizes missing diagnoses, medications, therapeutic actions, and causal organisms, while the semantic and lexical term preserves broader similarity assessment (Lee et al., 2020, Ma et al., 24 Jun 2026).

The KPQA-derived formulation has several explicit limitations. It relies on accurate keyphrase prediction; when the model misidentifies salient tokens, the weighted score can diverge from human judgment. The paper notes such cases especially for NUMERIC questions, including overweighting units such as “million” or “years.” Performance also degrades under domain shift, although it remains above baselines in the reported out-of-domain experiment. Exact-match variants such as BLEU-1-KPQA and ROUGE-L-KPQA do not directly account for synonymy, and while BERTScore-KPQA mitigates this through embeddings, KPQA itself does not incorporate external synonym lists. Gains are smaller for multi-sentence or DESCRIPTION-type answers, where keyphrases are more diffuse, and there is no explicit penalty term for contradictions beyond absence of matched keyphrases.

The MedGuards formulation has a different limitation profile. The keyword component uses unweighted recall over required keywords and exact string matching only. The paper explicitly states that this is not robust to clinically equivalent synonyms or morphological variants and proposes ontology-based extensions as future work. The score also depends on the quality and coverage of keyword annotations; misannotations or incomplete keyword sets can distort evaluation. In addition, contextual correctness beyond string presence remains only partly modeled. Dosages, units, and negation are not explicitly handled in the keyword check.

A worked example from MedGuards clarifies this behavior. If the reference contains “Continue heparin drip, Continue HFNC …” with required keyword set $\alpha$16, and the prediction instead contains “Continue lepirudin drip, Continue HFNC …”, then $\alpha$17, $\alpha$18, and $\alpha$19. With $\alpha$20, the final score becomes $\alpha$21. If the prediction preserves “heparin,” then $\alpha$22, and with high similarity the final score approaches 1. This demonstrates that KPCS can strongly penalize failure on the critical term even when the remaining sentence is similar.

Methodologically, the two formulations show different routes to the same goal. The KPQA line treats importance as a learned, question-conditioned distribution over tokens and then defines weighted overlap relative to an alignment rule. The MedGuards line treats importance as externally annotated domain knowledge and combines keyword recall with conventional generation metrics. A plausible implication is that KPCS-style evaluation can be adapted either by learning salience from supervision or by curating a set of required terms, depending on task structure and available annotations.