Weighted Association Erasure in Clinical NLP

• Weighted Association Erasure is a metric that measures shifts in clinically significant, demographically-associated terms between reference and generated radiology reports.
• It employs statistical methods, including Dirichlet smoothing and standardized log-ratio comparisons, to detect semantic erasure and template collapse.
• Flexible weighting strategies in WAE reveal both omission and overrepresentation errors, providing actionable insights to improve fairness in clinical NLP models.

Weighted Association Erasure (WAE) quantifies the aggregate loss of clinically significant, demographically-associated terminology during automatic radiology report generation. It captures the extent to which a model’s outputs shift or erase key clinical word associations related to demographic variables—such as sex—compared to reference corpora. WAE complements standard validation metrics, exposing cases where models with high surface-level text similarity fail to preserve critical clinical specificity or fairness. It is particularly sensitive to phenomena like template collapse, semantic erasure of risk-sensitive language, and the inadvertent introduction of demographic bias through both deterministic and stochastic decoding strategies (Parikh et al., 2 Mar 2026).

1. Formal Definition and Mathematical Framework

Let $\mathcal{V}$ denote the clinical vocabulary under consideration. For each $w \in \mathcal{V}$, the standardized association displacement $z_\text{disp}(w)$ measures the shift in demographic association between reference and generated reports. This displacement is weighted by a scalar $\omega(w) \geq 0$, which reflects the term’s importance. The global WAE is defined as: $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$ The group-specific WAE, $\mathrm{WAE}_g$, restricts aggregation to words that are originally associated with group $g \in \{F, M\}$ (typically female or male), and disparity between groups is reported as $$\Delta \mathrm{WAE} = \mathrm{WAE}_F - \mathrm{WAE}_M$$.

The key quantities in WAE computation include:

• $c_g^{(D)}(w)$: Raw token count of $w$ for group $w \in \mathcal{V}$0 in corpus $w \in \mathcal{V}$1.
• $w \in \mathcal{V}$2: Dirichlet-smoothed count (typically $w \in \mathcal{V}$3).
• $w \in \mathcal{V}$4: Total smoothed tokens for group $w \in \mathcal{V}$5.
• $w \in \mathcal{V}$6: Smoothed unigram probability.
• $w \in \mathcal{V}$7: Log-ratio association.
• $w \in \mathcal{V}$8.
• $w \in \mathcal{V}$9: Standardized association score.
• $z_\text{disp}(w)$0.

2. Quantifying Clinical Term–Demographic Associations

The assessment of term–demographic associations proceeds in parallel for both the reference and predicted corpora, segmenting data by demographic group (e.g., sex). For each word:

• Token counts $z_\text{disp}(w)$1 are collected for both groups from each corpus $z_\text{disp}(w)$2.
• Dirichlet smoothing is applied to avoid zero counts.
• Smoothed probabilities are computed for each group and term.
• The log-ratio $z_\text{disp}(w)$3 indicates whether the term is more characteristic of one group; its sign encodes association (positive for female, negative for male), and its magnitude reflects strength in standard deviation units.
• Variance is estimated using a multinomial delta-method.
• Scores are standardized as $z_\text{disp}(w)$4 for subsequent analysis.

This construction enables explicit measurement of how model generations preserve, neutralize, or invert clinically meaningful language patterns associated with demographic groups.

3. From Clinical Association Displacement (CAD) to WAE

Clinical Association Displacement (CAD) is the primary per-term metric: it computes $z_\text{disp}(w)$5, indicating how much the association of each word shifts between the generated and reference corpora. WAE aggregates the squared displacements across the vocabulary, weighted by $z_\text{disp}(w)$6, so that larger shifts (regardless of direction) disproportionately affect the aggregate measure.

Two main weighting strategies are employed:

• Prediction-based ($z_\text{disp}(w)$7): Prioritizes terms the model actually generates.
• Reference-based ($z_\text{disp}(w)$8): Penalizes omissions of clinically significant reference terms more heavily.

This flexibility allows WAE to be tuned toward either omission or overrepresentation errors, reflecting different evaluation priorities in clinical NLP settings.

4. Computation and Step-by-Step Aggregation Procedure

Implementation of WAE follows these ordered steps:

1. Construct reference and predicted corpora, grouping tokens by demographic variable.
2. For each $z_\text{disp}(w)$9:
   • Compute Dirichlet-smoothed group counts and corresponding total token counts.
   • Calculate $\omega(w) \geq 0$0, $\omega(w) \geq 0$1, $\omega(w) \geq 0$2, and $\omega(w) \geq 0$3.
   • Derive $\omega(w) \geq 0$4 using reference and prediction statistics.
3. (Optional) Statistically flag terms with significant displacement, using two-sided tests and false discovery rate control, but include all terms for WAE aggregation.
4. Select $\omega(w) \geq 0$5 according to evaluation focus.
5. Compute global WAE using the formula in Section 1.
6. For group-specific analysis, restrict aggregation to words with $\omega(w) \geq 0$6.
7. Report differences $\omega(w) \geq 0$7 as disparity indicators.
8. Employ nonparametric bootstrap to estimate uncertainty and calibrate with permutation testing.

5. Interpretive Framework and Theoretical Properties

WAE is explicitly non-negative and equals zero only with perfect preservation of all reference associations. It is unbounded above; in practice, realized $\omega(w) \geq 0$8 rarely exceeds 5–6 due to finite counts. The global WAE can be construed as the expectation of the squared displacement variable under the weighting distribution $\omega(w) \geq 0$9.

Low WAE indicates robust maintenance of demographic-language patterns, while high WAE flags either widespread semantic erasure (dampening of key associations) or the emergence of new, potentially spurious biases. Both loss (erasure) and gain (artifactual associations) of demographic signal increase WAE, as the aggregation squares all displacements. Group-specific WAE calculations localize errors within clinically salient lexical subsets.

6. Empirical Example and Interpretation

A representative toy example with vocabulary $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$0 demonstrates WAE computation. Counts of each term by group in both reference and predicted corpora are smoothed, used to calculate association log-ratios and variances, standardized to $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$1-scores, and then differenced and normalized for $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$2. Using reference-based weighting, the WAE aggregates to a small value (0.016) when clinical terminology is largely preserved. Notably, total omission of a group-associated clinical term (e.g., “effusion”) in predictions would sharply increase its respective $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$3 and the overall WAE, reflecting severe clinical erasure. This calibration illustrates WAE’s sensitivity to both minor perturbations and gross neglect of demographic specificity.

Term	$$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$4, $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$5	$$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$6, $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$7
nodule	50, 10	5, 1
effusion	5, 25	0, 0
normal	80, 80	60, 60

Reference weight: $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$8, $$\mathrm{WAE} = \frac{ \sum_{w \in \mathcal{V}} \omega(w)\,[z_\text{disp}(w)]^2 }{ \sum_{w \in \mathcal{V}} \omega(w) }.$$9, $\mathrm{WAE}_g$0.

7. Applications and Contextual Significance

WAE is designed as a clinical-fairness diagnostic for radiology report generation, supplementing surface-level token overlap measures (e.g., BLEU, ROUGE) with a direct probe for semantic erasure and demographic bias. Its use is motivated by evidence that dominant validation metrics may obscure failures in clinical signal preservation, such as overreliance on generic templates that omit critical findings or demographic context. WAE analysis can reveal model behaviors like metric gaming, the tradeoff between deterministic and stochastic decoding strategies, and unintended demographic bias amplification or erasure (Parikh et al., 2 Mar 2026).

This suggests that incorporating WAE into model evaluation protocols significantly improves the transparency and clinical adequacy of automatic report generation, particularly for patient populations historically subject to diagnostic bias or underdiagnosis. A plausible implication is that future reporting standards should require lexical diversity and fairness analysis alongside traditional metrics to ensure both generalization and equity in clinical NLP systems.