Reproducing FACTER: Fairness via Conformal Thresholding and Prompt Repair

Abstract: Fayyazi et al. (2025) recently proposed FACTER, a model-agnostic framework designed to jointly enforce fairness and statistical coverage in LLM-based recommendation through conformal thresholding and iterative prompt repair. In this work, we conduct a reproducibility study of the FACTER framework across diverse architectures and dataset sparsity levels, evaluating both the original open-ended generation task and a constrained re-ranking extension. Under the strict reproduction, we observe a divergence in recommendation utility, which we trace to underspecified target-set evaluation in the original study. We then use the constrained re-ranking setting to evaluate FACTER when the candidate set is fixed, and introduce a static Fair Zero-Shot baseline to isolate the contribution of the iterative prompt repair loop. Our analysis shows that FACTER consistently reduces adaptive-threshold violation counts, but that these reductions are not consistently reflected under the fixed threshold or in global fairness metrics. In the constrained ranking setting, static fairness instructions achieve comparable semantic-parity outcomes to FACTER's dynamic repair loop, suggesting that the additional online repair mechanism provides limited benefit in this formulation. All code and reproduction artifacts are available at https://github.com/oscar-omlf/facter-repr.

• The paper re-implements FACTER across three open-weight LLMs and two recommendation datasets, finding adaptive violations fall sharply but fixed-threshold fairness and global metrics show little improvement.
• The study reports major utility divergence from the original results in open-ended generation, including NDCG@10 of .035 versus .458 on MovieLens-1M, largely due to underspecified evaluation choices.
• In constrained re-ranking, a static fairness prompt performs comparably to FACTER, suggesting online monitoring adds limited value while fairness constraints impose an alignment-related utility cost.

Background and motivation

FACTER (Fairness-Aware Conformal Thresholding and Prompt Engineering) is a model-agnostic, black-box framework for LLM-based recommendation that operationalises individual fairness as counterfactual stability: holding non-protected user context $x$ fixed, recommendations should remain semantically invariant to perturbations of protected attributes $\bm{a}$. The framework calibrates a conformal threshold over a composite nonconformity score combining predictive error and a fairness penalty against "counterfactual neighbours," then monitors an online query stream, injecting mined "AVOID" constraints into the system prompt when violations occur and adapting the threshold via exponential smoothing. The original study reported substantial fairness-violation reductions (up to 95.5%) without meaningful utility loss.

This paper presents a reproducibility study of FACTER conducted at the University of Amsterdam (2606.28620). The authors re-implemented the full pipeline from scratch across three open-weight backbones (Llama 3-8B, Llama 2-7B, Mistral-7B), two datasets (MovieLens-1M and Amazon Movies & TV), and two task formulations: a strict reproduction of the original open-ended generation setting and a new constrained re-ranking extension with a fixed candidate pool of 40 items. They also introduce a static Fair Zero-Shot baseline to isolate the contribution of the iterative repair loop, since the original UP5 baseline could not be reproduced due to missing checkpoints and absent prefix-tuning code.

Reproduction methodology

The reproduction follows the original hyperparameters ($\lambda = 0.7$, $\gamma = 0.95$, $\tau_\rho = 0.9$, $\alpha = 0.1$, buffer size $M = 50$, rule-mining threshold $n = 3$) and uses the same fine-tuned MPNet sentence encoder for all semantic representations. Offline calibration constructs a similarity matrix $W$ over cross-group context embeddings, computes the nonconformity score $S_i = d_i + \lambda \Delta_i$ (predictive error plus weighted maximum semantic distance to neighbour recommendations), and sets $\bm{a}$0 as the $\bm{a}$1-quantile. Online monitoring applies the update $\bm{a}$2 upon violations and mines frequent $\bm{a}$3 bias patterns for prompt injection.

The authors document substantial paper–code discrepancies that forced reconstruction choices: the threshold-update logic is internally inconsistent in the paper; the evaluation setup must be multi-target (the paper's reported NDCG@10 exceeding Recall@10 is mathematically impossible under single-target evaluation); SNSR/SNSV are defined in the paper as Frobenius norms of internal weights but implemented as group-centroid cosine distances; and the target-set construction for multi-target relevance was never specified, so the authors heuristically defined it as the user's next 10 interactions. These choices are documented exhaustively, and the study is explicitly framed as a best-effort audit rather than a definitive adjudication.

Utility divergence in open-ended generation

Under the strict open-generation protocol, reproduced utility collapses relative to the original study: NDCG@10 of .035 versus the original .458 on ML-1M, and .004 versus .351 on Amazon, with near-zero Recall@10 throughout. The Valid@k metric partially explains this — only 72–75% of generated items map to valid catalogue entries on ML-1M and 88–90% on Amazon — but task difficulty alone does not fully account for the magnitude of the gap given matched backbones. The authors attribute the divergence primarily to underspecified evaluation protocol details, particularly string-matching tolerances and the multi-target relevance-set definition: a more permissive relevance definition would widen the target space and raise measured utility. Importantly, FACTER's marginal utility loss relative to the zero-shot baseline does replicate directionally, consistent with the original claim of limited utility degradation.

The implication is that FACTER's empirical utility claims are highly sensitive to evaluation protocol rather than being robust properties of the method itself, and the open-generation utility claim (C2) remains unreproduced from public artefacts alone.

Fairness results and threshold dependence

Several fairness trends align qualitatively with the original study. FACTER reduces adaptive-threshold violation counts dramatically: from 42.7 (Neutral) to 9.0 on ML-1M and from 107.7 to 3.0 on Amazon under $\bm{a}$4, with monotonic reduction across iterations and consistent behaviour across all three LLM backbones (adaptive violations between 7.3 and 9.0 versus baseline ranges of 39.3–45.3). This provides partial support for the model-agnosticism claim (C4).

However, the central finding is that these reductions are largely an artefact of threshold adaptation rather than behavioural improvement. Under the fixed, population-calibrated threshold $\bm{a}$5, FACTER's violations closely mirror the baselines: 35.3 versus Neutral's 42.7 on ML-1M, and 110.7 versus 107.7 on Amazon — the latter meaning FACTER records more fixed-threshold violations than the unconstrained baseline. Global fairness metrics show no commensurate improvement: CFR worsens under FACTER (.702 vs. .667 Neutral on ML-1M open generation; .553 vs. .533 in re-ranking), and SNSR differences are marginal. Component analysis refutes a "null penalty" explanation: the fairness penalty contributes roughly 50% of violating instances' scores, yet over three iterations the average fairness penalty for violations increases from 0.872 to 1.283 while predictive error slightly improves. The authors conclude that injected "AVOID" constraints fail to override the model's predictive distribution, and FACTER satisfies its coverage guarantee primarily by relaxing its own acceptance region.

A further structural limitation surfaced: the original $\bm{a}$6 multi-attribute group constraint renders SNSR incomputable on Amazon (zero qualifying groups among 294 possible intersections) and nearly so on ML-1M (two groups). Single-attribute analysis reveals attribute-dependent disparity — age shows the largest gap on ML-1M open generation (.143 Neutral, reduced to .088 by FACTER) — but also that ranking naturally constrains semantic spread far more than open generation.

Static versus dynamic prompting

In the constrained re-ranking setting, the static Fair Zero-Shot baseline achieves outcomes comparable to the full dynamic loop: on ML-1M, both yield SNSR of .024, with fixed-threshold violations of 55.7 (Fair) versus 60.3 (FACTER); on Amazon, Fair attains better CFR (.681 vs. .707) with similar fixed violations. Meanwhile, re-ranking restores utility to realistic levels (Neutral NDCG@10 of .466 on ML-1M, with FACTER at .438). The implication is that in discriminative ranking contexts, explicit static fairness instructions account for most of the achievable effect, and the computational overhead of online monitoring yields diminishing returns. The authors note this conclusion may not extend to unconstrained generative settings with higher stochastic drift, and observe an "alignment tax": strict fairness constraints cost utility regardless of whether applied statically or dynamically.

Limitations and open questions

The study concedes several constraints plainly. The multi-target relevance set is heuristic and may differ from the original protocol, leaving the utility gap unresolved. The candidate pool size of 40 was fixed by compute budget without sensitivity analysis. UP5 could not be reproduced, so comparisons rely on originally reported reference values. Amazon's protected attributes are synthetic, so its fairness results measure stability under artificial perturbations rather than real demographic disparities. All fairness metrics depend on a fine-tuned MPNet encoder whose own biases may propagate into thresholds and scores. The re-ranking formulation assumes relevant items are present in the candidate set, so upstream retrieval bias remains unaddressed. Finally, the original hyperparameters were preserved without dataset-specific tuning of the predictive/fairness trade-off. Open questions include whether decoding-time semantic bounds can outperform soft prompt constraints, whether online scoring can operate without ground-truth labels ("blind" inference), and how to calibrate fairness penalties to prevent the score-conservation effect observed here.

Conclusion

This reproducibility study delivers a nuanced verdict on FACTER. Conformal thresholding reliably bounds internal violation counts and the framework transfers across backbones and sparsity regimes, supporting claims C3 and C4 in part. However, the headline utility result did not reproduce under matched conditions, and the apparent fairness gains prove largely attributable to adaptive threshold relaxation: fixed-threshold violations and global metrics (SNSR, CFR) do not improve, and component analysis shows the fairness penalty worsening despite prompt repair. In controlled ranking settings, a simple static fairness prompt matches the dynamic loop. The study's principal value lies in demonstrating how sensitive black-box fairness interventions are to underspecified evaluation protocols, and in providing documented reconstruction choices, baselines, and open-source artefacts for subsequent work.