Ref-Free Counterfactual Financial Misinformation

• The paper presents an automated system that identifies counterfactual perturbations in financial news without relying on external references, enabling direct analysis of internal consistency.
• It leverages a benchmark dataset (RFC Bench) of 1,404 Yahoo Finance articles to evaluate LLM detection performance across numerical, sentiment, causal, and flipping perturbations.
• Performance metrics reveal robust task accuracy in paired comparisons, while exposing challenges in outright reference-free detection under zero-shot settings.

Reference-free counterfactual financial misinformation detection is the automated identification of false or manipulated information within financial news content, performed without recourse to any external references, ground-truth databases, or paired comparative inputs. In this paradigm, a system must determine, given a single financial news paragraph, whether it asserts factual content or contains a counterfactual perturbation—an internally consistent yet misleading revision intended to simulate misinformation. The RFC Bench framework provides both benchmark data and task definitions to systematically evaluate and advance LLMs in this domain, exposing key failure modes and offering a platform for methodological innovation (Jiang et al., 7 Jan 2026).

1. Formal Task Definition and Objective

Let $\mathcal{N}$ denote the set of all real-world financial news paragraphs, and $N \in \mathcal{N}$ be an input paragraph. The core “reference-free misinformation detection” (Task 1) objective is, for each $N$, to predict whether it is “True” (factual) or “False” (contains a counterfactual perturbation). This is formalized with label space $\mathcal{Y} = \{\text{True}, \text{False}\}$, and LLM inference proceeds via

$$y^* = \arg\max_{y \in \mathcal{Y}} P_{\text{LLM}}(y|N)$$

Fine-tuning, where applied, minimizes binary cross-entropy loss. The comparative diagnosis task (Task 2) is supplied for reference: given both a factual paragraph $N^f$ and its minimally perturbed counterpart $N^m$, the model classifies the manipulation mechanism $$m \in \mathcal{M} = \{\text{Numerical}, \text{Flipping}, \text{Sentiment}, \text{Causal}\}$$ via:

$$m^* = \arg\max_{m \in \mathcal{M}} P_{\text{LLM}}(m|N^f, N^m)$$

The reference-free setting explicitly withholds $N^f$—ground truth variants—forcing the system to rely strictly on internal coherence and world modeling.

2. Dataset Composition and Perturbation Design

RFC Bench comprises 1,404 unique Yahoo Finance articles (April 25–Dec 15, 2025), spanning 223 U.S. stock tickers. Paragraph-level samples are generated post-rewriting by LLMs (notably GPT-4.1), then subjected to expert review, dual-annotator QA, and adjudication to ensure factual integrity and perturbation validity. The dataset contains 1,845 original–perturbed pairs, with category breakdowns:

• Flipping: 532
• Numerical: 703
• Sentiment: 253
• Causal: 338

A hard-case subset (123 pairs) isolates ambiguous or edge cases. Category correctness annotation achieves AC1 ≥ 0.963, rewrite validity AC1 ≥ 0.896. Perturbation mechanisms—operationalized through category-specific GPT-4.1 rewriting under hard constraints—include:

• Flipping: Semantic reversal (e.g., profit to loss)
• Numerical: Quantitative changes (e.g., “8%” to “28%” growth)
• Sentiment: Polarity intensification (e.g., “compress profit margins” to “bankruptcy crisis”)
• Causal: Alteration of event relations (e.g., changing causal drivers)

3. Benchmark Evaluation and Metrics

Task 1 (reference-free) is evaluated as binary classification, while Task 2 (comparative) is multi-class. Metrics are rigorously defined:

• Accuracy: $N \in \mathcal{N}$0
• Precision: $N \in \mathcal{N}$1
• Recall: $N \in \mathcal{N}$2
• Macro-F1: Arithmetic mean of class-wise F1
• Matthews Correlation Coefficient (MCC): Measures overall correlation between predictions and ground truth
• Invalid-rate: Share of outputs not matching label vocabulary

For Task 2, metrics additionally include Macro-F1 (averaged over all manipulation categories) and per-class AUROC when class imbalance arises. All models are evaluated zero-shot, with few-shot (2-/8-shot) ablations. Outputs not exactly matching label sets are counted as invalid.

4. Model Architectures, Prompting, and Results

Investigated models include both open-source (LLaMA 3.1, Qwen3, Qwen2.5) and closed-source (GPT-4.1, GPT-5 Mini, DeepSeek) LLMs. Experiments use unified zero-shot prompts:

“You are a financial misinformation detector. Check whether the following paragraph is false [True/False]. Output one label.”

Data generation uses calibrated hyperparameters per manipulation class (e.g., temp=0.2, top_p=0.8 for Flipping). Key findings for zero-shot performance:

Model	Task 1 Acc.	Macro-F1₁	Inv.	Task 2 Acc.	Macro-F1₂	Inv.
LLaMA 3.1-8B	51.0 %	0.467	1099	57.5 %	0.499	886
Qwen3-8B (non-think)	53.0 %	0.528	441	85.0 %	0.790	53
GPT-4.1	52.7 %	0.507	0	96.9 %	0.965	2
DeepSeek-reasoner	53.6 %	0.528	3	93.6 %	0.937	0
GPT-5 Mini	45.2 %	0.450	208	97.7 %	0.970	0

Best zero-shot accuracy for Task 1 is 53.6% (DeepSeek-reasoner), with Macro-F1 ≤ 0.53, MCC ≈ 0.07. Task 2 (comparative context) achieves 85–97% accuracy; GPT-5 Mini attains 97.7% accuracy, Macro-F1 0.97. Invalid outputs are a major failure mode for smaller models, reaching 1,099 (LLaMA 8B) on Task 1; closed models exhibit near-zero invalids.

5. Failure Modes and Diagnosis

In the reference-free setting, LLMs exhibit several notable failure patterns:

• Over-skepticism: Systematically reject plausible, forward-looking projections as “unsupported.”
• Over-accommodation: Accept stylized but subtly manipulated narratives, failing to detect minor numerical or logical drifts.
• Temporal anchoring: Discount the veracity of future-dated or prospective claims without sufficient reasoning.
• Surface cue reliance: Prioritize linguistic fluency, journalistic style, or perceived authoritativeness over actual logical or factual coherence.

The observed “accommodation-first” failure mode highlights a structural weakness: current LLMs tend to let misleading yet coherent text pass scrutiny due to lack of comparative or grounding context, rather than examining internal consistency to guard common ground.

6. Comparative Context and Its Implications

Providing both the factual and perturbed paragraphs transforms detection into a local “difference” attribution task, at which LLMs excel due to their capacity for textual alignment and edit localization. Such paired format:

• Transforms the task from world modeling to direct comparison of semantic/explicit changes
• Makes minor, surface-plausible manipulations salient
• Reduces reliance on full external knowledge and world modeling

Performance increases dramatically in comparative diagnosis with accuracy up to 97.7% (GPT-5 Mini), indicating fundamental limitations in reference-free reasoning for factuality.

7. Recommendations and Directions for Future Research

Addressing the limitations surfaced by RFC Bench requires methodological advancements:

• Self-consistency checks and internal belief modeling to detect latent counterfactual shifts absent explicit reference.
• Training with adversarial, semantically minimal perturbations to increase sensitivity to subtle meaning changes.
• Coherence-based probing (e.g., self-contrastive inference, chain-of-thought prompting) to surface unsupported or logically inconsistent assertions.
• Reference-free calibration objectives such as model uncertainty quantification and introspection mechanisms to minimize overconfidence on manipulated text.
• Scaling beyond paragraph level by evaluating multi-paragraph consistency, document-level contradictions, and, where possible, retrieval-augmented techniques.

The RFC Bench framework and findings delineate critical gaps and challenges in achieving robust, reference-free counterfactual detection in financial information streams, providing a structured testbed and foundation for future research (Jiang et al., 7 Jan 2026).