Anecdoctoring: Narrative Bias in Research

• Anecdoctoring is the practice of selectively using anecdotes and narrative framing to lend undue evidentiary weight to preferred methods.
• It involves constructing multiple plausible post-hoc interpretations from real-data examples, which can bias methodological comparisons and findings.
• The phenomenon impacts both research integrity and clinical decision-making, urging the need for objective safeguards and complementary evidence.

Anecdoctoring denotes a family of practices in which anecdotal material, narrative framing, or individualized judgment acquires evidentiary force beyond what the underlying data warrant. In its most explicit metascientific sense, it is the “storytelling fallacy”: “the selective use of anecdotal domain-specific knowledge to support the superiority of specific methods in real data examples,” treated as a previously underappreciated form of researcher degrees of freedom in methodological research (Mandl et al., 5 Mar 2025). The broader literature attached to the term is heterogeneous. It includes formal reasoning from unreported observations in diagnosis, computational systems that synthesize or red-team narratives, critiques of clinician discretion relative to trial averages, and designs that deliberately combine anecdotes with scientific evidence rather than treating them as mutually exclusive (Peot et al., 2013, Cuevas et al., 23 Sep 2025, Shahn et al., 1 May 2026, Bali et al., 18 Feb 2026).

1. Definition and semantic range

In the narrowest usage, anecdoctoring refers to selective post-hoc storytelling around real-data examples in methodological papers. In adjacent literatures, related usages concern how anecdotes, omissions, defaults, and clinician intuitions shape inference and decision-making.

Usage	Core formulation	Representative source
Methodological research	Selective anecdotal domain knowledge used to support a preferred method	(Mandl et al., 5 Mar 2025)
Diagnostic inference	Learning from observations not made in open-probe interactions	(Peot et al., 2013)
ICU decision bias	Subconscious, culturally shaped numerical preferences in treatment settings	(Ercole, 2019)
Automated red-teaming	Multilingual, place-specific disinformation prompts built from fact-checked narratives	(Cuevas et al., 23 Sep 2025)
VLM associative bias	“What makes a doctor” should be grounded in context rather than appearance	(Magid et al., 2024)
Physician discretion	How often doctors outperform the better-performing trial arm	(Shahn et al., 1 May 2026)
Health information support	Co-presentation of scientific evidence and anecdotes to address uncertainty	(Bali et al., 18 Feb 2026)

This range matters because the term does not name a single doctrine. In some settings it is a warning about bias; in others it is a modeling principle, a benchmark for red-teaming, or a design strategy for uncertainty management. The common thread is not simply “anecdotes,” but the way anecdotes or anecdote-like structures interact with inference, interpretation, and action.

2. A researcher-degrees-of-freedom problem

The paper on methodological research places anecdoctoring within the literature on researcher degrees of freedom (RDF). RDF refers to the flexibility scientists have in making decisions related to data analysis, with such choices arising across preprocessing, modeling, summarization, and reporting. Combined with selective reporting, RDF can yield over-optimistic claims and false positives. The storytelling fallacy extends this logic from analytic choices to interpretive choices: a real-data analysis produces results; the author can choose among many plausible domain-specific interpretations; the chosen interpretation is typically the one that best supports the preferred method; competing interpretations that would favor another method are ignored (Mandl et al., 5 Mar 2025).

The central mechanism is multiplicity of plausible post-hoc narratives. The paper characterizes the bias quasi-formally as

$$\text{Real data result} \;\to\; \text{multiple plausible interpretations} \;\to\; \text{select one favorable interpretation}.$$

The bias need not require conscious cherry-picking from a fixed menu of stories. The paper explicitly notes that the interpretation may be constructed after seeing the results, so that another equally plausible story might have been written had the method performed differently. The selectivity therefore resides in narrative construction and emphasis, not only in dataset choice or metric choice.

The paper also distinguishes the storytelling fallacy from ordinary expert judgment. Expert judgment is inherently subjective, but the identified problem is not subjectivity alone. It is the combination of subjective interpretation, multiplicity of possible stories, and selective reporting of the most favorable story. For that reason, the fallacy is described as analogous to selective reporting and spin, but specifically targeted at post-hoc interpretive narratives grounded in domain knowledge.

3. Interpretive instability in real-data examples

The methodological paper argues that real-data examples are often treated as evidence of method quality, especially when their outputs can be embedded in a plausible substantive narrative. Its claim is not that real examples are useless. Rather, a real example rarely has a single unambiguous “truth story,” and different methods may highlight different variables, SNPs, sensors, or patterns, each of which can support a convincing explanation. Real-data examples are therefore interpretively unstable: the same dataset can support competing stories, each making a different method appear more meaningful or more correct (Mandl et al., 5 Mar 2025).

The first illustrative case concerns Mendelian randomisation and pleiotropic SNP detection. One method detected one SNP and another detected none. The same SNP could be narrated in two opposing ways: as evidence of horizontal pleiotropy, supporting the method that flagged it, or as a known vitamin D proxy that should remain in the analysis, making the other method look like a false-positive detector. The example is important because the real-data result does not uniquely validate either method; the apparent validation depends on which domain-specific story is foregrounded.

The second case concerns SARS-CoV-2 breath analysis with E-nose sensors. Two predictive models ranked different sensors as important. One biologically plausible narrative emphasized sulfur compounds and H$_2$S-related biology; another emphasized methane/hydrogen-related gasotransmitters. Again, both outputs could be made meaningful. What changes is the rhetorical burden placed on each output by selective interpretation.

These examples support the paper’s warning that domain plausibility is not empirical validation. A plausible story does not prove superiority, and readers may mistake narrative coherence for methodological evidence. The stated consequence is a biased picture of method performance, with a risk of non-replicable methodological conclusions and overconfidence in new methods.

4. Diagnostic omission, physician discretion, and hidden clinical bias

Outside methodological comparison studies, related work shows that anecdote-like reasoning can be formalized, bounded, or exposed rather than merely denounced. In diagnostic reasoning, “what a patient does not say can be evidence.” Peot and Shachter model open-probe interactions such as “What is bothering you?” by augmenting a Bayesian network with report nodes, one for each symptom, and introducing reportability

$$P_o(S_i) = P(\text{Report}_Q(S_i)=\text{true} \mid S_i=\text{present}),$$

reporting bias

$$B_o(S_i) = \frac{P(\text{Report}_Q(S_i)=\text{true} \mid S_i=\text{present})}{P(\text{Report}_Q(S_i)=\text{true} \mid S_i=\text{absent})},$$

and the likelihood ratio for no report

$$\lambda_g(S_i) = \frac{P(\neg \text{Report}_Q(S_i) \mid S_i=\text{present})}{P(\neg \text{Report}_Q(S_i) \mid S_i=\text{absent})}.$$

The model formalizes the intuition that unmentioned symptoms can lower the probability of diseases that would likely have produced them, thereby improving the differential diagnosis and follow-up questioning in open-probe settings (Peot et al., 2013).

A different clinical literature examines whether physician discretion genuinely outperforms trial-based recommendations. “Trust Me, I’m a Doctor?” studies a randomized trial nested within an observational cohort and defines the population gain score

$$G \equiv V_U - \bigl(pV_T + (1-p)V_C\bigr) = \operatorname{cov}\bigl(A,\, Y(1)-Y(0)\bigr),$$

where $V_T = \mathbb{E}[Y(1)]$, $V_C = \mathbb{E}[Y(0)]$, and $V_U = \mathbb{E}[Y]$. The main identified quantity is the fraction of physicians whose strategy beats “treat everyone with the better-performing trial treatment,” formalized via $\delta_d \equiv V_d - V_T$. Under the paper’s assumptions, the sharp upper bound is

$_2$0

The result is explicitly balanced: physician judgment may exploit effect modifiers, but the data can still imply that only a limited fraction of physicians outperform the trial winner, especially when $_2$1 lies much closer to $_2$2 than to $_2$3 (Shahn et al., 1 May 2026).

A third strand documents hidden non-evidence-based influences in bedside care. A retrospective observational analysis of ICU ventilator settings found that patients spent significantly longer with odd choices for PEEP ($_2$4, $_2$5), RR ($_2$6, $_2$7), and Pinsp ($_2$8, $_2$9), alongside aversion to the number 13 for RR and Pinsp but not for PEEP, where 13 was more prevalent than expected by chance. The paper frames these findings as evidence of subconscious treatment bias and a reminder that even highly technical clinical environments are not immune to defaults, heuristics, or culturally shaped preferences (Ercole, 2019).

Taken together, these papers show that anecdoctoring-adjacent phenomena in medicine are not uniform. Some are rationally modellable, as with open-probe omission; some are only partially identifiable, as with physician discretion; and some are bias signatures in behavior itself, as with numerical preference in ICU settings.

5. Computational reinterpretations in AI safety and multimodal ML

In AI safety, “Anecdoctoring” names a red-teaming framework for generating disinformation-style adversarial prompts that are grounded in real-world misinformation and adapted across language and place. The pipeline begins with 9,815 fact-check articles from three languages and two geographies, restricted to January 1, 2022 through December 31, 2024 and excluding items marked true. Claims are embedded with Cohere’s embed-multilingual-v3.0, reduced with UMAP to $$P_o(S_i) = P(\text{Report}_Q(S_i)=\text{true} \mid S_i=\text{present}),$$0 dimensions, and clustered with HDBSCAN separately for each language-geography pair, yielding 501 narrative clusters. GPT-4o is then used to extract a knowledge graph for each cluster, and an attacker LLM generates prompts that instruct a target model to produce a viral tweet aligned with the narrative. A separate Judge LLM scores attack success on a 5-point harm scale, with attack success rate defined as the fraction of outputs receiving a harm score of at least 4. On GPT-4o, the KG-augmented method attains ASRs of 0.881 for English/USA, 0.878 for Spanish/USA, 0.880 for English/India, and 0.945 for Hindi/India. The paper reports that clustering improves ASR by about 14% on average relative to individual claims, and KG augmentation adds about 9% over clustered claims without graphs, while also improving interpretability (Cuevas et al., 23 Sep 2025).

A separate multimodal line addresses what the paper describes as “anecdoctoring” or doctor-associated bias in VLMs. The problem is that a model such as CLIP may encode “doctor” through correlations with visible demographic traits rather than through contextual cues. To mitigate this, the paper generates synthetic counterfactual images by creating diverse base images from neutral profession captions, segmenting body regions that reveal protected attributes with SegFormer fine-tuned on the ATR dataset, and inpainting those regions with SDXL 1.0 while preserving profession-specific context. The fine-tuned model, denoted $$P_o(S_i) = P(\text{Report}_Q(S_i)=\text{true} \mid S_i=\text{present}),$$1, is trained to pull counterfactual images of the same profession together in embedding space and is later blended with original CLIP weights by interpolation. The reported effect is a 40–66% improvement in MaxSkew, MinSkew, and NDKL for image retrieval tasks while maintaining similar downstream performance. The paper’s core claim is that “what makes a doctor” should be grounded in background, attire, and held objects rather than skin color, body type, or other visible attributes (Magid et al., 2024).

These computational reinterpretations differ sharply from the storytelling fallacy in methodological statistics, but both retain the same structural concern: narrative or label meaning can be made to hinge on spurious, culturally local, or selectively foregrounded associations unless the representational pipeline is explicitly constrained.

6. Anecdotes and evidence as complementary supports

Not all work associated with anecdoctoring treats anecdotes as a bias to be eliminated. “Evidotes” proposes an information support system that augments peer health posts with both PubMed scientific articles and Reddit anecdotes, using three user-selectable lenses: Dive Deeper, Focus on Positivity, and Big Picture. The system is implemented as a Chrome browser extension and follows a three-stage retrieval-augmented generation pipeline: lens-specific query generation, parallel multi-source retrieval, and lens-specific synthesis. Retrieved results come from PubMed’s Best Match ranking and Reddit search sorted by relevance; GPT-4o-mini then produces short claims with direct quotes and source support (Bali et al., 18 Feb 2026).

The paper’s central conceptual contribution is “information symbiosis.” Anecdotes make research accessible and contextual, while research helps filter and generalize peer stories. In a mixed-methods study with 17 chronic illness patients, Evidotes improved self-reported information satisfaction from 3.2 to 4.6 and reduced self-reported emotional cost from 3.4 to 1.9 compared with baseline browsing. The system produced 293 synthesized claims, about 2.25 claims per post, with 2.12 sources per claim on average; 96.7% of quote-source pairs were judged faithful, and 98.2% of claims were topically relevant overall.

This literature is significant because it complicates any simple opposition between anecdote and evidence. Here the design goal is not to replace peer narratives with formal evidence, but to structure their interaction so that each source type performs a distinct epistemic function. Anecdotes provide relatability, emotional realism, and contextualization; research provides credibility, generalization, and filtering.

7. Methodological implications and safeguards

The methodological paper that introduced the storytelling fallacy does not argue for eliminating real-data examples. Its recommendation is to keep them, but to treat them as illustrative rather than determining evidence. It also recommends interpreting real-data examples together with simulation studies where the ground truth is known, using multiple datasets where plausibility matters, and preferring objective criteria for plausibility over ad hoc narratives. The same paper cautions that LLMs make it easier to generate convincing literature-based stories, thereby increasing the risk of storytelling bias, and urges a move away from the implicit “one beats them all” expectation in method comparison (Mandl et al., 5 Mar 2025).

Several common misconceptions follow from this literature. One is that any use of domain knowledge in interpretation is itself a fallacy; the more precise claim is that the problem arises when subjective interpretation is combined with multiplicity of plausible stories and selective reporting. Another is that individualized clinical judgment automatically vindicates anecdotal override of trial evidence; the partial-identification results on physician performance show that such confidence can require stronger evidence than intuition alone. A third is that technical systems are naturally insulated from informal or cultural bias; ICU ventilator-setting preferences and multilingual disinformation red-teaming both indicate otherwise (Shahn et al., 1 May 2026, Ercole, 2019, Cuevas et al., 23 Sep 2025).

A plausible implication is that anecdoctoring is best understood not as a single error but as a recurrent evidentiary pattern. It appears whenever narrative plausibility, omitted alternatives, or individualized intuition are allowed to stand in for independently validated performance. Yet the literature also suggests a constructive counterpart: anecdotes can be useful when their status is explicit, their complementarity with formal evidence is designed rather than assumed, and their inferential role is limited to illustration, uncertainty management, or adversarial evaluation rather than proof of superiority.