A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing

Abstract: LLMs can shift human beliefs across high-stakes domains, but most persuasion studies rely on pre/post belief change. These endpoint measures identify whether persuasion occurred, yet miss where and how beliefs moved within a dialogue. We present PERSUASIONTRACE, a framework for studying persuasion in human-LLM interaction. Built on a web-based experimental platform, PERSUASIONTRACE contributes a tool for multi-turn persuasion studies and a process-level evaluation protocol: it records multi-turn belief reports from human or simulated targets of persuasion, annotates persuader turns with rhetorical dimensions (logos/pathos/ethos), and evaluates simulators by fidelity to real human belief dynamics. Using this framework, we find that human targets group into two clusters of multi-turn belief updates and exhibit susceptibility to rhetorical strategies, and that LLMs are persuasive across generic and personalized topics, text and audio modalities, and multi-turn interactions. Prior work has chiefly used vanilla-prompted LLMs to simulate human targets, but we show that these simulators fail to replicate human belief dynamics. We introduce a Bayesian-network simulated target that maintains an explicit latent belief state over time so each persuader message yields cognitively realistic belief updates. In human-likeness evaluation, our Bayesian target scores near a human reference (81 vs 80), while baseline LLM targets score substantially lower (64). PERSUASIONTRACE reframes persuasion evaluation from endpoint movement alone to process fidelity, providing a stronger basis for scientific analysis and safer optimization of persuasive systems.

• The paper introduces the PERSUASIONTRACE framework, capturing high-resolution, multi-turn belief updates in human-LLM dialogues.
• Key findings show that while pathos drives significant belief shifts, ethos surprisingly exhibits a negative association with persuasion outcomes.
• A Bayesian-network simulator is developed to mimic human persuasion responses, outperforming traditional LLM baselines in fidelity.

Process-Level Modeling of Human Persuadability in Multi-Turn LLM Dialogues

Motivation and Limitations of Pre/Post Persuasion Measurement

Traditional persuasion studies employing LLMs overwhelmingly rely on pre/post belief-change assessments, which restrict analysis to scalar endpoint effects. This approach obscures the granular temporal dynamics of belief updating, impeding identification of conversational mechanisms driving persuasion. As LLMs increasingly permeate domains where influencing human beliefs is pivotal—including politics, misinformation, and decision support—probing the process-level evolution of beliefs during interaction becomes requisite.

The PERSUASIONTRACE Framework: Multi-Turn Belief Tracing and Rhetorical Annotation

PERSUASIONTRACE is designed to capture high-resolution belief trajectory data in human-LLM persuasion dialogues. The platform integrates:

• Multi-turn belief elicitation: Targets report beliefs (on a 0-100 scale) after every persuader turn, producing belief-update trajectories $(b_{\text{pre}}, b_1, ..., b_t, b_{\text{post}})$.
• Rhetorical dimension annotation: Persuader messages are annotated for logos, pathos, and ethos using an ordinal scale via LLM pipelines.
• Modality and proposition coverage: Experiments range across standard, personalized, and control propositions; interface includes both text and audio.

Empirical analysis demonstrates that LLM persuaders (gpt-5) effect significant belief shifts over generic, personalized, and audio modalities compared to control dialogues (mean persuasion delta significantly higher, $p < 0.001$ for text and personal, $p=0.002$ for audio; $N=255$) and reveals heterogeneity in susceptibility to rhetorical strategies. Clustering belief trajectories exposes two distinct update patterns: low-shift (near-zero movement) and high-shift (large early movement followed by partial regression/stabilization), with pathos positively associated with high-shift cluster membership.

Regression analysis indicates a negative association for ethos with persuasion delta (b = -0.097, $p=0.048$), while logos and pathos lack significant impacts within annotated cohorts. This finding notably contradicts conventional expectations that credibility appeals facilitate persuasion.

Simulation of Human Persuadability: Bayesian-Network Target Model

The paper addresses the inadequacy of vanilla LLMs as simulatees for persuasion targets by introducing a Bayesian-network (BN) simulated target. This model:

• Explicitly maintains latent belief states over time, grounded in proposition-specific belief graphs (average of 3.45 nodes per graph) derived from LLM outputs and fitted CPTs.
• Updates beliefs at each turn using atomized argument extraction, rhetoric-weighted evidence scaling, and Bayesian propagation.
• Instantiates target personas with differential rhetorical susceptibilities (logos, pathos, ethos).

Quantitative evaluation of simulator fidelity utilizes transcript-level human-likeness scoring by an LLM judge (gpt-5.4). The BN target attains a human-likeness score near human reference (81 vs 80), while unstructured LLM and structure-conditioned LLM targets score substantially lower (64.7, 64.2). Forced-initialization replay analysis reveals BN targets yield lowest strict conditional replay errors, though gaps vs human LOO are modest.

Simulator Diagnostics: Stance Bias, Naive Responsiveness, and Policy Sensitivity

The BN simulator outperforms baselines on key diagnostic metrics:

• Stance bias—BN target exhibits least asymmetry in for-vs-against movement across matched proposition/belief pairs ($0.077$ vs $0.154$, $0.236$).
• Naive responsiveness—BN resists trivial persuasion, showing negative naive-excess movement ($-0.069$), while LLM baselines are overly responsive.
• Policy ranking sensitivity—Policy ordering of frontier LLM persuaders varies dramatically depending on the simulator, cautioning against optimization with simulators lacking human fidelity.

Implications and Future Directions

Process-level tracing and simulation recalibrate scientific analysis and optimization of persuasive AI systems. Endpoint metrics risk optimizing against targets that are insufficiently human-like, misranking persuasive strategies, and favoring manipulative or fragile approaches. Simulator fidelity-based evaluation is paramount for safe deployment and regulatory benchmarking.

Future work should extend longitudinal and relational measurement of persuasion, exploit richer mental-state modeling (including dynamically evolving belief graphs), scale experimentation, and probe transferability of simulator-optimized policies to real targets. Utilization of multi-turn belief trajectories and process-level diagnostics can serve as levers for more robust and ethical optimization and auditing of persuasive systems.

Conclusion

PERSUASIONTRACE reframes persuasion research from endpoint-only movement to process fidelity, offering a rigorous methodology for analyzing belief-update dynamics and evaluating persuasive system safety and efficacy. The Bayesian-network simulated target sets a benchmark for human-likeness in simulators, enabling process-faithful optimization and deepening understanding of heterogeneous susceptibility to rhetorical mechanisms. Process-level measures and fidelity diagnostics should be incorporated into future responsible AI persuasion protocols.

(2606.05330)