- The paper evaluates 12 open-weight LLMs using 341 digital twins and finds major gaps in initial belief distributions, with only 0.31 maximum matched-human Spearman correlation versus a human baseline of KL = 0.006.
- The paper shows that LLMs are more socially conformist than humans, producing stronger belief shifts toward peer opinions in 11 of 12 comparisons and potentially overstating consensus in simulations.
- The paper finds partial success in reproducing whom humans follow but weaker belief-based homophily, while larger models do not consistently improve fidelity, cautioning against using LLMs as human proxies in opinion-dynamics research.
This paper evaluates whether LLMs can serve as faithful proxies for human belief formation and change in networked settings. The authors replicate a prior human-subjects experiment on belief dynamics (2605.18781) using 12 open-weight LLMs spanning four families (Gemma, Llama, Qwen3, gpt-oss) and both non-thinking and thinking variants. Their answer is negative and systematic: LLMs fail to reproduce initial human belief distributions, are substantially more conformist than humans when exposed to peer opinions, and only partially reproduce homophilic network restructuring.
Experimental design
The study builds on an established three-stage experiment in which participants (a) rate statements on a 5-point Likert scale, (b) observe peers' ratings and optionally update their beliefs, and (c) select individuals to "follow" for subsequent rounds. The authors construct 341 "digital twins" — LLM agents conditioned on each participant's demographics and Big-5 personality traits — covering two topics ("immigration" and "oil and fuel"), yielding 1,023 samples across three rounds per twin. Each round is treated as an independent sample to keep prompts within context windows of smaller models and prevent error cascading; within a round, the model retains memory of its earlier stages. All models run locally via Ollama at temperature 1.2 for privacy reasons.
Evaluation uses KL divergence and Wasserstein distance between belief distributions, Mann-Whitney U tests, Spearman correlations for one-to-one comparisons, a "social influence" metric (correlation between peer-average disagreement and belief change), a Follow Signal (mean rating of followed peers), and Belief Network Distance (absolute difference between own initial belief and Follow Signal), which proxies homophily.
Finding 1: Initial belief distributions are not reproduced
Against a human-vs-human split-half baseline of KL = 0.006 and Wasserstein = 0.064, all twelve models show far larger divergence; the Mann-Whitney U test rejects distributional equality in 10 of 12 cases. The failure is structured rather than random: non-thinking models overestimate mean human belief (e.g., gemma3_4b at 3.10 vs. human 2.25), while all thinking models underestimate it, peaking at Likert 0 (qwen3_4b mean 0.86). The authors attribute this partly to reasoning capabilities or knowledge cutoffs, since two-thirds of the statements are false. Individual-level emulation is weak: the best Spearman correlation between any model's and its matched human's initial rating is only 0.31 (llama3.3_70b). This implies that even persona-conditioned agents with limited persona information cannot recover population-level attitude distributions, let alone individual ones.
Humans change beliefs less than any LLM; LLMs shift responses toward observed peer opinions more readily. The social influence correlation is higher for LLMs than humans in 11 of 12 comparisons (human average ≈ 0.32 vs. non-thinking average 0.47), significantly so by Fisher r-to-z in 8 of those 11. One outlier is qwen3_4b, whose social influence correlation is essentially zero (-0.02). Cross-model correlations of belief change are weak, indicating no consistent pattern in how different models update. The authors connect this asymmetry to documented sycophancy in LLMs (Sharma et al., 2023), where models defer to user-provided opinions even against ground truth. The implication is direct: simulations built on such agents will exaggerate consensus formation and understate the belief rigidity characteristic of human populations.
Finding 3: Partial success in network selection, failure in homophily
Stage 3 results are mixed. The Follow Signal distributions of LLMs closely resemble humans', with high Spearman correlations between model and human follow preferences — suggesting LLMs can emulate what humans select. However, Belief Network Distance is consistently higher for LLMs than humans (non-thinking average 1.14 vs. 0.95; thinking average 1.14 vs. 0.95), with significant Mann-Whitney differences in several models (e.g., llama3.2_3b at 1.60, p < 0.001). Because homophily is defined relative to one's own beliefs, this indicates that LLM selections are less anchored to their prior attitudes than human selections are. Notably, gemma3_27b, gemma3_4b, and llama3.3_70b could not produce the strict JSON required for stage 3 and were excluded from these analyses.
Scaling behavior
Contrary to scaling-law expectations, larger parameter counts do not reliably improve fidelity to human benchmarks. For example, gptoss_120b shows worse Wasserstein distance (1.06) than smaller models, and qwen3_1.7b outperforms qwen3_32b on KL divergence. The authors suggest that emulating belief dynamics requires reproducing psychological processes such as motivated reasoning, which differ from the reasoning, retrieval, and instruction-following tasks for which scaling laws were established.
Limitations and open questions
The paper concedes several constraints. Personas consist only of demographics and Big-5 traits; richer personas (as in generative-agent work with extensive interviews) may perform better, but the point of the design was to test whether digital twins work with minimal information — and they do not fully. Models lack memory across rounds, and only open-source models were tested, leaving closed-source performance unexamined. Whether cross-round memory, elaborate personas, or proprietary models would close the observed gaps remains an open empirical question raised but not answered by this paper.
Conclusion
Across 12 models, LLMs systematically diverge from humans in belief dynamics: they misestimate initial belief distributions along a thinking/non-thinking axis, exhibit stronger conformity than humans under social influence, and select network peers without the degree of belief-based homophily humans display. These results caution against deploying LLM agents as human proxies in opinion-dynamics simulations, particularly where polarization, persuasion, or intervention design depend on realistic belief rigidity and network anchoring.
