SocSci210: Standardized Social Science Dataset

Updated 4 July 2026

SocSci210 is a large, standardized dataset of over 2.9 million responses from 210 social experiments, capturing diverse personas and experimental conditions.
It uses an LLM-driven reconstruction pipeline to convert open-source experimental data into a unified, instruction-style format for behavioral prediction.
The dataset supports fine-tuning of models with protocols like SFT and DPO, yielding improved distributional alignment and fairness in social science simulations.

SocSci210 is a large, standardized dataset of individual-level responses from open-source social science experiments, constructed to support finetuning LLMs for human behavior prediction at scale. In the formulation of the paper that introduces it, SocSci210 is the central data contribution underpinning a broader effort to build “simulated participants” that better match empirical response distributions in social science experiments. The dataset reconstructs experiments from the NSF Time-sharing Experiments for the Social Sciences repository into a common persona–condition–outcome–response format, yielding 2.9 million responses from 400,491 participants across 210 experiments, with 1,197 outcomes, 1,194 conditions, and 5,998 unique stimuli (Kolluri et al., 6 Sep 2025).

1. Definition, scope, and substantive domain

SocSci210 is described as a standardized dataset of open-source social science experiments reconstructed from the TESS repository. TESS is characterized as containing peer-reviewed, nationally representative, high-powered experiments spanning multiple social science domains, including economics, political science, and behavioral psychology. The dataset was built because, in the authors’ view, the main bottleneck in using LLMs as simulated participants is not only prompting quality but also the absence of broad, high-quality, individual-level training data that captures how real people respond under experimental manipulations (Kolluri et al., 6 Sep 2025).

The paper defines each experiment as

$E = ([c_1, \ldots, c_j], [o_1, \ldots, o_i]),$

with participants answering one or more questions tied to condition–outcome pairs, and each datapoint as a tuple

$(P, c, o, r),$

where $P$ is the participant persona, $c$ the experimental condition, $o$ the outcome question, and $r$ the recorded response. The learning task is expressed as

$F(P,c,o) \implies r.$

Component	Value
Reconstructed experiments	210
Participants	400,491
Individual responses	2.9 million
Outcomes	1,197
Conditions	1,194
Unique stimuli	5,998

The dataset is intentionally standardized around binary and ordinal outcomes. The paper states that usable outcomes were primarily ordinal and binary questions, and that the corpus was restricted to those response types to standardize training and evaluation while losing little data. This design makes SocSci210 a broad but still structured resource: it is not a general archive of all experimental outputs, but a corpus optimized for comparable behavioral prediction tasks across many studies (Kolluri et al., 6 Sep 2025).

2. Source archive and reconstruction pipeline

SocSci210 was sourced from the public Open Science Framework repository for TESS. At scrape time, the authors pulled 443 publicly available OSF projects associated with the TESS account; after deduplication of nested subprojects, they identified 321 unique studies. Of these, the automatic reconstruction system successfully reconstructed 210 studies (Kolluri et al., 6 Sep 2025).

The reconstruction pipeline was designed to avoid manual intervention. It used an LLM-based “data construction agent” powered by OpenAI o4-mini-high. The workflow had four explicit stages. First, source files were downloaded and standardized: data files were converted to CSV and PDFs and DOCX files were converted to text. Second, the full study context was provided to the agent, including the paper description, data files, codebook, and stimuli. Third, the agent identified all experimental conditions, identified all outcome questions, and wrote and executed parsing code to merge and clean CSV files, reconstruct participant records, and map conditions and outcomes to each participant response. Fourth, the code-generation stage used a generate-and-test cycle, iterating until the script successfully reconstructed the study (Kolluri et al., 6 Sep 2025).

A reconstruction counted as successful only if two checks were met: the parsing code compiled and executed without errors, and the parsing code generated non-empty outputs when run row by row. A study was included only if the system could find a description of the stimuli present for each condition, could reconstruct ordinal or binary questions, and could match outcome questions to a condition-specific stimulus. The paper notes that many failures were due not to conceptual ineligibility but to practical limitations of the LLM agent, especially failure to generate correct parsing code from long and complex source contexts (Kolluri et al., 6 Sep 2025).

This pipeline is substantively important because it converts fragmented experimental archives into a unified training substrate. The resulting corpus is not a simple metadata index; it is a participant-level reconstruction that preserves treatment assignment, outcome text, demographic persona, and observed answer in a consistent machine-readable form.

3. Internal structure and representational format

The paper repeatedly characterizes SocSci210 as containing three core components—persona, stimuli, and response—with stimuli tagged by both condition and outcome question. A datapoint therefore includes the participant demographic profile, the experimental condition or stimulus, the outcome question, and the observed participant answer. The available demographic fields vary by study, but the paper lists examples such as age, gender, education, employment, marital status, housing, geography, income, ideology, phone service, party identification, and ethnicity (Kolluri et al., 6 Sep 2025).

Operationally, the corpus is converted into instruction-style examples suitable for causal language-model finetuning. In the direct-prediction format, the model input includes a textual persona, followed by the condition stimulus and the outcome question, with an explicit response-format constraint such as “Only return an integer from 1 to 6.” For reasoning-augmented training, the target includes a trace segment:

<trace>
...reasoning...
</trace>
PREDICTION: <answer>

The paper describes these reasoning traces as “oracle reasoning traces” generated by GPT-4o-mini given the prompt and the known human response (Kolluri et al., 6 Sep 2025).

This representational choice is methodologically consequential. It means SocSci210 is not merely a tabular benchmark. It is also a text-serialized experimental corpus in which demographic attributes, treatment content, and outcome wording are jointly exposed to an autoregressive model. The paper presents this as a way to train a single model across studies, conditions, questions, and demographic profiles rather than fitting separate task-specific classifiers for each experiment.

4. Finetuning protocols, split regimes, and evaluation logic

SocSci210 is used to finetune two open-source instruction-tuned base models: LLaMA3-8B-Instruct and Qwen2.5-14B-Instruct. The resulting finetuned models are named Socrates-LLaMA-8B and Socrates-Qwen-14B. The paper also compares these against prompting-based baselines using GPT-4o (Kolluri et al., 6 Sep 2025).

The modeling setup includes supervised finetuning and direct preference optimization. Training is framed as causal language modeling or next-token prediction of the textual response rather than as fixed-head classification. The paper also explores DPO using contrastive pairs built primarily around demographic contrasts under the same condition and outcome, with the focal participant’s observed response treated as preferred over an alternative response from another persona (Kolluri et al., 6 Sep 2025).

The evaluation design is central to the meaning of SocSci210, because the corpus is explicitly organized for several generalization regimes.

Split protocol	Training setup	Test target
Unseen-study split	170 studies	40 held-out studies
Unseen-condition split	75% of conditions in selected studies	25% held-out conditions
Unseen-outcome split	75% of outcomes in selected studies	25% held-out outcomes
Unseen-participant / pilot-data split	50% participant-train, with 1%, 5%, 10%, 20%, 30%, 40%, 50% pilot subsets	Remaining 50% participant-eval

The paper argues that the most important metric for social science is not exact individual-level prediction, but distributional alignment under treatment conditions. Individual response prediction is measured by a normalized distance-based accuracy:

$\text{Acc.} = 1 - \frac{1}{N} \sum_{(P, c, o)} \frac{\bigl|F'(P,c,o) - r\bigr|}{r_{\max}-r_{\min}}.$

Distributional alignment is evaluated by standardizing response scales to $[0,1]$ , computing the Wasserstein distance between predicted and empirical distributions for each condition–outcome pair, averaging within studies, and then averaging over studies. Lower Wasserstein distance is better (Kolluri et al., 6 Sep 2025).

The paper also defines two reference points: a Uniform Guess lower baseline and an Empirical Best upper-performance bound estimated by bootstrap resampling of the human data. For fairness analysis, it defines demographic parity verbally as “the absolute gap between the highest- and lowest-performing demographic subgroups” (Kolluri et al., 6 Sep 2025).

5. Reported performance and empirical findings

On the primary unseen-study split, SocSci210 finetuning materially improves distributional alignment. The paper reports a distribution distance of 0.205 for the Qwen2.5-14B base model and 0.151 for Qwen2.5-14B + SFT, corresponding to a 26.3% improvement over the base model. The same table reports 0.174 for GPT-4o, so Socrates-Qwen-14B outperforms GPT-4o by 13.2%. For LLaMA3-8B, the reported distribution distance improves from 0.219 to 0.153 under SFT, a 30.1% improvement (Kolluri et al., 6 Sep 2025).

The strongest model on the paper’s preferred metric is Socrates-Qwen-14B, with Wasserstein distance 0.151 on unseen studies. The empirical-best bound is reported as 0.125. This positions the finetuned model as substantially better than its base model and better than GPT-4o on the distributional criterion the paper prioritizes for experimental simulation (Kolluri et al., 6 Sep 2025).

A notable result is that the best model for distributional alignment is not the same as the best model for individual response accuracy. On unseen studies, Qwen2.5-14B + DPO achieves the highest individual accuracy, 74.0, but has a weaker distribution score, 0.181. By contrast, Qwen2.5-14B + SFT reaches a lower individual accuracy, 69.5, but a much stronger distribution score, 0.151. The paper interprets this as evidence that reproducing a human response distribution is not equivalent to maximizing pointwise accuracy on single draws (Kolluri et al., 6 Sep 2025).

Within-study generalization is even stronger. On the unseen-condition split, the paper reports that LLaMA3-8B improves from 0.219 to 0.077 under SFT and to 0.063 under SFT with reasoning, corresponding to 64.8% and 71.2% improvements, respectively. On the unseen-outcome split, the same model improves from 0.224 to 0.125 under SFT and to 0.114 under SFT with reasoning, corresponding to 44.2% and 49.0% improvements (Kolluri et al., 6 Sep 2025).

The pilot-data analysis shows that performance often saturates around 10% of participant data. In seen studies, with just 10% of participant data, DPO raises individual accuracy from 71% to 75%, which the paper describes as about 13% relative error reduction. For subgroup performance, the paper reports a 28.5% average relative improvement in subgroup distributional alignment after finetuning and a reduction in demographic parity of about 10.5–10.6%. Appendix results further report subgroup improvements in Wasserstein distance such as 29.72% for female participants, 35.80% for male participants, 32.17% for Hispanic participants, 32.52% for White participants, and 32.70% for Black participants (Kolluri et al., 6 Sep 2025).

SocSci210 belongs to a broader computational social science trajectory in which large datasets, supervised learning, and standardized corpora are used to formalize and scale social inquiry. A plausible implication is that it exemplifies the later data-driven phase of computational social science described in work on the emergence of the field: a phase organized around large-scale digital data, machine learning, and computational analysis rather than earlier simulation-centered work (Bao et al., 2024). It also differs from other CSS data infrastructures that infer latent traits from digital traces, such as SES estimation from brand-following behavior on Twitter (He et al., 2022), or combine behavioral sensors with surveys in bounded social settings such as scientific conferences (Génois et al., 2022). SocSci210 is instead an experimental-response corpus organized around demographic personas, treatments, and closed-form outcomes.

The paper is explicit about several limitations. SocSci210 is built primarily from representative samples of the U.S. population and mostly closed-form questions. The authors do not demonstrate generalization to non-U.S. populations or to open-ended responses. Only 210 of 321 unique studies were reconstructed, and many failures were caused by missing or unclear stimulus descriptions, inability to map conditions to outcomes, or practical parsing failures of the LLM agent (Kolluri et al., 6 Sep 2025).

The paper also cautions that some stimuli are contentious, that models may still generate unsafe or incorrect outputs, and that improved realism may produce overconfidence in simulated results. Its methodological claim is therefore narrower than full behavioral equivalence: because social sciences routinely generate rich, topic-specific datasets, finetuning on such data could enable more accurate simulations for experimental hypothesis screening. Within that framing, SocSci210 is significant less as a universal model of human behavior than as a standardized empirical substrate for training and evaluating LLMs on social science experiments at scale (Kolluri et al., 6 Sep 2025).