BenCSSmark: Social Science LLM Benchmark

• BenCSSmark is a social science benchmark that aggregates 27 French datasets, organizing them by both NLP task type and social-scientific constructs.
• It categorizes tasks using multidimensional taxonomy—including genre, time period, and annotator details—to capture nuanced and context-dependent phenomena.
• The benchmark emphasizes construct validity and perspectivist evaluation, challenging LLMs to address ambiguous, culturally sensitive, and evolving social contexts.

BenCSSmark is a social-science-centered benchmark for evaluating LLMs, introduced to address the under-representation of social science tasks in mainstream LLM benchmarking. It aggregates datasets annotated by computational social scientists and organizes them not only by conventional NLP task type but also by the social-scientific concepts they operationalize, with the explicit aim of making evaluation more robust, transparent, and socially relevant. In its current form, it contains 27 datasets, all in French, and is framed as a living resource rather than a finished benchmark suite (Chatelain et al., 6 May 2026).

1. Rationale and intellectual setting

BenCSSmark is motivated by a diagnosis of contemporary benchmark culture: benchmarks do not merely measure model quality, but structure research agendas, reputations, and commercial outcomes. Within that regime, social science tasks remain largely absent, even though disciplines such as sociology, political science, history, economics, geography, anthropology, and demography routinely produce carefully annotated datasets. The benchmark is presented as a corrective to that omission, on the grounds that models optimized primarily on standard benchmark regimes risk being tuned to a narrow conception of language use and reasoning (Chatelain et al., 6 May 2026).

The benchmark’s stated motivation is both practical and epistemic. Practically, it gathers scattered datasets into a standardized evaluation resource. Epistemically, it seeks to make model assessment more faithful to the interpretive character of social-scientific inquiry. The paper argues that social science tasks stress-test models in ways generic benchmarks often do not: they involve ambiguous categories, meanings that shift across time and place, multiple valid perspectives, and annotations that are frequently expert-driven rather than crowd-sourced. A central implication is that social science evaluation is not treated as peripheral application work, but as a source of constraints on what counts as adequate language-model competence (Chatelain et al., 6 May 2026).

A recurrent claim in the benchmark’s framing is that the exclusion of such tasks is consequential rather than incidental. If social science datasets remain outside benchmark ecosystems, then direct comparison becomes difficult, task-specific progress remains fragmented, and benchmark-driven optimization continues to privilege tasks that fit prevailing evaluation conventions. BenCSSmark is therefore positioned as a bridge between computational social science and LLM evaluation, and as an argument for incorporating social-scientific validity and perspectivism into benchmark design (Chatelain et al., 6 May 2026).

2. Scope, corpus composition, and task coverage

At the time described in the paper, BenCSSmark comprises 27 datasets drawn from ongoing or published computational social science projects, all in French. The benchmark indexes each dataset along multiple axes: conventional NLP task category, the social-scientific concept being operationalized, and metadata describing genre, period, size, annotation unit, and annotator type. This multidimensional organization is intended to distinguish superficially similar NLP tasks that in practice encode different constructs and research questions (Chatelain et al., 6 May 2026).

The benchmark covers task categories including argumentative strategy detection, bias detection and bias labeling, concept detection, coreference resolution, frame detection, hate speech detection, other detection, quote detection, topic classification, and topic labeling. The paper also notes that its table organizes the datasets under 7 broad categories, including “other detection.” Rather than reducing social-scientific annotation to generic labels, the benchmark preserves task-specific conceptualization. Examples listed in the paper include appeal to authority and appeal to majority in radio opinion pieces and political speeches; non-neutral statements and bias type in Wikipedia articles about politicians; gender, social class, and inclusive language in academic social science abstracts; press political source intensity via coreference resolution; immigration, LGBT rights, and taxation framing in newspaper coverage; abusive, critical, and supportive political comments on Twitter; unattributed quotes and source diversity in newspaper articles; and thematic or policy-related categorization tasks (Chatelain et al., 6 May 2026).

The corpus is heterogeneous in both annotation format and textual genre. Annotation types include binary classification, multiclass classification, multilabel classification, span detection, and pairwise coreference resolution. Genres include social media, news, political discourse, broadcast discussions, broadcast news, electoral manifestos, Wikipedia, and academic or social science articles. The benchmark also spans written text and speech transcription, the latter introducing additional difficulty through ASR errors and multi-speaker formatting. Temporally, the material extends from the second half of the 20th century through 2025, with some data reaching back to 1945. The paper emphasizes that older datasets are especially useful for robustness testing because they capture archaic vocabulary, changing social categories, and distinct discourse conventions (Chatelain et al., 6 May 2026).

The benchmark further includes less standardized tasks that do not map neatly onto canonical NLP templates. These include detecting prescriptive writing in music-related news, identifying reform pledges in electoral manifestos, and detecting political forecasting in radio opinion pieces. This suggests that BenCSSmark is not limited to repackaging established NLP categories, but also aims to preserve task formulations that originate in substantive social-science research programs (Chatelain et al., 6 May 2026).

3. Design principles and annotation philosophy

BenCSSmark is explicitly governed by three principles. The first is to represent the diversity of social science perspectives. The paper stresses that a task label such as “frame detection” does not identify a single universal problem, because the construct depends on medium, historical period, and disciplinary interpretation. A frame-detection dataset built from contemporary social media and one built from early twentieth-century newspapers may share a label while differing substantially in construct validity. BenCSSmark responds by adding conceptual specificity rather than relying solely on generic task names (Chatelain et al., 6 May 2026).

The second principle is to reflect contextual, socio-cultural, and temporal variation. Social science treats meaning as dependent on setting, historical moment, and cultural background, and the benchmark attempts to mirror that orientation by collecting datasets across genres and periods instead of treating text as interchangeable. This is intended to make the benchmark useful for testing generalization beyond the narrow contexts in which models are commonly trained or evaluated. A plausible implication is that BenCSSmark treats distributional variation not as nuisance variance to be normalized away, but as part of the phenomenon under study (Chatelain et al., 6 May 2026).

The third principle is to preserve annotator diversity whenever possible. For datasets annotated by the benchmark authors themselves, two versions are created: one with a single adjudicated ground truth and another retaining all individual annotations. The paper situates this decision within a perspectivist approach, according to which disagreement is not automatically noise. Preserving individual labels enables both standard single-label evaluation and analysis of ambiguity or disagreement, and is aligned with a broader call for disaggregated data. The benchmark also accommodates datasets annotated in great detail by a single expert, which are common in social science but often awkward to incorporate into agreement-centric benchmark paradigms (Chatelain et al., 6 May 2026).

The data collection logic follows two complementary strategies: producing new annotated datasets with trained annotators while preserving both the adjudicated decision and the individual annotations, and reusing existing datasets that were annotated by a single expert in a highly detailed fashion. The associated metadata situates each dataset within its disciplinary context, temporal frame, text type, and, where possible, annotator characteristics. In methodological terms, BenCSSmark attempts to preserve the provenance of annotation rather than flattening it into a decontextualized label set (Chatelain et al., 6 May 2026).

4. Taxonomy, construct validity, and evaluation modes

A notable feature of BenCSSmark is its multidimensional taxonomy. Each dataset is indexed by standard NLP task type and by the social-scientific concept it measures. The benchmark thereby distinguishes between formal task similarity and substantive conceptual equivalence. This is important because the paper defines social science tasks as tasks that exhibit construct validity with respect to social scientific concepts or questions, and describes them as first-best operationalizations of those concepts. The benchmark’s taxonomy is thus intended not merely for cataloguing, but for preserving the relation between annotation and theory (Chatelain et al., 6 May 2026).

The paper does not present a new global scoring formula or a leaderboard-style evaluation framework. Instead, it describes an implicit two-mode evaluation structure. One mode is conventional: datasets with a single adjudicated ground truth support standard scoring. The other is perspectivist: datasets with preserved individual annotations support analysis of disagreement, ambiguity, and task difficulty. The paper explicitly notes the distinction between adjudicated “ground truth” and preserved disagreement, but does not introduce a formal metric to reconcile them. It cites disagreement-aware evaluation literature as relevant background, while also acknowledging that the translation of perspectivist insights into metrics remains unresolved in the benchmark itself (Chatelain et al., 6 May 2026).

This restraint is central to the benchmark’s design. BenCSSmark is not presented as a single-number evaluation suite in which heterogeneous tasks are collapsed into a universal scalar. Instead, it emphasizes filtered interpretation, concept-sensitive evaluation, and the possibility that multiple annotation layers may be analytically meaningful. This suggests an evaluation philosophy in which construct validity and annotation provenance condition the interpretation of model performance rather than serving merely as dataset metadata (Chatelain et al., 6 May 2026).

5. Role in LLM assessment and social-science methodology

BenCSSmark is not primarily a benchmark-results paper. It does not present a large-scale leaderboard or a battery of model scores across the 27 datasets. Its empirical claims are instead qualitative and synthetic: it situates the benchmark within a broader literature suggesting that state-of-the-art models can still struggle to satisfy social science needs, that performance varies sharply across superficially similar tasks, and that models tend to fare better on standard NLP-like classifications than on tasks requiring contextual interpretation, cultural sensitivity, or temporal robustness (Chatelain et al., 6 May 2026).

Within that framing, BenCSSmark functions as a stress test for LLMs. The benchmark is designed to expose weaknesses that are masked by evaluation regimes prioritizing precision, factual recall, or narrow reasoning. Social-scientific tasks are presented as demanding a different profile of competence: interpretation in context, management of ambiguity, and handling of conflicting perspectives. The benchmark therefore has a dual significance. For AI evaluation, it argues for a more pluralistic and validity-oriented assessment ecosystem. For computational social science, it argues that datasets already produced in substantive research can shape the direction of model development rather than being treated as downstream applications (Chatelain et al., 6 May 2026).

The benchmark also intervenes in a common misconception about benchmark value. It does not claim that aggregation alone produces better science, nor does it propose that social science tasks should simply be appended to existing leaderboards without methodological adaptation. Instead, it argues that the omission of these tasks changes what models are optimized to do. The benchmark’s importance thus lies less in reporting a definitive ranking and more in redefining what should count as meaningful evaluation for LLMs deployed in research and social settings (Chatelain et al., 6 May 2026).

6. Limitations, risks of benchmarkization, and future directions

The paper is explicit that BenCSSmark is only a first step. Its present limitations are substantial: it is limited in scale and scope, includes only French data, is entirely textual, and is restricted to relatively conventional media forms such as newspapers, speeches, and social media. The benchmark is also not fully open at present, despite the original intention to build an open repository, because some datasets are proprietary—especially newspaper archives—and some may contain personally identifiable information. These constraints matter because they limit both reproducibility and cross-linguistic generalization (Chatelain et al., 6 May 2026).

A further limitation is benchmark obsolescence. The paper warns that benchmark contamination can arise as LLM developers ingest benchmark-like data into training corpora, thereby undermining evaluation validity. This is presented as a general benchmark problem rather than a pathology unique to BenCSSmark. The benchmark’s current French-only design likewise means that claims about socially grounded evaluation remain bounded by a specific linguistic and institutional context. A plausible implication is that BenCSSmark should be understood as a methodological template as much as a fixed dataset collection (Chatelain et al., 6 May 2026).

The paper also addresses the risks of benchmarkization itself. It warns that the benchmark could reproduce familiar pathologies: over-optimizing for the metric rather than the underlying phenomenon, Goodhart-style overfitting, narrowing research questions to what is easily measurable, and privileging technical improvement over conceptual or theoretical innovation. The authors treat this as especially serious in social science, where pluralism and interpretive nuance are constitutive rather than incidental. As a mitigation, they recommend including multiple labels such as time period, discipline, cultural area, and task type; supporting filtered and context-sensitive evaluation rather than one collapsed score; and continuously expanding the benchmark with new datasets and viewpoints (Chatelain et al., 6 May 2026).

Future work is framed in expansionary rather than merely scaling terms. The benchmark is intended to grow as more social scientists are engaged, and the paper mentions possible inclusion of semantic text similarity, clustering, other task forms relevant to social science, and additional modalities and languages. In that sense, BenCSSmark is best understood as an evolving benchmark program: a French-language starting point for integrating construct-valid, context-rich, and disagreement-aware social science tasks into LLM evaluation, while keeping open the methodological question of how such tasks should be compared without erasing the interpretive structure they are meant to capture (Chatelain et al., 6 May 2026).