BenCSSmark: Making the Social Sciences Count in LLM Research

Abstract: This position paper argues that the under-representation of social science tasks in contemporary LLM benchmarks limits advances in both LLM evaluation and social scientific inquiry. Benchmarks -- standardized tools for assessing computational systems -- are pivotal in the development of AI, including LLMs. Benchmarks do more than measure progress -- they actively structure it, shaping reputations, research agendas, and commercial outcomes. Despite this central role, the social sciences are largely absent from mainstream evaluation frameworks, even though scholars in these fields generate dozens of rigorously annotated, context-sensitive datasets each year. Integrating this work into benchmark design could significantly improve the generalization and robustness of AI models. In turn, models trained on social scientific tasks would likely yield better performance on classic and contemporary tasks in disciplines as diverse as history, sociology, political science or economics. This is all the more pressing as these disciplines are quickly turning to LLMs for assistance. To address this gap, we introduce BenCSSmark, a benchmark composed of datasets annotated by computational social scientists. By integrating social scientific perspectives into benchmarking, BenCSSmark seeks to promote more robust, transparent, and socially relevant AI systems and to foster efficient collaboration.

• The paper presents a novel benchmark that integrates social science tasks into LLM evaluation to expose limits in model reasoning and contextual understanding.
• It employs disciplinary-specific datasets and perspectivist annotations to capture cultural, temporal, and contextual nuances in empirical social research.
• The framework encourages continuous refinement, multi-modal expansion, and methodological innovation to bridge gaps between AI and social science communities.

BenCSSmark: Integrating Social Science Tasks into LLM Benchmarking

Motivations for Social Science-Oriented LLM Benchmarks

Standardized benchmarks have played a determinative role in shaping research directions, perceived progress, and even the reputational dynamics of both models and research communities within AI, especially in the context of LLMs. Existing benchmarks predominantly assess linguistic, reasoning, or general knowledge skills, with some expansion into vertical professional domains such as law or medicine. However, there is a striking absence of benchmark coverage for the social sciences, despite this domain's longstanding tradition of producing annotated, expert-driven datasets. This underrepresentation has direct implications: LLMs are insufficiently evaluated for tasks that are central to sociology, political science, history, and economics—domains that increasingly depend on such models for analysis and research augmentation.

The lack of social science tasks in mainstream benchmarks is not merely a gap in coverage, but shapes both innovation and the breadth of evaluations deemed consequential for high-impact models. This omission impedes the generalization and robustness of LLMs and constrains the advancement of computational social science. It also separates the interests and practices of social science and NLP communities, further restricting the flow of methodological innovations between them.

Related Work and Limitations in Current Benchmarking

While projects such as GLUE, SuperGLUE, MMLU, and broad multi-task efforts like BIG-bench dominate the evaluation landscape, social science–specific tasks remain marginal or absent. Even initiatives designed to extend coverage towards human-centered or “social knowledge” domains (e.g., SocKET (Choi et al., 2023), HSSBench (Kang et al., 4 Jun 2025)) primarily address dialogical or multimodal reasoning and do not directly operationalize constructs or inquiries central to empirical social science research.

Some recent efforts, such as the aggregation of 480 datasets for "social intelligence" (Li et al., 2024), approach the periphery of social scientific evaluation but generally focus on conversational tasks or alignment, not on the breadth of empirical constructs in social inquiry. Moreover, perspectivist approaches in annotation—preserving annotator disagreement as a reflection of underlying social reality—are gaining traction but have yet to be systematically operationalized in mainstream LLM benchmarks [1719--1746].

Thus, current LLM benchmarks do not sufficiently characterize the nuances, context-dependence, disciplinary specificity, and diversity of viewpoints constitutive of social science tasks. This leads to several structural issues:

• Dataset visibility and standardization: Social science data are fragmented, inadequately cataloged, and infrequently shared in standardized formats, restricting reproducibility and evaluation.
• Task operationalization: Many social science tasks are context-specific and adhere to domain-dependent frameworks, impeding their assimilation into generic evaluation pipelines.
• Outcome metrics and subjectivity: Evaluations often collapse complexity into single metrics, potentially obscuring interpretive ambiguity, annotator perspectives, and contestation over definitional boundaries.

The BenCSSmark Initiative: Scope and Principles

BenCSSmark, built within the Pantagruel project, directly addresses the aforementioned limitations by systematically curating and structuring datasets originating in computational social science. Its construction is premised on three principles:

1. Disciplinary and conceptual plurality: Tasks are sourced from domains (e.g., sociology, political science) and operationalized in ways aligned with their research traditions, ensuring that standard NLP categories (frame detection, text classification, etc.) capture the specificities relevant to each sub-discipline.
2. Contextual, temporal, and socio-cultural sensitivity: The benchmark prioritizes diverse data genres, time periods (e.g., late 20th century onwards, including archival texts), and modalities (written, spoken), thereby stress-testing the robustness of LLMs to linguistic, cultural, and historical shifts.
3. Preservation of subjectivity and annotation diversity: For self-annotated subsets, both adjudicated ground truths and individual annotator judgments are retained, explicitly supporting perspectivist evaluation protocols and enabling nuanced analyses of task ambiguity, inter-annotator variance, and the epistemic situatedness of labels.

Datasets are categorized dually: by mainstream NLP task taxonomy and by underlying social science concepts, enriching both selection and comparative assessment. At the time of writing, the benchmark includes 27 French-language datasets across genres such as news, broadcast, social media, political discourse, and academic texts. The design supports a range of task types—binary, multiclass, multilabel, span detection, and coreference—with plans to expand towards text similarity and clustering.

Theoretical and Empirical Implications

From a theoretical standpoint, BenCSSmark asserts that social science tasks offer a rigorous stress test for LLM capabilities beyond the current frontiers of precision and factual recall. Evaluating models on contextually rich, ambiguity-laden, and culturally contingent tasks exposes deficits in pragmatic reasoning, semantic flexibility, and the handling of multiple valid perspectives—qualities central to human language understanding and social analysis.

This approach also opens new avenues for metric development: integrating perspectivist annotations necessitates moving beyond accuracy-centric evaluations toward metrics that accommodate ambiguity, subjectivity, and disagreement [1719--1746, 341(1764.34454)23]. BenCSSmark thus provides a real-world foundation for operationalizing and advancing the evaluation of model interpretive robustness and socio-cognitive flexibility.

Pragmatically, BenCSSmark demonstrates that existing LLMs, even when strong on standard tasks, may underperform or behave unpredictably on social science datasets, given their distinct annotation schemes, granularity, and cultural specificity. As such, the benchmark is well positioned to both diagnose current model limitations and to catalyze the development of models more robust to real-world complexity.

Risks, Limitations, and Future Directions

Several limitations constrain the current instantiation of BenCSSmark:

• Scale and scope: The dataset portfolio is currently limited to the French language and to textual and audio modalities; key genres and languages remain uncovered.
• Accessibility: A significant portion of the data is proprietary or contains sensitive information, which restricts open dissemination and hinders reproducibility.
• Benchmark ossification: As with other benchmarks, continuous model improvement and potential incorporation of benchmark data into pre-training corpora may rapidly erode the diagnostic utility of fixed datasets.
• Risk of "benchmarkisation": There is a danger that over-standardization will replicate the well-known pitfalls of mainstream benchmarking—namely, the telescoping of rich, pluralistic phenomena into narrow evaluation criteria, and the marginalization of alternative epistemic perspectives [100336, (Eriksson et al., 10 Feb 2025)].

To mitigate these risks, the architecture of BenCSSmark encourages multidimensional filtering (e.g., time, discipline, annotator), continual expansion, and foregrounding of epistemic diversity in both evaluation protocols and dataset curation. Further, the project is positioned as an open invitation for expansion and collaboration across communities, rather than as a final authoritative standard.

Implications for the Future of AI and Social Science Integration

BenCSSmark marks an inflection in the integration of LLM research and social scientific inquiry. By bridging disciplinary divides, it addresses the critical need for domain-sensitive, context-rich, and epistemically pluralistic evaluation frameworks. For NLP, this means broader, more realistic assessment of generative models; for social scientists, it provides the infrastructure to shape the development and calibration of AI tools suited to their analytic needs.

Going forward, communal efforts are needed to expand coverage (multi-language, new modalities), develop privacy-preserving sharing mechanisms, and continuously refresh and diversify task suites. Theoretical innovation in annotation, metric construction, and operationalization of subjectivity will be essential. Ultimately, BenCSSmark sets an agenda for re-centering LLM evaluation around the diverse, contested, and context-dependent realities encountered in real social scientific practice.

Conclusion

BenCSSmark delivers a theoretically motivated and operationally robust framework for embedding social science tasks into the core of LLM evaluation. By foregrounding disciplinary expertise, annotation diversity, and contextual nuance, it provides new tools for both AI and social science communities to interrogate and improve the interpretive and analytical capacities of LLMs. Its evolution will inform best practices in both benchmark design and collaborative research at the intersection of computational and social inquiry (2605.04886).