TAZ2024full Corpus: German News Dataset

• TAZ2024full corpus is a large-scale, diachronic collection of German newspaper articles from taz spanning 1980 to 2024, designed with full-text JSON records.
• The dataset supports actor-level analysis using methodologies such as NER, coreference resolution, sentiment modeling, and quotation attribution to examine gender bias.
• It incorporates innovative exclusion and balancing techniques to mitigate structural gender imbalances, enabling reproducible fairness-aware NLP research.

Searching arXiv for the specified corpus and follow-up paper to ground the article in the cited sources. The taz2024full corpus is a large-scale, diachronic corpus of German newspaper articles drawn from taz – die tageszeitung and spanning 1980 to 2024. It was introduced as a full-text, open, non-commercial resource for German-language NLP and computational social science, with a particular emphasis on studying gender bias and discrimination in journalistic discourse (Urchs et al., 3 Jun 2025). The corpus comprises over 1.8 million articles and is structurally designed as article-level JSON records with metadata and full text, while downstream analyses operate on actor-centric representations derived from named entity recognition, coreference resolution, sentiment modeling, quotation attribution, and syntactic parsing (Urchs et al., 3 Jun 2025, Urchs et al., 7 Aug 2025). As both a corpus resource and a methodological substrate, taz2024full has been used to analyze long-term shifts in gender representation and to construct filtered, more balanced corpus variants for fairness-aware NLP research (Urchs et al., 7 Aug 2025).

1. Corpus identity, provenance, and scope

taz2024full is a corpus of German newspaper articles only, collected from the Berlin-based, progressive or left-leaning daily taz – die tageszeitung (Urchs et al., 3 Jun 2025). It was built to provide a longitudinal, full-text resource for German-language NLP and computational social science and to support systematic studies of gender bias and discrimination over more than four decades (Urchs et al., 3 Jun 2025). The corpus is explicitly positioned against German-language resources that are either not downloadable as full text or are limited to keyword querying and snippet retrieval, which constrains large-scale reproducible analysis (Urchs et al., 3 Jun 2025).

The temporal coverage is 1980–2024, with collection performed by web crawling of taz’s public website between August and November 2024 (Urchs et al., 3 Jun 2025). Only publicly accessible articles were included; paywalled material is absent (Urchs et al., 3 Jun 2025). Early years are sparse, with four articles or fewer per year before 1986, article counts grow steadily from 1980, and the yearly maximum is reported for 2004 with 73,002 articles (Urchs et al., 3 Jun 2025). The corpus also contains a noticeable dip in 1991, which the authors state they cannot explain (Urchs et al., 3 Jun 2025).

The reported size is 1,834,026 articles in the main statistics section, although one subsection mentions 1,834,370 (Urchs et al., 3 Jun 2025). In the actor-based discrimination study, the authors refer to 1,834,018 articles with clearly gender-coded actors for the fairness analysis pipeline (Urchs et al., 7 Aug 2025). The corpus vocabulary is reported as 6,944,197 unique tokens, tokenized with SoMaJo (Urchs et al., 3 Jun 2025). The resource is described as, to the authors’ knowledge, the largest publicly available corpus of German newspaper articles meeting three criteria simultaneously: full-article downloadability, coverage across decades from a single outlet, and academic usability under non-commercial constraints (Urchs et al., 3 Jun 2025).

The corpus is a single-source resource, and its editorial provenance matters analytically. The newspaper is explicitly left-leaning, which likely affects topic distribution and editorial positioning (Urchs et al., 7 Aug 2025). This suggests that taz2024full is highly suitable for within-source diachronic analysis, but less suitable as a basis for claims about German media as a whole.

2. Data model, metadata, and distribution format

The corpus is distributed as JSON on Zenodo, with each article stored under a unique ID key and structured into metadata and text fields (Urchs et al., 3 Jun 2025). The metadata include published_on, contains_actors, crawled_on, language, type, author, keywords, and token_count, while the text object includes title, teaser, and text (Urchs et al., 3 Jun 2025). The language is always "de" and the type is always "article" (Urchs et al., 3 Jun 2025).

At the document level, segmentation is strictly article-based. The released corpus does not include sentence boundaries, paragraph segmentation, lemmas, POS tags, morphological annotation, NER labels, coreference chains, or sentiment labels (Urchs et al., 3 Jun 2025). Linguistic processing is deliberately separated from the corpus release and performed in downstream analysis pipelines (Urchs et al., 3 Jun 2025). This division is important: taz2024full is a raw-text-plus-metadata corpus, not a pre-annotated benchmark.

The metadata are sufficient for longitudinal and filtered analyses, but they are not exhaustive. The authors explicitly note that section labels such as politics, culture, or regional desk were not available in the crawled metadata and therefore could not be incorporated into the dataset (Urchs et al., 3 Jun 2025). At minimum, article ID and publication year are used downstream for aggregation and filtered corpus reconstruction (Urchs et al., 7 Aug 2025).

The following table summarizes the basic released structure.

Component	Fields	Notes
meta_data	published_on, contains_actors, crawled_on, language, type, author, keywords, token_count	Raw metadata and derived actor-presence flag
text	title, teaser, text	Headline, optional lede, and full article body
ID	unique article key	Used in filtering and reconstruction workflows

The contains_actors flag is derived from the actor detection pipeline rather than manual annotation (Urchs et al., 3 Jun 2025). This implies that the corpus’s effective analytical coverage depends partly on the properties of the extraction system. A plausible implication is that alternative NER or coreference systems may produce different estimates of person-entity prevalence and thus different downstream fairness diagnostics.

3. Construction, preprocessing, and corpus statistics

Data collection was performed via web crawling of https://taz.de/ during August–November 2024, restricted to content that was publicly accessible online (Urchs et al., 3 Jun 2025). The publisher explicitly granted permission for academic research use and release of the dataset, while commercial use is strictly prohibited (Urchs et al., 3 Jun 2025). Articles with fewer than 3 tokens according to SoMaJo were excluded to remove fragments and extremely short notices (Urchs et al., 3 Jun 2025).

The released corpus is only lightly processed. The authors state that there is no heavy linguistic preprocessing in the distribution itself (Urchs et al., 3 Jun 2025). For descriptive statistics and later analysis, they use SoMaJo for tokenization and sentence splitting (Urchs et al., 3 Jun 2025). Reported corpus statistics include the following: token length in characters has min 1, max 2,238, mean 5.15, median 4; sentence length in tokens has min 1, max 18,872, mean 20.07, median 17; article length in tokens has min 3, max 26,855, mean 396.89, median 276; and article length in sentences has min 1, max 1,027, mean 19.77, median 13 (Urchs et al., 3 Jun 2025). The extreme maxima are interpreted as occasional tokenization artifacts, whereas the mean values are considered reasonable for German news text (Urchs et al., 3 Jun 2025).

A notable descriptive statistic is that 83% of all articles contain person entities, according to the authors’ actor detection (Urchs et al., 3 Jun 2025). This high proportion is consequential because the principal research use case of the corpus is actor-level gender analysis. In the fairness study, output is stored as actor-level knowledge bases per article and yearly aggregate reports in plain text (Urchs et al., 7 Aug 2025). For each actor, the stored information includes gender pronoun group, surface forms, mentions, syntactic roles, quote occurrences, sentiment, and PMI contexts (Urchs et al., 7 Aug 2025).

The corpus is therefore best understood as having two analytical layers. The first is the released dataset of raw texts and metadata (Urchs et al., 3 Jun 2025). The second is an inferred annotation layer generated by the actor-based pipeline, which is not embedded in the corpus release but underpins gender representation analysis and filtering (Urchs et al., 7 Aug 2025).

4. Actor-based representation and discourse-aware analysis

The principal methodological innovation associated with taz2024full is an actor-level, discourse-aware pipeline for analyzing gender discrimination in large text corpora (Urchs et al., 7 Aug 2025). Actors are entities that participate in discourse and can be gendered. They are extracted primarily through Named Entity Recognition for persons plus coreference resolution linking names, titles, generic references, and pronouns (Urchs et al., 7 Aug 2025). Generic person terms such as “Mutter,” “Vater,” “Frau,” and “Mann” are also incorporated in the earlier corpus-analysis pipeline (Urchs et al., 3 Jun 2025).

Each actor record stores all references to that actor, including names and pronouns, and the actor is assigned a gender pronoun group of she/her or he/him (Urchs et al., 7 Aug 2025). The framework conceptually acknowledges non-binary gender, but the implemented analysis is binary because of German linguistic constraints and the rarity of neo-pronouns in the dataset; the corpus paper reports only 5 texts containing such pronouns (Urchs et al., 3 Jun 2025). Only actors with at least one resolved pronoun are considered in the pronoun-driven analysis (Urchs et al., 3 Jun 2025).

The analysis is described as actor-level because metrics are computed for each actor and then aggregated by gender and by article or year, rather than being derived only at the document level (Urchs et al., 7 Aug 2025). It is described as discourse-aware because it operationalizes structures emphasized in systemic functional linguistics and critical discourse analysis: nomination, predication, syntactic roles, and quotation (Urchs et al., 7 Aug 2025). The pipeline therefore goes beyond simple token-frequency imbalance by tracking who is named versus pronominalized, who is syntactic subject versus object, who speaks directly versus indirectly, and how actors are framed sentimentally and lexically (Urchs et al., 7 Aug 2025).

The processing stack includes NER and coreference resolution, sentence segmentation, dependency parsing for syntactic roles such as nsubj and obj, quote detection and attribution, a German BERT-based sentiment classifier trained on news, lexicon-based gender-coded word counts derived from Gaucher et al., and heuristics for gender-neutral forms and generic masculine usage (Urchs et al., 7 Aug 2025). In the corpus paper, the sentiment component is specified as "oliverguhr/german-sentiment-bert" integrated via the transformers pipeline (Urchs et al., 3 Jun 2025).

Several metrics are formalized at the group level. For a gender group $g$ and corpus unit $u$, counts include actors $A_{g,u}$, mentions $M_{g,u}$, named mentions $M^{\text{name}}_{g,u}$, pronoun mentions $M^{\text{pro}}_{g,u}$, subject roles $S_{g,u}$, object roles $O_{g,u}$, direct quotes $Q^{\text{dir}}_{g,u}$, indirect quotes $Q^{\text{ind}}_{g,u}$, and counts of feminine- and masculine-coded words (Urchs et al., 7 Aug 2025). Ratios include share of mentions, share of actors, subject-role agency, direct versus indirect quotation proportions, and name-versus-pronoun usage (Urchs et al., 7 Aug 2025). Sentiment is computed at the actor level as

$u$0

and aggregated by gender group within a unit (Urchs et al., 7 Aug 2025).

In addition, the corpus and fairness papers use PMI to analyze associations between actors and contextual descriptors. The corpus paper gives the explicit formulation

$u$1

for actor–adjective association (Urchs et al., 3 Jun 2025), while the fairness paper describes PMI-based framing over adjectives, nouns, and verbs associated with actors of each gender (Urchs et al., 7 Aug 2025).

5. Empirical gender patterns in taz2024full

The corpus has been used to study longitudinal gender representation from 1980 to 2024, and the findings consistently indicate male overrepresentation, albeit with gradual movement toward greater balance in recent years (Urchs et al., 3 Jun 2025, Urchs et al., 7 Aug 2025). In the unfiltered corpus, male actors dominate both mentions and actor counts overall (Urchs et al., 7 Aug 2025). The fairness paper reports that article-level gender ratios are often 0% or 100% female share, meaning that many articles are effectively single-gender in terms of coded actors (Urchs et al., 7 Aug 2025).

The diachronic trajectory is not static. The authors report a crossing around 2018, where female share becomes comparable to or surpasses male share in some dimensions (Urchs et al., 7 Aug 2025). The corpus paper likewise reports that from the 2010s onward the share of women actors increases and approaches parity in recent years, although men still receive more mentions, implying greater textual space even when actor counts become similar (Urchs et al., 3 Jun 2025).

A concrete annual example is provided for 2023 in the fairness study. There are 10,019 texts, all with actors; pronoun counts are 6,892 she/her and 9,194 he/him; total mentions are 91,639, split into 35,595 she/her and 56,044 he/him; named mentions are 22,544 she/her versus 36,047 he/him; pronoun mentions are 13,051 she/her versus 19,997 he/him (Urchs et al., 7 Aug 2025). Subject roles are 18,625 she/her versus 30,303 he/him, corresponding to 38.1% and 61.9% of known roles; object roles are 1,119 she/her versus 1,540 he/him, corresponding to 42.1% and 57.9% (Urchs et al., 7 Aug 2025). Direct quotes are 6,501 she/her versus 10,588 he/him, and indirect quotes are 2,529 she/her versus 4,215 he/him (Urchs et al., 7 Aug 2025).

The corpus paper gives a different 2023 aggregate from its own yearly reporting configuration: 26,357 texts, 17,161 actors, pronoun distribution 9,088 he/him versus 8,073 she/her actors, total mentions 109,634, average sentiment −0.03 overall, −0.03 for she/her, and −0.02 for he/him, with mean generic masculine indicator 0.31 and gender-neutral majority indicator 0.00 (Urchs et al., 3 Jun 2025). The discrepancy reflects different reporting scopes and pipeline stages rather than a contradiction explicitly resolved in the available descriptions.

The principal discourse-level asymmetries are threefold. First, men are more often in subject positions, while women are comparatively more often objects, which the authors interpret as greater narrative agency for male actors (Urchs et al., 7 Aug 2025). Second, male actors are more frequently quoted in direct speech, while female actors appear more often in indirect quotes, and direct quotation is interpreted as conferring greater authority and visibility (Urchs et al., 7 Aug 2025). Third, sentiment is close to neutral for both genders, but the average sentiment toward female-coded actors is reported as more negative in almost all years (Urchs et al., 7 Aug 2025). The corpus paper similarly states that sentiment toward women is consistently slightly more negative over the full 44-year period, though the differences are small (Urchs et al., 3 Jun 2025).

At the lexical framing level, overt gender-coded language appears limited. The corpus paper reports that top PMI adjectives associated with women and men are relatively stable over time and do not show strong gender differentiation, and that feminine-coded and masculine-coded words are rare overall (Urchs et al., 3 Jun 2025). The fairness paper, however, reports that PMI lists for 2023 associate women more with terms such as “Frau, Mutter, Tochter, Kinder” and men more with “Präsident, Mann, Sohn”, indicating persistent thematic role differentiation (Urchs et al., 7 Aug 2025). This suggests that absence of strong overt lexical coding does not eliminate subtler framing asymmetries.

6. Exclusion-based balancing and fairer corpus construction

A major extension of the taz2024full research program is the multi-stage, exclusion-based balancing framework for constructing less discriminatory corpus variants (Urchs et al., 7 Aug 2025). The pipeline consists of four stages: actor-based discrimination analysis of the full corpus; reuse of article-level results for exclusion; corpus-level balancing to bring global gender ratios into a user-defined range; and final reconstruction of the filtered corpus by removing excluded article IDs from the original JSON files (Urchs et al., 7 Aug 2025).

Text-level exclusion is based on four heuristic discrimination scores for an article $u$2: sentiment disparity $u$3, grammatical role asymmetry $u$4, quote attribution imbalance $u$5, and naming versus pronoun imbalance $u$6 (Urchs et al., 7 Aug 2025). All ratios are Laplace-smoothed to mitigate instability when counts are low (Urchs et al., 7 Aug 2025). The default thresholds are $u$7 for sentiment and $u$8 for role, quote, and naming imbalances (Urchs et al., 7 Aug 2025). The default decision rule excludes an article if two out of four criteria exceed their thresholds (Urchs et al., 7 Aug 2025).

After text-level filtering, a second stage applies corpus-level balancing to actor and mention ratios. The authors define

$u$9

and target a default equilibrium interval of $A_{g,u}$0, allowing up to $A_{g,u}$1 deviation from parity (Urchs et al., 7 Aug 2025). Articles are iteratively excluded based on their marginal contribution to global imbalance until both ratios fall within the target range (Urchs et al., 7 Aug 2025).

The reported scale of filtering is substantial. Starting from 1,834,018 articles with identifiable gendered actors, 279,772 are removed at the text level and an additional 17,815 at the corpus-balancing stage, for a total of 297,587 removed articles and a final corpus of approximately 1.54 million articles (Urchs et al., 7 Aug 2025). After filtering, article-level gender ratios become less polarized, and after full balancing the distribution becomes more centered around parity (Urchs et al., 7 Aug 2025). The authors emphasize referential parity in the final corpus, meaning that male and female actors appear in comparable proportions across both mentions and actor counts (Urchs et al., 7 Aug 2025).

Discourse-level asymmetries are reduced but not eliminated. The subject-role gap decreases from approximately 30 percentage points to approximately 5, quotation imbalance is reduced, and women appear in direct speech more frequently than before (Urchs et al., 7 Aug 2025). Sentiment, however, remains slightly more negative for women, even if the remaining gap stays below the exclusion threshold of $A_{g,u}$2 (Urchs et al., 7 Aug 2025). The authors interpret this as evidence that balancing improves structural asymmetries but does not fully remove evaluative bias (Urchs et al., 7 Aug 2025).

7. Accessibility, methodological limitations, and research significance

The corpus is hosted on Zenodo with DOI https://doi.org/10.5281/zenodo.15480855, and the analysis pipeline is released on GitHub as https://github.com/Ognatai/corpus_pipeline (Urchs et al., 3 Jun 2025). The follow-up balancing work releases tools and reports at https://github.com/Ognatai/corpus_balancing, including the extended actor-level pipeline, the two-stage exclusion framework, an interactive interface for setting thresholds and equilibrium ranges, and scripts for report generation and corpus reconstruction (Urchs et al., 7 Aug 2025). The papers state that the pipeline and reports are released, but they do not explicitly state that a fully processed or filtered version of taz2024full is released (Urchs et al., 7 Aug 2025).

Several limitations are explicit. First, taz2024full is a single-newspaper corpus from a left-leaning outlet and is not representative of the full German media spectrum (Urchs et al., 3 Jun 2025). Second, only publicly available content is included, so later years may be skewed by the absence of paywalled material, especially after around 2007 (Urchs et al., 3 Jun 2025). Third, early years are sparse, and the authors caution against over-interpreting strong fluctuations in yearly percentages from the 1980s (Urchs et al., 7 Aug 2025).

Methodologically, the actor pipeline depends heavily on NER, coreference resolution, and pronoun identification (Urchs et al., 7 Aug 2025). German coreference is specifically acknowledged as weak, and no gold-standard evaluation, precision, recall, or F1 is reported for the full actor-level German pipeline (Urchs et al., 3 Jun 2025). Articles without clearly identifiable actors or gender cues are excluded from discrimination analysis, producing incomplete coverage (Urchs et al., 7 Aug 2025). The gender model is also restricted to a binary approximation, despite the conceptual acknowledgment that gender is socially constructed and non-binary (Urchs et al., 3 Jun 2025, Urchs et al., 7 Aug 2025).

The fairness framework is equally bounded by its chosen metrics. It relies on surface-level indicators such as syntactic roles, sentiment scores, quote types, and counts of gendered terms (Urchs et al., 7 Aug 2025). It cannot capture irony, sarcasm, omission, topic-selection bias, or higher-level narrative structure (Urchs et al., 7 Aug 2025). Threshold choices are normative and materially affect outcomes; the authors mitigate this with interactive tools, logging, and visual diagnostics, but they do not claim that the procedure is fully objective (Urchs et al., 7 Aug 2025).

Within German NLP and computational social science, taz2024full is significant because it combines full-text accessibility, diachronic depth, and a documented bias profile in a way that supports both descriptive and intervention-oriented research (Urchs et al., 3 Jun 2025). It enables studies of language change, media discourse, agenda setting, sentiment trajectories, and actor-centric representation over four decades (Urchs et al., 3 Jun 2025). In the LLM context, the fairness paper argues that structural gender imbalances in training data shape the representational landscape of model outputs, and positions taz2024full as a basis for constructing fairer training subsets with reduced asymmetries in counts, agency, and quotation (Urchs et al., 7 Aug 2025). At the same time, the authors warn that exclusion-based balancing reduces corpus size and possibly diversity, may distort historical reality, and does not by itself yield “fully fair” training data (Urchs et al., 7 Aug 2025).

Taken together, taz2024full functions both as a large German news corpus and as a methodological platform for actor-centered fairness auditing. Its core contribution lies not only in scale, but in the combination of article-level accessibility, longitudinal scope, and discourse-aware analytical infrastructure for studying how representation, visibility, and framing evolve across decades of German journalism (Urchs et al., 3 Jun 2025, Urchs et al., 7 Aug 2025).