---
title: 'TAZ2024full Corpus: German News Dataset'
url: https://www.emergentmind.com/topics/taz2024full-corpus
type: topic
---

# TAZ2024full Corpus: German News Dataset

Searching arXiv for the specified corpus and follow-up paper to ground the article in the cited sources.
The **taz2024full corpus** is a large-scale, diachronic corpus of German newspaper articles drawn from *taz – die tageszeitung* and spanning 1980 to 2024. It was introduced as a full-text, open, non-commercial resource for German-language NLP and computational social science, with a particular emphasis on studying gender bias and discrimination in journalistic discourse [2506.05388]. The corpus comprises over 1.8 million articles and is structurally designed as article-level JSON records with metadata and full text, while downstream analyses operate on actor-centric representations derived from named entity recognition, coreference resolution, sentiment modeling, quotation attribution, and syntactic parsing [2506.05388; 2508.13169]. As both a corpus resource and a methodological substrate, taz2024full has been used to analyze long-term shifts in gender representation and to construct filtered, more balanced corpus variants for fairness-aware NLP research [2508.13169].

## 1. Corpus identity, provenance, and scope

taz2024full is a corpus of **German newspaper articles only**, collected from the Berlin-based, progressive or left-leaning daily *taz – die tageszeitung* [2506.05388]. It was built to provide a **longitudinal, full-text resource** for German-language NLP and computational social science and to support systematic studies of **gender bias and discrimination** over more than four decades [2506.05388]. The corpus is explicitly positioned against German-language resources that are either not downloadable as full text or are limited to keyword querying and snippet retrieval, which constrains large-scale reproducible analysis [2506.05388].

The temporal coverage is **1980–2024**, with collection performed by web crawling of taz’s public website between **August and November 2024** [2506.05388]. Only **publicly accessible articles** were included; paywalled material is absent [2506.05388]. Early years are sparse, with **four articles or fewer per year** before 1986, article counts grow steadily from 1980, and the yearly maximum is reported for **2004 with 73,002 articles** [2506.05388]. The corpus also contains a noticeable dip in **1991**, which the authors state they cannot explain [2506.05388].

The reported size is **1,834,026 articles** in the main statistics section, although one subsection mentions **1,834,370** [2506.05388]. In the actor-based discrimination study, the authors refer to **1,834,018 articles** with clearly gender-coded actors for the fairness analysis pipeline [2508.13169]. The corpus vocabulary is reported as **6,944,197 unique tokens**, tokenized with SoMaJo [2506.05388]. The resource is described as, to the authors’ knowledge, **the largest publicly available corpus of German newspaper articles** meeting three criteria simultaneously: full-article downloadability, coverage across decades from a single outlet, and academic usability under non-commercial constraints [2506.05388].

The corpus is a **single-source** resource, and its editorial provenance matters analytically. The newspaper is explicitly **left-leaning**, which likely affects topic distribution and editorial positioning [2508.13169]. This suggests that taz2024full is highly suitable for within-source diachronic analysis, but less suitable as a basis for claims about German media as a whole.

## 2. Data model, metadata, and distribution format

The corpus is distributed as **JSON** on Zenodo, with each article stored under a unique ID key and structured into metadata and text fields [2506.05388]. The metadata include `published_on`, `contains_actors`, `crawled_on`, `language`, `type`, `author`, `keywords`, and `token_count`, while the text object includes `title`, `teaser`, and `text` [2506.05388]. The language is always `"de"` and the type is always `"article"` [2506.05388].

At the document level, segmentation is strictly **article-based**. The released corpus does **not** include sentence boundaries, paragraph segmentation, lemmas, POS tags, morphological annotation, NER labels, coreference chains, or sentiment labels [2506.05388]. Linguistic processing is deliberately separated from the corpus release and performed in downstream analysis pipelines [2506.05388]. This division is important: taz2024full is a **raw-text-plus-metadata corpus**, not a pre-annotated benchmark.

The metadata are sufficient for longitudinal and filtered analyses, but they are not exhaustive. The authors explicitly note that **section labels** such as politics, culture, or regional desk were not available in the crawled metadata and therefore could not be incorporated into the dataset [2506.05388]. At minimum, article ID and publication year are used downstream for aggregation and filtered corpus reconstruction [2508.13169].

The following table summarizes the basic released structure.

| Component | Fields | Notes |
|---|---|---|
| `meta_data` | `published_on`, `contains_actors`, `crawled_on`, `language`, `type`, `author`, `keywords`, `token_count` | Raw metadata and derived actor-presence flag |
| `text` | `title`, `teaser`, `text` | Headline, optional lede, and full article body |
| `ID` | unique article key | Used in filtering and reconstruction workflows |

The `contains_actors` flag is derived from the actor detection pipeline rather than manual annotation [2506.05388]. This implies that the corpus’s effective analytical coverage depends partly on the properties of the extraction system. A plausible implication is that alternative NER or coreference systems may produce different estimates of person-entity prevalence and thus different downstream fairness diagnostics.

## 3. Construction, preprocessing, and corpus statistics

Data collection was performed via **web crawling** of `https://taz.de/` during **August–November 2024**, restricted to content that was publicly accessible online [2506.05388]. The publisher explicitly granted permission for academic research use and release of the dataset, while **commercial use is strictly prohibited** [2506.05388]. Articles with fewer than **3 tokens** according to SoMaJo were excluded to remove fragments and extremely short notices [2506.05388].

The released corpus is only lightly processed. The authors state that there is **no heavy linguistic preprocessing** in the distribution itself [2506.05388]. For descriptive statistics and later analysis, they use **SoMaJo** for tokenization and sentence splitting [2506.05388]. Reported corpus statistics include the following: token length in characters has **min 1, max 2,238, mean 5.15, median 4**; sentence length in tokens has **min 1, max 18,872, mean 20.07, median 17**; article length in tokens has **min 3, max 26,855, mean 396.89, median 276**; and article length in sentences has **min 1, max 1,027, mean 19.77, median 13** [2506.05388]. The extreme maxima are interpreted as occasional tokenization artifacts, whereas the mean values are considered reasonable for German news text [2506.05388].

A notable descriptive statistic is that **83% of all articles contain person entities**, according to the authors’ actor detection [2506.05388]. This high proportion is consequential because the principal research use case of the corpus is actor-level gender analysis. In the fairness study, output is stored as **actor-level knowledge bases per article** and yearly aggregate reports in plain text [2508.13169]. For each actor, the stored information includes gender pronoun group, surface forms, mentions, syntactic roles, quote occurrences, sentiment, and PMI contexts [2508.13169].

The corpus is therefore best understood as having two analytical layers. The first is the released dataset of raw texts and metadata [2506.05388]. The second is an inferred annotation layer generated by the actor-based pipeline, which is not embedded in the corpus release but underpins gender representation analysis and filtering [2508.13169].

## 4. Actor-based representation and discourse-aware analysis

The principal methodological innovation associated with taz2024full is an **actor-level, discourse-aware pipeline** for analyzing gender discrimination in large text corpora [2508.13169]. Actors are entities that participate in discourse and can be gendered. They are extracted primarily through **Named Entity Recognition** for persons plus **coreference resolution** linking names, titles, generic references, and pronouns [2508.13169]. Generic person terms such as “Mutter,” “Vater,” “Frau,” and “Mann” are also incorporated in the earlier corpus-analysis pipeline [2506.05388].

Each actor record stores all references to that actor, including names and pronouns, and the actor is assigned a **gender pronoun group** of **she/her** or **he/him** [2508.13169]. The framework conceptually acknowledges non-binary gender, but the implemented analysis is binary because of German linguistic constraints and the rarity of neo-pronouns in the dataset; the corpus paper reports only **5 texts** containing such pronouns [2506.05388]. Only actors with at least one resolved pronoun are considered in the pronoun-driven analysis [2506.05388].

The analysis is described as **actor-level** because metrics are computed **for each actor** and then aggregated by gender and by article or year, rather than being derived only at the document level [2508.13169]. It is described as **discourse-aware** because it operationalizes structures emphasized in systemic functional linguistics and critical discourse analysis: nomination, predication, syntactic roles, and quotation [2508.13169]. The pipeline therefore goes beyond simple token-frequency imbalance by tracking who is named versus pronominalized, who is syntactic subject versus object, who speaks directly versus indirectly, and how actors are framed sentimentally and lexically [2508.13169].

The processing stack includes NER and coreference resolution, sentence segmentation, dependency parsing for syntactic roles such as `nsubj` and `obj`, quote detection and attribution, a **German BERT-based sentiment classifier trained on news**, lexicon-based gender-coded word counts derived from Gaucher et al., and heuristics for gender-neutral forms and generic masculine usage [2508.13169]. In the corpus paper, the sentiment component is specified as **"oliverguhr/german-sentiment-bert"** integrated via the transformers pipeline [2506.05388].

Several metrics are formalized at the group level. For a gender group \(g\) and corpus unit \(u\), counts include actors \(A_{g,u}\), mentions \(M_{g,u}\), named mentions \(M^{\text{name}}_{g,u}\), pronoun mentions \(M^{\text{pro}}_{g,u}\), subject roles \(S_{g,u}\), object roles \(O_{g,u}\), direct quotes \(Q^{\text{dir}}_{g,u}\), indirect quotes \(Q^{\text{ind}}_{g,u}\), and counts of feminine- and masculine-coded words [2508.13169]. Ratios include share of mentions, share of actors, subject-role agency, direct versus indirect quotation proportions, and name-versus-pronoun usage [2508.13169]. Sentiment is computed at the actor level as

$$
S(a) = \frac{1}{n_a} \sum_{i=1}^{n_a} s(a,i)
$$

and aggregated by gender group within a unit [2508.13169].

In addition, the corpus and fairness papers use **PMI** to analyze associations between actors and contextual descriptors. The corpus paper gives the explicit formulation

$$
\textsf{PMI}(x, y) = \log_2 \frac{P(x, y)}{P(x)P(y)}
$$

for actor–adjective association [2506.05388], while the fairness paper describes PMI-based framing over adjectives, nouns, and verbs associated with actors of each gender [2508.13169].

## 5. Empirical gender patterns in taz2024full

The corpus has been used to study longitudinal gender representation from 1980 to 2024, and the findings consistently indicate **male overrepresentation**, albeit with gradual movement toward greater balance in recent years [2506.05388; 2508.13169]. In the unfiltered corpus, male actors dominate both **mentions** and **actor counts** overall [2508.13169]. The fairness paper reports that article-level gender ratios are often **0% or 100% female share**, meaning that many articles are effectively single-gender in terms of coded actors [2508.13169].

The diachronic trajectory is not static. The authors report a crossing around **2018**, where female share becomes comparable to or surpasses male share in some dimensions [2508.13169]. The corpus paper likewise reports that from the 2010s onward the share of women actors increases and approaches parity in recent years, although men still receive **more mentions**, implying greater textual space even when actor counts become similar [2506.05388].

A concrete annual example is provided for **2023** in the fairness study. There are **10,019 texts**, all with actors; pronoun counts are **6,892 she/her** and **9,194 he/him**; total mentions are **91,639**, split into **35,595 she/her** and **56,044 he/him**; named mentions are **22,544 she/her** versus **36,047 he/him**; pronoun mentions are **13,051 she/her** versus **19,997 he/him** [2508.13169]. Subject roles are **18,625 she/her** versus **30,303 he/him**, corresponding to **38.1%** and **61.9%** of known roles; object roles are **1,119 she/her** versus **1,540 he/him**, corresponding to **42.1%** and **57.9%** [2508.13169]. Direct quotes are **6,501 she/her** versus **10,588 he/him**, and indirect quotes are **2,529 she/her** versus **4,215 he/him** [2508.13169].

The corpus paper gives a different 2023 aggregate from its own yearly reporting configuration: **26,357 texts**, **17,161 actors**, pronoun distribution **9,088 he/him** versus **8,073 she/her actors**, total mentions **109,634**, average sentiment **−0.03 overall**, **−0.03 for she/her**, and **−0.02 for he/him**, with mean generic masculine indicator **0.31** and gender-neutral majority indicator **0.00** [2506.05388]. The discrepancy reflects different reporting scopes and pipeline stages rather than a contradiction explicitly resolved in the available descriptions.

The principal discourse-level asymmetries are threefold. First, men are more often in **subject positions**, while women are comparatively more often objects, which the authors interpret as greater narrative agency for male actors [2508.13169]. Second, male actors are more frequently quoted in **direct speech**, while female actors appear more often in **indirect quotes**, and direct quotation is interpreted as conferring greater authority and visibility [2508.13169]. Third, sentiment is close to neutral for both genders, but the average sentiment toward female-coded actors is reported as **more negative in almost all years** [2508.13169]. The corpus paper similarly states that sentiment toward women is consistently slightly more negative over the full 44-year period, though the differences are small [2506.05388].

At the lexical framing level, overt gender-coded language appears limited. The corpus paper reports that top PMI adjectives associated with women and men are relatively stable over time and do **not show strong gender differentiation**, and that feminine-coded and masculine-coded words are **rare overall** [2506.05388]. The fairness paper, however, reports that PMI lists for 2023 associate women more with terms such as **“Frau, Mutter, Tochter, Kinder”** and men more with **“Präsident, Mann, Sohn”**, indicating persistent thematic role differentiation [2508.13169]. This suggests that absence of strong overt lexical coding does not eliminate subtler framing asymmetries.

## 6. Exclusion-based balancing and fairer corpus construction

A major extension of the taz2024full research program is the **multi-stage, exclusion-based balancing** framework for constructing less discriminatory corpus variants [2508.13169]. The pipeline consists of four stages: actor-based discrimination analysis of the full corpus; reuse of article-level results for exclusion; corpus-level balancing to bring global gender ratios into a user-defined range; and final reconstruction of the filtered corpus by removing excluded article IDs from the original JSON files [2508.13169].

Text-level exclusion is based on four heuristic discrimination scores for an article \(u\): **sentiment disparity** \(\Delta_S(u)\), **grammatical role asymmetry** \(\Delta_A(u)\), **quote attribution imbalance** \(\Delta_Q(u)\), and **naming versus pronoun imbalance** \(\Delta_N(u)\) [2508.13169]. All ratios are **Laplace-smoothed** to mitigate instability when counts are low [2508.13169]. The default thresholds are \(|\Delta_S(u)| > 0.3\) for sentiment and \(> 0.5\) for role, quote, and naming imbalances [2508.13169]. The default decision rule excludes an article if **two out of four** criteria exceed their thresholds [2508.13169].

After text-level filtering, a second stage applies **corpus-level balancing** to actor and mention ratios. The authors define

$$
R^{\text{actors}} = \frac{A_{\text{she/her}}}{A_{\text{he/him}}}
\quad \text{and} \quad
R^{\text{mentions}} = \frac{M_{\text{she/her}}}{M_{\text{he/him}}}
$$

and target a default equilibrium interval of \([0.75, 1.25]\), allowing up to \( \pm 25\% \) deviation from parity [2508.13169]. Articles are iteratively excluded based on their marginal contribution to global imbalance until both ratios fall within the target range [2508.13169].

The reported scale of filtering is substantial. Starting from **1,834,018 articles** with identifiable gendered actors, **279,772** are removed at the text level and an additional **17,815** at the corpus-balancing stage, for a total of **297,587** removed articles and a final corpus of approximately **1.54 million articles** [2508.13169]. After filtering, article-level gender ratios become less polarized, and after full balancing the distribution becomes more centered around parity [2508.13169]. The authors emphasize **referential parity** in the final corpus, meaning that male and female actors appear in comparable proportions across both mentions and actor counts [2508.13169].

Discourse-level asymmetries are reduced but not eliminated. The subject-role gap decreases **from approximately 30 percentage points to approximately 5**, quotation imbalance is reduced, and women appear in direct speech more frequently than before [2508.13169]. Sentiment, however, remains **slightly more negative for women**, even if the remaining gap stays below the exclusion threshold of \(0.3\) [2508.13169]. The authors interpret this as evidence that balancing improves structural asymmetries but does not fully remove evaluative bias [2508.13169].

## 7. Accessibility, methodological limitations, and research significance

The corpus is hosted on **Zenodo** with DOI **https://doi.org/10.5281/zenodo.15480855**, and the analysis pipeline is released on GitHub as **`https://github.com/Ognatai/corpus_pipeline`** [2506.05388]. The follow-up balancing work releases tools and reports at **`https://github.com/Ognatai/corpus_balancing`**, including the extended actor-level pipeline, the two-stage exclusion framework, an interactive interface for setting thresholds and equilibrium ranges, and scripts for report generation and corpus reconstruction [2508.13169]. The papers state that the pipeline and reports are released, but they do **not** explicitly state that a fully processed or filtered version of taz2024full is released [2508.13169].

Several limitations are explicit. First, taz2024full is a **single-newspaper corpus** from a **left-leaning** outlet and is not representative of the full German media spectrum [2506.05388]. Second, only **publicly available** content is included, so later years may be skewed by the absence of paywalled material, especially after around 2007 [2506.05388]. Third, early years are sparse, and the authors caution against over-interpreting strong fluctuations in yearly percentages from the 1980s [2508.13169].

Methodologically, the actor pipeline depends heavily on **NER**, **coreference resolution**, and pronoun identification [2508.13169]. German coreference is specifically acknowledged as weak, and no gold-standard evaluation, precision, recall, or F1 is reported for the full actor-level German pipeline [2506.05388]. Articles without clearly identifiable actors or gender cues are excluded from discrimination analysis, producing incomplete coverage [2508.13169]. The gender model is also restricted to a **binary approximation**, despite the conceptual acknowledgment that gender is socially constructed and non-binary [2506.05388; 2508.13169].

The fairness framework is equally bounded by its chosen metrics. It relies on **surface-level indicators** such as syntactic roles, sentiment scores, quote types, and counts of gendered terms [2508.13169]. It cannot capture irony, sarcasm, omission, topic-selection bias, or higher-level narrative structure [2508.13169]. Threshold choices are normative and materially affect outcomes; the authors mitigate this with interactive tools, logging, and visual diagnostics, but they do not claim that the procedure is fully objective [2508.13169].

Within German NLP and computational social science, taz2024full is significant because it combines **full-text accessibility**, **diachronic depth**, and a **documented bias profile** in a way that supports both descriptive and intervention-oriented research [2506.05388]. It enables studies of language change, media discourse, agenda setting, sentiment trajectories, and actor-centric representation over four decades [2506.05388]. In the LLM context, the fairness paper argues that structural gender imbalances in training data shape the representational landscape of model outputs, and positions taz2024full as a basis for constructing **fairer training subsets** with reduced asymmetries in counts, agency, and quotation [2508.13169]. At the same time, the authors warn that exclusion-based balancing reduces corpus size and possibly diversity, may distort historical reality, and does not by itself yield “fully fair” training data [2508.13169].

Taken together, taz2024full functions both as a large German news corpus and as a methodological platform for actor-centered fairness auditing. Its core contribution lies not only in scale, but in the combination of article-level accessibility, longitudinal scope, and discourse-aware analytical infrastructure for studying how representation, visibility, and framing evolve across decades of German journalism [2506.05388; 2508.13169].

Source: https://www.emergentmind.com/topics/taz2024full-corpus