FinHeadlineMix: Synthetic Finance Headline Benchmark

• FinHeadlineMix is a synthetic financial headline classification dataset comprising 493,000 examples across 35 event classes, designed for quantitative finance research.
• The dataset supports controlled evaluation of model distillation by comparing label-only and chain-of-thought supervision formats under iterative pruning conditions.
• Its design facilitates rigorous scaling-law analyses by isolating the impact of supervision granularity and compression on in-domain performance and general knowledge retention.

Searching arXiv for the specified paper and closely related work to ground the article. FinHeadlineMix is a synthetic headline classification dataset introduced in the context of task-specific LLM distillation for quantitative finance. It was released alongside a study of empirical scaling laws for domain-specific compression, where it serves as the principal in-domain benchmark for analyzing how performance varies with dataset size, compression ratio, supervision format, and iterative pruning schedule (Ghita et al., 23 Jun 2026). The dataset is organized around multiclass event classification from single news headlines, with supervision available either as label-only targets or as chain-of-thought traces followed by a final class label. Within the associated experimental framework, FinHeadlineMix is not merely a benchmark corpus but a controlled substrate for studying the interaction between structural pruning, KL-based logit matching, LoRA-based supervised fine-tuning, and blended chain-of-thought supervision under deployment-oriented constraints.

1. Origin, construction, and corpus statistics

FinHeadlineMix was synthetically generated using NVIDIA Nemotron-3-Nano-30B, instruction-tuned through NeMo Data Designer (Ghita et al., 23 Jun 2026). After generation, a fuzzy-deduplication stage implemented with NeMo Curator reduced the initial pool to approximately 500,000 unique headlines. Each headline was then labeled by Qwen3-32B in “thinking mode,” producing chain-of-thought plus final-class outputs, with a small discard rate for failed chain-of-thought runs of less than 1% (Ghita et al., 23 Jun 2026).

The released dataset contains 493,000 examples, described as approximately 500k after deduplication, and is partitioned into 400,000 training examples, 93,000 validation examples, and 10,000 test examples, with splits stratified by class (Ghita et al., 23 Jun 2026). The class distribution is naturally imbalanced, and that distribution is identical across train, validation, and test sets. This identical distribution across splits is important for the scaling-law analysis because it removes one obvious confound between data regime and class prior shift.

The underlying label taxonomy contains 35 classes: 34 event types plus an “OTHER” category (Ghita et al., 23 Jun 2026). The paper explicitly lists examples such as Earnings Surprise, M&A Transaction Event, and Cybersecurity Incident. This framing places FinHeadlineMix in the category of event-oriented financial text classification, but with a granularity designed to support compression studies rather than only downstream benchmarking.

A concise summary of the published corpus statistics is as follows:

Attribute	Value
Total examples	493,000
Train / validation / test	400,000 / 93,000 / 10,000
Number of classes	35
Source generation model	NVIDIA Nemotron-3-Nano-30B
Labeling model	Qwen3-32B in “thinking mode”

Because the headlines are synthetic and the labels are teacher-generated, FinHeadlineMix should be understood as a teacher-mediated supervision resource rather than a directly human-annotated benchmark. A plausible implication is that its primary research value lies in controlled distillation and scaling experiments, where consistency of supervision may matter as much as, or more than, the provenance of the raw text.

2. Task formulation and supervision formats

The core task on FinHeadlineMix is multiclass event classification from a single news headline string $x$, with output $y$ restricted to one of 35 discrete classes (Ghita et al., 23 Jun 2026). This is a standard conditional generation setup at the model interface, but the supervision is provided in two distinct formats that materially alter the distillation dynamics.

In the label-only format, the student produces the single-token or short multi-token class label, described as 1–3 tokens (Ghita et al., 23 Jun 2026). In the chain-of-thought format, the student reproduces the full reasoning trace, approximately 10–50 tokens, followed by the final label. The paper gives an illustrative example: for the headline “ACME Corp Q4 EPS tops estimates as pricing power offsets softer volumes,” the label-only target is [[Earnings Surprise]], whereas the abbreviated chain-of-thought target explains that EPS beat expectations because pricing power offset lower volumes and then concludes with [[Earnings Surprise]] (Ghita et al., 23 Jun 2026). A second example maps “Port congestion delays shipments; retailer warns of stockouts” to [[Supply Chain Disruption]].

These two supervision formats define more than alternative label encodings. They instantiate different objectives over the output sequence: either compression toward a terminal decision token span or compression toward both intermediate reasoning tokens and the terminal decision. In the associated study, supervision format is identified as the primary lever governing the trade-off between in-domain quality and preservation of general knowledge under compression (Ghita et al., 23 Jun 2026).

This suggests that FinHeadlineMix functions simultaneously as a classification dataset and as a sequence-level distillation dataset. The distinction is central: the chain-of-thought targets introduce substantially longer output trajectories, thereby changing both the token-level optimization landscape and the extent to which the student is constrained to mimic teacher behavior beyond the final class assignment.

3. Preprocessing and token-level training interface

FinHeadlineMix uses minimal preprocessing. The paper states that there is no additional text normalization beyond prompt conditioning, and tokenization and vocabulary are inherited from the teacher, Qwen3-32B (Ghita et al., 23 Jun 2026). This design keeps the data interface tightly aligned with the teacher model’s representational space, which is especially relevant for logit-based KL distillation.

During distillation, input tokens are masked, and only teacher outputs contribute to the loss, whether those outputs consist of chain-of-thought tokens plus label or label only (Ghita et al., 23 Jun 2026). This is a consequential detail. It means the optimization target is entirely output-side imitation rather than bidirectional sequence modeling over concatenated prompt and completion. In practice, such masking isolates the loss to the generated supervision sequence and makes the student’s objective a direct approximation of the teacher’s conditional response distribution on the task.

Because the vocabulary is inherited from the teacher and the student is trained against teacher outputs rather than normalized labels alone, FinHeadlineMix is structurally tailored to teacher-student transfer. It is therefore distinct from conventional text classification corpora that can be consumed interchangeably by discriminative encoders, generative decoders, or instruction-tuned models with arbitrary tokenizers. Here, the dataset and the training protocol are co-specified.

A plausible implication is that cross-model portability may be lower than for traditional benchmark datasets, while within-family compression studies become cleaner. That trade-off aligns with the paper’s focus on deriving empirical scaling laws for task-specific compression rather than maximizing broad benchmark interoperability.

4. Distillation methods used with FinHeadlineMix

The study evaluates two principal distillation regimes on FinHeadlineMix: LoRA-based label-only supervised fine-tuning and logit-based soft KL distillation (Ghita et al., 23 Jun 2026). These are compared under iterative structural pruning.

In the LoRA-based method, low-rank adapters are inserted into all Q/K/V and MLP projections, with rank $r$ scaled to dataset size, for example from $r=4$ to $64$ (Ghita et al., 23 Jun 2026). The loss is standard cross-entropy on the teacher’s discrete label. This makes the LoRA condition a label-only supervised fine-tuning baseline in which the student is optimized to produce the class label directly, without matching teacher token distributions over extended reasoning traces.

In the logit-based method, the student matches the teacher’s per-token probability distributions over the entire output sequence using KL divergence. The per-token loss is given as

$$\ell_k = \mathrm{KL}\bigl(p_{t,k}(\tau)\,\|\,p_{s,k}(\tau)\bigr),$$

where $\tau$ is the softmax temperature, set to 1 (Ghita et al., 23 Jun 2026). Because this objective is defined over every output position, it supports both label-only and chain-of-thought supervision, but the paper emphasizes a particular variant for chain-of-thought targets.

That variant is the blended chain-of-thought loss. The motivation is that when the output sequence contains many chain-of-thought tokens, gradients on final label tokens can be “washed out” by the longer reasoning region. To address this, the paper introduces the loss

$$\mathcal{L} = \lambda\,\overline{\ell}_{\mathrm{label}} + (1-\lambda)\,\overline{\ell}_{\mathrm{cot}},$$

with default $\lambda = 0.5$, where $\overline{\ell}_{\mathrm{label}}$ is the mean KL over final label positions and $y$0 is the mean over preceding chain-of-thought tokens (Ghita et al., 23 Jun 2026). Per-token weights are chosen so that each pool contributes equally:

$y$1

with $y$2 (Ghita et al., 23 Jun 2026).

Within the published experimental interpretation, blended chain-of-thought KL distillation is described as sample-efficient, as stabilizing reasoning traces, and as recovering general knowledge that pruning erases (Ghita et al., 23 Jun 2026). By contrast, direct label-only KL distillation is described as fragile, collapsing at large data sizes and erasing general knowledge. These conclusions are part of the paper’s practical recommendations and are tightly tied to FinHeadlineMix as the in-domain task.

5. Iterative pruning and scaling-law analysis

FinHeadlineMix is the in-domain evaluation target for a study of scaling laws in task-specific compression. The empirical model reported for in-domain performance is

$y$3

where $y$4 denotes training tokens and $y$5 the compression ratio (Ghita et al., 23 Jun 2026). The paper reports approximate coefficients for Macro F1 on FinHeadlineMix. For logit-based blended chain-of-thought distillation, $y$6, $y$7, $y$8, and $y$9. For LoRA-based label-only supervision, $r$0, $r$1, $r$2, and $r$3 (Ghita et al., 23 Jun 2026).

The interpretation given in the paper is that doubling the dataset size yields an approximately 20–25% relative gain in F1, while halving model size incurs a roughly 20–50% cost depending on method (Ghita et al., 23 Jun 2026). These coefficients were obtained by fitting log-log lines to the curves in Figures 2 and 4 of the paper.

Compression is carried out through iterative pruning plus re-distillation. Starting from Qwen3-32B, the model is pruned in one to four steps, with an example being uniform 21 percentage-point reductions to reach 16% of original size, and each intermediate checkpoint is re-distilled from the original teacher (Ghita et al., 23 Jun 2026). LoRA students at each size are trained from scratch rather than iteratively because adapter dimensions change under pruning.

The paper’s recommendations emphasize that iterative pruning is essential below approximately 37% of original size. A single-step reduction to 16% fails, with $r$4, whereas a three- or four-step uniform schedule recovers $r$5–$r$6 (Ghita et al., 23 Jun 2026). Among decayed schedules to 16%, linear decay—specified as $r$7 percentage points $r$8 percentage points $r$9 percentage points $r=4$0 percentage points—achieves the best in-domain metrics, with Macro F1 approximately 0.77, NLL approximately 0.65, and Brier approximately 0.31. Exponential decay front-loads pruning most aggressively and still recovers F1 approximately 0.76, leaving headroom for a fifth step below 16% (Ghita et al., 23 Jun 2026).

These results position FinHeadlineMix as a benchmark for compression sensitivity, not just task performance. The dataset’s large scale and synthetic consistency permit explicit estimation of how in-domain quality degrades under aggressive structural pruning and how supervision choices modulate that degradation.

6. In-domain performance versus general-knowledge retention

A defining claim of the study is that in-domain task quality on FinHeadlineMix degrades predictably under compression, while general-knowledge benchmarks collapse much earlier, and that supervision format is the key driver of this trade-off (Ghita et al., 23 Jun 2026). In this sense, FinHeadlineMix is used to expose a tension between narrow task preservation and broader capability retention in compressed LLMs.

The practical recommendations make this relationship explicit. Blended chain-of-thought KL distillation is said to recover general knowledge wiped out by pruning, whereas LoRA label-only training does not restore general benchmarks beyond the pruned baseline (Ghita et al., 23 Jun 2026). Direct label-only KL distillation is reported to be especially unstable, both collapsing at large data sizes and erasing general knowledge. The general-knowledge benchmark specifically referenced is MMLU, for which scores at extreme compression collapse to random guess unless chain-of-thought supervision is used (Ghita et al., 23 Jun 2026).

The data-vs-compression trade-off is quantified in several operating regimes. With 100k–200k FinHeadlineMix examples, blended chain-of-thought knowledge distillation at 50% model size reaches Macro F1 approximately 0.75; LoRA surpasses this only once more than 200k examples are used (Ghita et al., 23 Jun 2026). At 16% of original size, both methods plateau in-domain, at approximately 0.77 Macro F1 for blended chain-of-thought and approximately 0.70 for LoRA, but only the chain-of-thought-based setting avoids complete collapse of MMLU (Ghita et al., 23 Jun 2026).

A plausible implication is that FinHeadlineMix supports evaluation of “capability-preserving compression” rather than just domain adaptation. The paper does not redefine this as a formal benchmark category, but its findings suggest that a finance-domain dataset with reasoning-trace supervision can be informative about non-financial retention under pruning when the objective matches teacher distributions over intermediate reasoning tokens.

7. Practical role in finance-domain model compression

Within the study, FinHeadlineMix is presented together with scaling-law results and practical recommendations intended as a reusable framework for domain-specific compression decisions (Ghita et al., 23 Jun 2026). For a finance-domain LLM under latency and cost constraints, the paper’s key takeaway is a three-part recipe: generate or collect 100k–200k headlines with chain-of-thought labels; use logit-based KL distillation with blended chain-of-thought supervision and a three-step linear or exponential pruning schedule down to the target size; and, if at least 400k examples are available and only the 35-way classification task matters, consider LoRA label-only training because it yields slightly higher in-domain quality (Ghita et al., 23 Jun 2026).

These recommendations clarify the operational position of FinHeadlineMix. It is a dataset for a 35-way financial event taxonomy, but also a decision-support instrument for compression strategy selection. The study identifies supervision format as the dominant control variable, iterative pruning as necessary in the low-parameter regime, and dataset scale as mediating when LoRA overtakes blended chain-of-thought KL on the in-domain objective (Ghita et al., 23 Jun 2026).

Several misconceptions are precluded by the published results. First, label-only supervision is not uniformly preferable just because the task is classification; the paper reports that chain-of-thought supervision can materially improve general-knowledge retention under pruning. Second, compression is not treated as a single post-training step; below roughly 37% of original size, iterative schedules are described as essential. Third, more in-domain data does not by itself solve the compression problem for all objectives, since direct label-only KL distillation is reported to remain fragile even at large data sizes (Ghita et al., 23 Jun 2026).

Taken together, FinHeadlineMix occupies a specific niche in the contemporary LLM literature: a synthetic, large-scale, finance-domain headline corpus engineered for controlled study of task-specific distillation. Its main significance lies less in establishing a canonical public benchmark than in enabling systematic analysis of how supervision granularity, token-level distillation targets, and pruning schedules interact when compressing large generative models for specialized deployment settings (Ghita et al., 23 Jun 2026).