---
title: 'KorFinSTS: Korean Financial Semantic Similarity'
url: https://www.emergentmind.com/topics/korfinsts
type: topic
---

# KorFinSTS: Korean Financial Semantic Similarity

Searching arXiv for the specified paper and closely related resources to ground the article.
arxiv_search(query="2507.09601 OR KorFinSTS OR NMIXX finance embeddings", max_results=10, sort_by="submittedDate")
arxiv_search(query="KorFinSTS", max_results=5, sort_by="relevance")
KorFinSTS, short for Korean Financial Semantic Textual Similarity, is a purpose-built evaluation suite for measuring how well sentence-embedding models capture the fine-grained semantics of Korean financial language. It was introduced alongside the NMIXX cross-lingual finance embedding models as part of the effort to address the difficulty that general-purpose sentence embeddings face with domain-specific jargon, temporal meaning shifts, and bilingual misalignment in finance, especially for low-resource languages such as Korean [2507.09601]. Within that framework, KorFinSTS serves as an evaluation benchmark rather than a training resource, and is intended to expose semantic distinctions that general STS benchmarks do not reliably surface.

## 1. Domain and design rationale

KorFinSTS is defined in the source paper as an evaluation suite that draws exclusively on authentic Korean finance texts rather than on general-domain paraphrase material [2507.09601]. Its scope spans four sub-domains: financial news, regulatory disclosures, investment research reports, and legal or regulatory texts. The benchmark was designed specifically to surface domain-specific phenomena such as market-cycle jargon, regulatory formality, perspectival framing, and temporal semantic drift.

This domain restriction is central to its methodological role. General STS resources such as KorSTS, and English-focused financial resources such as FinSTS, do not directly target the semantic idiosyncrasies of Korean financial discourse. KorFinSTS therefore occupies the intersection of financial NLP, Korean-language representation learning, and sentence-level semantic evaluation. A plausible implication is that the benchmark is not merely testing lexical overlap or general paraphrase sensitivity, but the ability to preserve financially consequential distinctions in meaning.

## 2. Corpus construction and curation workflow

KorFinSTS was constructed by sampling from the same open-license Korean financial corpora used to train NMIXX. From six primary sources, described as news aggregators, disclosures, research portals, and regulatory filings, the authors filtered and down-sampled the data to ensure equal representation of the four sub-domains [2507.09601].

The workflow then applied sentence-length and artifact filters. Sentences shorter than 128 or longer than 4,096 tokens were removed, and artifacts such as HTML and OCR noise were purged. After this stage, the retained pool comprised approximately 46 K high-quality sentences. Candidate sentence pairs were then selected to cover a spectrum of relatedness, ranging from near-identical paraphrases to more subtle semantic divergences, including contrasts such as “steady growth” versus “significant growth” and temporal shifts such as “pre-announcement” versus “post-announcement.”

Manual review was performed by six domain experts, each described as having a graduate degree in economics or business and current industry experience. Their role was to verify factual accuracy and preserve genuine financial nuance. This emphasis on expert review distinguishes the benchmark from automatically mined or weakly supervised STS datasets and suggests a deliberate prioritization of semantic validity over scale.

## 3. Annotation protocol and dataset profile

KorFinSTS uses a traditional 0–5 similarity scale, where 0 denotes no meaningful semantic overlap and 5 denotes near-perfect paraphrase [2507.09601]. The high-level annotation guidelines instructed experts to preserve financial core meaning rather than surface word overlap, to distinguish shifts in tone or modality such as planned versus actualized events, and to penalize surface similarity when economic implications diverge.

The paper does not report a formal inter-annotator agreement statistic such as Cohen’s $\kappa$. Instead, conflicting scores were reconciled through group adjudication to produce a single gold rating for each pair. This is a notable methodological feature: the benchmark emphasizes adjudicated gold labels, but does not provide a formal agreement coefficient for independent annotations.

The dataset statistics contain a minor revision detail. The benchmark is described in the abstract as a 1,921-pair Korean financial STS benchmark, while the detailed dataset section states that KorFinSTS comprises 1,991 sentence pairs and notes that the paper text occasionally refers to 1,921, reflecting a minor revision [2507.09601]. The detailed sub-domain breakdown is as follows:

| Sub-domain | Pairs |
|---|---:|
| Financial News | 355 |
| Disclosures | 500 |
| Investment Reports | 421 |
| Legal Texts | 715 |

The average token lengths of Sentence 1 and Sentence 2 are 535.5 and 457.5, respectively, and the combined vocabulary size across all sentences is 4,057 unique tokens. Similarity scores on the 0–5 scale have a mean of 0.59 and a standard deviation of 0.49. The resource is released as an evaluation benchmark only: all 1.9 K pairs are provided for testing, and no separate train or validation splits are defined.

## 4. Evaluation formalism

Performance on KorFinSTS is measured by Spearman’s rank correlation $\rho$ between model cosine-similarity scores and human judgments [2507.09601]. The benchmark therefore evaluates whether an embedding model preserves the ranking induced by expert semantic similarity scores rather than optimizing for absolute regression error.

The paper gives the metric in the following form:

$$
\rho = 1 - \frac{6 \sum_{i=1}^{n} d_i^2}{n (n^2 - 1)},
$$

where $d_i$ is the difference between the human rank and model rank for the $i$-th pair.

This choice is standard for STS, but in KorFinSTS it has particular importance because many distinctions of interest are ordinal and domain-sensitive: models are rewarded for ordering semantically close and semantically divergent financial statements in a way that aligns with expert judgment. This suggests that the benchmark is especially suitable for comparing embedding spaces intended for retrieval, clustering, and semantic search in finance.

## 5. Baselines, adaptation effects, and empirical findings

The source paper evaluates KorFinSTS against seven open-license baselines and reports Spearman’s $\rho$ before and after domain adaptation under the NMIXX triplet objective [2507.09601]. The multilingual BGE-m3 variant records the strongest absolute score and the largest improvement on KorFinSTS: it moves from $\rho = 0.0512$ before adaptation to $\rho = 0.2732$ after adaptation, a gain of $+0.2220$.

Other reported changes are also substantial for some models. gte-Qwen2-1.5B improves from $0.0094$ to $0.2204$, a gain of $+0.2110$. By contrast, bge-large-en improves from $-0.2119$ to $-0.1586$, all-MiniLM from $-0.1837$ to $-0.1590$, and instructor-base from $-0.0982$ to $-0.0679$. Two baselines degrade after adaptation: bge-en-icl moves from $0.0511$ to $-0.0745$, and e5-mistral from $0.1099$ to $-0.1738$.

A further result reported in the paper is that BGE-m3’s post-adaptation KorFinSTS performance is accompanied by a Spearman drop of only approximately $0.039$ on general English STS. The paper characterizes this as strong domain transfer with limited general degradation. In the broader NMIXX evaluation, the multilingual bge-m3 variant is also reported to achieve a Spearman gain of $+0.10$ on English FinSTS and $+0.22$ on KorFinSTS relative to its pre-adaptation checkpoint, while outperforming its pre-adaptation checkpoint and surpassing other models by the largest margin [2507.09601].

These outcomes indicate that KorFinSTS is difficult for general-purpose and English-centric encoders, including models that may perform competitively on broader STS tasks. A plausible implication is that the benchmark operationalizes semantic phenomena that are weakly represented in general multilingual pretraining.

## 6. Tokenization, Korean coverage, and benchmark significance

The paper’s analysis emphasizes tokenizer design as a decisive factor in low-resource Korean financial STS [2507.09601]. In Table 7, BGE-m3 is reported to contain 5,413 full-syllable Korean tokens, corresponding to 2.17% of its 250,002-token vocabulary. Many baselines have either no Korean token coverage or very little; the paper gives Qwen2-1.5B as an example with 0% Korean token coverage. Models with richer Korean token coverage are reported to achieve both higher absolute $\rho$ on KorFinSTS and larger gains from domain adaptation.

This makes KorFinSTS significant beyond its immediate use as a leaderboard benchmark. It functions as a diagnostic instrument for studying how vocabulary granularity, tokenizer coverage, and domain adaptation interact in multilingual sentence-embedding systems. The benchmark therefore bears directly on representation learning choices for specialized Korean NLP, especially where sentence embeddings are expected to preserve fine distinctions in regulatory, market, and analyst language.

The same analysis also clarifies a potential misconception. Poor baseline performance on KorFinSTS does not necessarily imply that a model is broadly ineffective at STS; rather, it may reflect a mismatch between tokenizer design and the semantic density of Korean financial text. The benchmark is specifically tuned to reveal that mismatch.

## 7. Limitations and prospective extensions

The paper identifies several limitations that remain open for future work [2507.09601]. First, KorFinSTS does not provide formal train, development, and test splits, and it does not report inter-annotator agreement statistics. Both omissions limit certain forms of method development and reproducibility analysis. Second, several training strategies were not explored, including hybrid regimens that combine domain adaptation with general STS, progressive pre-training followed by fine-tuning, and the inclusion of LLM-based embedders such as llm2vec.

The paper also points to broader multilingual extensions. It identifies other low-resource language pairs, including Vietnamese, Thai, and Indonesian, as promising targets, and mentions alternative tokenization strategies such as Unigram LM and domain-specific vocabularies. This suggests that KorFinSTS is both a benchmark and a template: a concrete Korean financial STS resource, but also an instance of a more general program for domain-adapted, multilingual evaluation in finance.

In that sense, KorFinSTS occupies a specific methodological niche. It is a benchmark-only resource with expert-adjudicated labels, authentic domain text, and explicit sensitivity to tokenizer coverage and semantic drift. Its primary contribution is not scale, but the concentration of financially meaningful semantic phenomena into an evaluation setting that is difficult for general-purpose embeddings and informative for domain-adapted multilingual models [2507.09601].

Source: https://www.emergentmind.com/topics/korfinsts