---
title: 'SEGALE: Long-Form Translation Evaluation'
url: https://www.emergentmind.com/topics/segale
type: topic
---

# SEGALE: Long-Form Translation Evaluation

SEGALE most prominently denotes an evaluation scheme for long-document machine translation that extends sentence-level metrics to continuous texts by **SEGment**, **ALign**, and **Evaluate** operations, thereby enabling document-level scoring under boundary mismatch, under-translation, and over-translation [2509.17249]. In subsequent work on long-form simultaneous speech-to-speech translation, SEGALE functions as a sentence-embedding-based aligner that maps target speech-derived text to source sentences so that latency and quality can be computed over aligned spans [2606.15059]. The term also appears, in a separate and unrelated usage, as an informal label for the LEGEND-200 signal readout electronics, although that acronym is not explicitly defined in the corresponding LEGEND paper [1911.05847].

## 1. Definition and scope

In its machine-translation usage, SEGALE is designed for settings in which both source and translation are treated as continuous text without guaranteed sentence-boundary agreement. Its core objective is to recover a monotonic, contiguous, many-to-many alignment between automatically segmented source and target spans, permit null alignments on either side, and then apply existing sentence-level metrics such as COMET, MetricX, BLEU, chrF, or BERTScore to the aligned blocks [2509.17249]. This design directly addresses three constraints emphasized in the literature: dataset annotations are often sentence-level, automatic metrics have token limits, and long-context translations generated with document-level prompts may exhibit under- and over-translation as well as arbitrary sentence-boundary variation [2509.17249].

The formal setting introduces a source document $D_{\mathrm{src}}$ and a translation $D_{\mathrm{tgt}}$, sentence segmentation functions that produce sequences $S=\{s_1,\dots,s_N\}$ and $T=\{t_1,\dots,t_M\}$, and an alignment objective over contiguous spans on both sides [2509.17249]. The aligned spans are then concatenated locally before metric application. This local-span strategy is central because it sidesteps the global token-count restrictions of metrics such as COMET and MetricX while preserving document-level coverage [2509.17249].

A distinct but closely related usage arises in long-form simultaneous speech-to-speech translation. There, source speech, pre-segmented source transcripts, reference translations, and generated target speech are combined in a pipeline that performs target-side ASR, forced alignment, sentence boundary detection, and SEGALE-based alignment before computing group-level latency and quality scores [2606.15059]. In that setting, SEGALE is not itself the entire evaluator but the alignment backbone that makes sentence- or span-level aggregation practical over long continuous speech [2606.15059].

This suggests that SEGALE is best understood not as a single metric, but as an alignment-and-normalization layer that makes pre-existing metrics usable in long-form evaluation regimes.

## 2. Segmentation and alignment methodology

SEGALE begins with automatic sentence segmentation. The documented implementations use off-the-shelf segmenters: **ersatz**, described as a language-agnostic rule-based segmenter, and **spaCy**, with language-specific pipelines including `en_core_web_sm`, `de_core_news_sm`, `es_core_news_sm`, `ja_core_news_sm`, and `zh_core_web_sm` [2509.17249]. In the reported experiments, spaCy produced finer-grained segmentation than ersatz, but downstream evaluation correlation was similar; ersatz was the default choice because its segmentation granularity was closer to WMT test sets and its null-alignment behavior was slightly better aligned with gold annotations [2509.17249].

Alignment is built on **Vecalign**, modified for SEGALE with many-to-many monotonic alignment over contiguous spans, multilingual sentence embeddings for block similarity, explicit skip costs for null alignments, and an adaptive search over the skip threshold $B_{\mathrm{skip}}$ [2509.17249]. The aim is to avoid two complementary failure modes: a skip penalty that is too high forces semantically unrelated blocks to align, whereas a skip penalty that is too low yields excessive null alignments [2509.17249]. The search therefore starts at $B_{\mathrm{skip}}=0.2$ and decreases in steps of $0.005$, while monitoring average alignment cost and null-alignment ratio. The reported stopping heuristics are: stop when average alignment cost $<0.3$, stop when the null-alignment ratio exceeds $0.15$, and early-stop if average cost increases or exceeds $0.7$; the selected solution is the last safe alignment before over-deletion is triggered [2509.17249].

The conceptual dynamic program can be written as
$$
dp(i,j)=\max\left\{
dp(i-1,j-1)+g(i,j),\;
dp(i-1,j)-\gamma_{\mathrm{src}},\;
dp(i,j-1)-\gamma_{\mathrm{tgt}}
\right\},
$$
with monotonicity and contiguous-block extensions for many-to-many matches [2509.17249]. In practice, Vecalign implements this efficiently through bounded block search rather than exhaustive quadratic dynamic programming [2509.17249].

For document-level MT evaluation, SEGALE uses multilingual sentence embeddings, with LASER and BGE-M3 as baselines and a fine-tuned **BGE-M3** variant as the preferred encoder [2509.17249]. That fine-tuned model was trained on 130,436 synthetic triplets from News Commentary v18.1 for $\{\mathrm{zh},\mathrm{en},\mathrm{de},\mathrm{ja},\mathrm{es}\}$ using InfoNCE over adjacent-sentence concatenations and hard negatives that either drop a target sentence or substitute a nearby non-parallel sentence [2509.17249]. The overlap size for candidate span alignments is set to $K=16$, limiting candidate blocks by $N+M\le K$ [2509.17249].

In long-form simultaneous speech-to-speech translation, the alignment principle is similar but the inputs differ. Generated target speech is first transcribed and force-aligned into tokens with timestamps, then split into target sentences with spaCy, and finally aligned to source sentence spans using SEGALE, again allowing one-to-one, one-to-many, many-to-one, many-to-many, and null alignments [2606.15059]. The reported cost uses cosine similarity over sentence embeddings,
$$
s(u,v)=\frac{E(u)\cdot E(v)}{\|E(u)\|\|E(v)\|},
$$
with span cost
$$
C(i\!:\!j,p\!:\!q)=1-s(S(i\!:\!j),\hat{T}(p\!:\!q)),
$$
and an adaptive skip-penalty search analogous to the MT setting [2606.15059].

## 3. Extension of sentence-level metrics to long documents

Once an alignment is obtained, SEGALE converts any sentence-level metric into a document-level evaluator by scoring aligned spans after concatenation. For an aligned source span $I$ and target span $J$, the block score is
$$
f_{IJ}=f\!\left(\operatorname{concat}_{i\in I}s_i,\operatorname{concat}_{j\in J}t_j\right),
$$
where $f$ may be COMET, MetricX, BLEU, chrF, BERTScore, or another sentence-level metric [2509.17249]. Null alignments explicitly encode missing or hallucinated content. Under-translation corresponds to unmatched source spans, and over-translation to unmatched target spans; both receive a fixed worst-case penalty $f_{\mathrm{null}}$ [2509.17249]. The documented values are $f_{\mathrm{null}}=0$ for COMET and $f_{\mathrm{null}}=25$ for MetricX [2509.17249].

The document-level SEGALE score is then
$$
F(D_{\mathrm{src}},D_{\mathrm{tgt}};f)=
\frac{\sum_{(I,J)\in A} f_{IJ}+\sum_{u\in U} f_{\mathrm{null}}}{|A|+|U|},
$$
where $A$ is the set of aligned span pairs and $U$ the set of null alignments [2509.17249]. SEGALE also reports the **null alignment ratio**
$$
NA(D_{\mathrm{src}},D_{\mathrm{tgt}})=\frac{|U|}{|A|+|U|},
$$
which acts as an explicit diagnostic for under- and over-translation [2509.17249].

This null-aware formulation is essential because direct document concatenation was found inadequate: a preliminary study using MetricX-24 and MetricX-24-QE showed limited sensitivity to injected under- and over-translation even when token limits were not violated [2509.17249]. By contrast, SEGALE converts these failure modes into explicit alignment structure, after which any local metric can penalize them through null spans [2509.17249].

In the speech-to-speech setting, the same alignment outputs support both quality and latency. For each aligned group $A_k=(X_k,Y_k,\hat{Y}_k)$, sentence-level quality is computed as
$$
q_k=Q(X_k,Y_k,\hat{Y}_k),
$$
with $Q$ instantiated as **xCOMET-XL** in the reported experiments, while null groups receive the metric minimum $Q_{\min}$ [2606.15059]. This makes SEGALE compatible with both reference-based and quality-estimation-style scorers, provided they can operate on aligned local spans [2509.17249][2606.15059].

## 4. Empirical evaluation on long-form machine translation

SEGALE was meta-evaluated primarily on the WMT 2024 Metrics Shared Task for language pairs en-de, en-es, and ja-zh, using document-level Kendall’s $\tau$ against document-aggregated MQM z-scores [2509.17249]. The evaluation included the original data and three synthetic anomaly settings: **over-translate**, created by randomly removing 10% of sentences from source and reference; **under-translate**, created by removing 10% from system translations; and **flex-boundary**, created by merging 10% of adjacent source sentences using GPT-4o while preserving semantics with a BLEURT $>0.85$ filter [2509.17249].

On the original setting, SEGALE achieved correlations close to gold sentence alignments. Averaged over the three language pairs, the reported scores were: COMET $0.3085$, COMET-QE $0.2768$, MetricX $0.3074$, and MetricX-QE $0.2630$, compared with gold-alignment values of COMET $0.3110$, COMET-QE $0.2800$, MetricX $0.3085$, and MetricX-QE $0.2683$ [2509.17249]. The null-alignment ratio for SEGALE in this setting was $0.7\%$, whereas gold had $0\%$ [2509.17249].

Under explicit perturbation, SEGALE retained substantially stronger behavior than mwerSegmenter. In the **over-translate** setting, SEGALE reported NA $=11.2\%$, close to the gold value of $10\%$, while mwerSegmenter remained at $0\%$ and therefore failed to flag hallucinated content as null structure [2509.17249]. In the **under-translate** setting, SEGALE reported NA $=5.8\%$ versus gold $10\%$, whereas mwerSegmenter reported only $2.1\%$ and exhibited a marked drop in correlation, including COMET $0.2183$ and MetricX $0.1770$ [2509.17249]. In the **flex-boundary** setting, SEGALE remained close to gold with minimal NA inflation, showing that many-to-many alignment absorbs moderate sentence-boundary disagreement without major degradation [2509.17249].

The ablation results further identify the fine-tuned BGE-M3 encoder as consistently better than LASER and untuned BGE-M3 in both correlation and NA-ratio behavior, while the segmenter choice between ersatz and spaCy had only minor impact [2509.17249]. A plausible implication is that alignment quality is limited more by cross-lingual semantic matching under granularity mismatch than by the initial sentence splitter, provided the splitter does not produce pathological boundaries.

SEGALE was also applied to book-length translation in the WMT 2024 discourse-level literary translation setting for zh→en. Using segment sizes of 1k, 2k, 4k, and 8k tokens, it revealed that many open-weight LLMs degrade sharply at 4k–8k context, with document-level MetricX dropping and null-alignment ratio rising markedly [2509.17249]. The paper highlights EuroLLM as summarizing instead of translating at 4k, whereas Qwen2.5-72B-Instruct remained comparatively robust, with stable or slightly improved quality up to 4k [2509.17249].

## 5. Adaptation to long-form simultaneous speech-to-speech translation

In long-form SimulS2ST, SEGALE is embedded in a broader evaluation pipeline that operates on generated target speech rather than exposed target text [2606.15059]. The pipeline begins with target-side ASR and forced alignment. Specifically, generated target speech $\hat{Y}$ is chunked into consecutive 180-second segments, transcribed with **Qwen3-ASR-1.7B**, and token-aligned with **Qwen3-ForcedAligner-0.6B** to obtain target text $\hat{T}=(\hat{y}_1,\dots,\hat{y}_{|\hat{T}|})$ and token-level end timestamps $D=(d_1,\dots,d_{|\hat{T}|})$ [2606.15059]. Chunk-relative timestamps are then mapped back to the global timeline and concatenated [2606.15059]. Sentence boundary detection is performed with spaCy, and SEGALE aligns the recovered target sentences to source sentences and reference translations [2606.15059].

This alignment supports YAAL-style latency computation over sentence groups. For an aligned group $A_k=(X_k,Y_k,\hat{Y}_k)$ spanning source times $T_k^s$ to $T_k^e$ and target token end times $D_k=(d_1,\dots,d_{|\hat{Y}_k|})$, the ideal schedule is
$$
d_i^*=T_k^s+(i-1)\cdot\frac{T_k^e-T_k^s}{\max\{|Y_k|,|\hat{Y}_k|\}},
$$
and the group latency is
$$
l_k=\frac{1}{|\hat{Y}_k|}\sum_{i=1}^{|\hat{Y}_k|}(d_i-d_i^*).
$$
Groups with no source-target content are excluded from latency, and target tokens generated after the end of the full source stream are also excluded, following LongYAAL-style practice [2606.15059].

The final reported system-level latency and quality are simple averages over aligned groups [2606.15059]. On ACL 60/60 dev, the reported latency/xCOMET-XL pairs were: **SeamlessStreaming** En→De $4.333/67.56$, En→Ja $2.434/42.89$, En→Zh $1.725/40.66$; and **Seed LiveInterpret 2.0** En→De $7.939/85.39$, En→Ja $9.413/45.48$, En→Zh $5.306/72.78$ [2606.15059]. On Audio-NTREX-L, the reported values were: **SeamlessStreaming** Fr→En $3.520/77.50$, De→En $3.833/78.95$, Pt→En $3.566/76.11$, Es→En $3.608/77.88$; **Seed LiveInterpret 2.0** Fr→En $5.892/86.67$, De→En $5.933/88.63$, Pt→En $5.530/86.94$, Es→En $5.592/88.63$; and **Hibiki-Zero** Fr→En $3.271/80.21$, De→En $3.313/79.50$, Pt→En $3.312/79.00$, Es→En $3.657/81.39$ [2606.15059].

A key empirical result is that SEGALE achieved **90.9% segmentation accuracy** versus **SoftSegmenter’s 79.1%** on ACL 60/60 En→Zh, which the paper attributes to explicit boundary modeling and robust semantic alignment [2606.15059]. The same experiments also revealed pronounced latency accumulation on long talks: ending offsets of aligned sentences tend to grow over time, with especially strong accumulation for En→Ja, where target speech is longer than source and offsets can exceed 200 seconds [2606.15059]. This suggests that long-form simultaneous speech evaluation requires alignment methods that remain robust under drift, missing content, and variable sentence duration rather than assuming pre-segmented utterances.

## 6. Limitations, interpretive caveats, and unrelated usage in LEGEND-200

SEGALE inherits several limitations from its components. In machine translation, the underlying metrics remain sentence-level and do not themselves model discourse; SEGALE extends them to long documents by alignment and aggregation rather than by introducing new discourse-sensitive semantics [2509.17249]. Its adaptive skip-penalty search is tuned to current metric behavior and to Vecalign’s explicit null-penalty framework, so the documented mechanism is not readily portable to aligners without comparable skip controls [2509.17249]. The embedding fine-tuning is described as proof-of-concept on limited data and languages, implying that broader language coverage would likely improve robustness [2509.17249].

In speech-to-speech translation, the evaluation depends on target-side ASR and forced alignment accuracy. The reported target-side ASR comparison on ACL 60/60 dev gave Qwen3-ASR-1.7B versus WhisperX WER/CER values of De: $15.37$ vs $13.80$, Ja: $27.60$ vs $27.30$, and Zh: $4.50$ vs $5.52$ [2606.15059]. The paper notes that the high Japanese error rates suggest quality problems in the synthesized Japanese speech rather than a difference driven by ASR choice [2606.15059]. This indicates that SEGALE can expose system-level pathologies, but it cannot by itself separate synthesis errors from translation-policy errors.

The term “SEGALE” also appears in an unrelated nuclear-instrumentation context. The paper “Signal Readout Electronics for LEGEND-200” describes the **LEGEND-200 signal readout electronics** as a low-mass, low-background, two-stage charge-sensitive amplifier operated inside liquid argon, with a first-stage **Low Mass Front End** near each HPGe detector and a second-stage differential amplifier located approximately $30$–$150$ cm away [1911.05847]. The data explicitly note that the acronym **SEGALE is not explicitly used or defined** in that paper, even though it may be used informally to refer to the readout electronics [1911.05847]. Because this usage is separate from the MT and SimulS2ST literature, the two meanings should not be conflated.

Overall, the dominant technical meaning of SEGALE in current arXiv literature is the long-form alignment-and-evaluation framework for machine translation and its reuse as the sentence-embedding-based aligner in long-form simultaneous speech-to-speech translation [2509.17249][2606.15059].

Source: https://www.emergentmind.com/topics/segale