iLSU-T: Uruguayan SLT Dataset

• iLSU-T is a curated, multimodal Uruguayan sign language dataset that enables gloss-free video-to-text translation research.
• It offers over 187 hours of RGB interpreter videos with Spanish audio and subtitle-derived transcriptions, supporting studies on clip-level translation and signer variability.
• The episode-centric structure and diverse source conditions facilitate research on alignment, domain shifts, and discourse variation in realistic broadcast environments.

Searching arXiv for the iLSU-T paper and directly related sign language translation references. I’ll look up the iLSU-T paper and a few related SLT datasets/methods on arXiv to ground the article in current literature. iLSU-T, short for Lengua de Señas Uruguaya – Translation, is an open, curated dataset built from public Uruguayan TV broadcasts for research on automatic sign language translation and related tasks in Uruguayan Sign Language (LSU). It provides multimodal data consisting of RGB video of professional LSU interpreters, the original Spanish audio, and Spanish text transcriptions derived from the audio. The dataset is intended for gloss-free sign language translation, meaning that it does not include gloss annotations; instead, sentence-level Spanish text is aligned to video segments. In total, iLSU-T comprises more than 185 hours of interpreted sign language videos from public TV broadcasting, spans broadcast news and institutional sessions including the Uruguayan Parliament, covers diverse topics, and involves 18 professional interpreters (Stassi et al., 7 Jul 2025).

1. Scope, identity, and target task

iLSU-T is designed around a specific formulation of sign language processing: gloss-free sign language translation from LSU video to Spanish text. In this setting, the primary input is RGB video cropped to the interpreter’s Region of Interest (RoI), while the target output is Spanish text derived from the audio channel. The audio itself is also preserved, making the corpus multimodal rather than purely vision-text. This design distinguishes iLSU-T from corpora centered on gloss supervision or laboratory capture, and it places the dataset directly in the line of work that attempts end-to-end translation without an intermediate symbolic annotation layer (Stassi et al., 7 Jul 2025).

The corpus is organized as 571 episodes, each containing exactly one interpreter, with an average episode length of 19.7 minutes. Episodes are defined by the temporal boundaries of the interpreter’s on-screen presence or substitution, and each episode has a unique text ID encoding media source, source file, time range, and signer ID. This episode-centric structure is important because signer identity, source conditions, and text timelines are all episode-specific. A plausible implication is that iLSU-T supports both clip-level SLT experiments and broader studies of signer variation, source-dependent domain shift, and timeline alignment under real broadcast conditions.

The dataset scale is reported at 187.4 hours of LSU-interpreted RGB video and 86,550 automatically generated video clips with overlapping, for an aggregate clip duration of 201.52 hours. The average clip duration is 8.38 seconds with standard deviation 5.95. Across the whole dataset, the vocabulary size is 37k9, approximately 37,900 unique Spanish words (Stassi et al., 7 Jul 2025).

2. Modalities, sources, and linguistic coverage

The three modalities are explicitly defined. The video modality consists of RGB video of LSU interpreters, cropped to the interpreter’s RoI. The audio modality is Spanish and is used to produce text transcriptions with timestamps. The text modality consists of Spanish subtitles and sentence-level transcriptions that are subtitle-derived from the audio. No gloss labels are provided (Stassi et al., 7 Jul 2025).

The data comes from three public sources: Canal 5, TV Ciudad, and Uruguayan Parliament sessions. These sources are not merely alternative collection channels; they also induce distinct topical, visual, and interactional regimes. Source 1 and Source 2 correspond to public television, whereas Source 3 corresponds to parliamentary sessions. The dataset notes that interpreters are mutually exclusive across the three media sources, so the source split is also a signer split at the corpus level (Stassi et al., 7 Jul 2025).

The topical range is broad. Expert-labeled aligned parts include weather, traffic, health, human rights, politics, social, culture, news, security, laws and regulations, sports, and shows. The discourse genres include greetings and politeness formulae, reports, interviews, anecdotes and narratives, legal and normative procedures, debate and discussion, and argumentation. This is consequential for modeling because gloss-free SLT systems trained on iLSU-T are exposed not only to lexical diversity but also to substantial discourse-structural variation. This suggests that the dataset is relevant for topic-conditioned and genre-conditioned SLT, not solely for sentence translation.

The per-source profile is heterogeneous:

Subset	Duration, vocabulary, signers	Frame rate and typical RoI
Whole dataset	187.4 h; 37k9; 18 signers	343.2 ± 46.6 px width; 363.7 ± 60.5 px height
Source 1	18.1 h; 12k3; 1 signer	25 fps; 331.9 ± 2.7 by 312.9 ± 2.6
Source 2	22.4 h; 14k1; 5 signers	25 fps; 246.6 ± 11.8 by 240.1 ± 3.9
Source 3	146.9 h; 29k9; 12 signers	25 and 30 fps in approximately 3:7 ratio; 362.2 ± 27.8 by 393.2 ± 29.0

The frame-rate asymmetry is concentrated in Source 3, which mixes 25 fps and 30 fps. The paper reports that results are robust to this mixture in the tested setting, but the source-level differences in RoI size, scale, and background remain a structural characteristic of the corpus (Stassi et al., 7 Jul 2025).

3. Collection pipeline, alignment strategy, and annotation design

iLSU-T was derived through a five-stage curation pipeline. First, RoI identification was performed by manually labeling the interpreter bounding box with $(x,y)$ coordinates of the upper-left and lower-right corners for each raw video, using one RoI per video file. The dataset notes that interpreter scale and background vary across sources and episodes. Second, signer recognition was performed with a KNN-based face classifier derived from the ageitgey/face_recognition example, trained with 50 face samples per signer. Frames were sampled uniformly at 1 fps, and a median filter post-process enforced a minimum 30-second signer duration. Episode boundaries were then verified and refined by visual inspection so that each episode contains a single signer (Stassi et al., 7 Jul 2025).

Third, automatic captioning was performed with WhisperX (large-v3), which transcribes Spanish audio and outputs sentence segmentation with word-level timestamps. Each episode text track has its own independent timeline. Fourth, manual alignment was carried out by expert linguists, who aligned LSU video phrases to Spanish text while being guided by interpreting pauses and epenthesis, described as transition movements between signs or to or from resting position. These cues permit segmentation into phrase-like units without changing meaning. More than 20 hours of video have expert-produced ground-truth sentence-level alignment between video and text, and these manually aligned episodes are available from all three sources. Fifth, the manually aligned episodes were annotated for topic and discourse genre (Stassi et al., 7 Jul 2025).

For SLT training, clips are automatically generated around each text phrase using random pre- and post-delays to accommodate interpreter lag relative to speech:

$t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$

with

$$[a_1,b_1,a_2,b_2] = [0.4\,s,\,1.2\,s,\,2.1\,s,\,2.9\,s].$$

This procedure yields 86,550 clips. Clips can overlap, and the text content per clip is considered independent even when consecutive clips overlap (Stassi et al., 7 Jul 2025).

The annotation design is therefore asymmetric. Temporal alignment quality is highest in the manually aligned subset, whereas the majority of the corpus relies on heuristic clipping around ASR-derived phrase boundaries. A plausible implication is that iLSU-T simultaneously functions as a dataset for translation and as a benchmark for alignment-sensitive methods, including segmentation and phrase-boundary modeling.

4. Experimental protocols, feature extraction, and baseline systems

The paper evaluates three state-of-the-art gloss-free SLT approaches: SLT (Sign Language Transformers; Camgoz et al., 2020), STLCU (Stochastic Transformer Networks with Linear Competing Units; Voskou et al., 2021), and GASLT (Gloss Attention for Gloss-free SLT; Yin et al., 2023) (Stassi et al., 7 Jul 2025). In the iLSU-T implementation of SLT, gloss loss is disabled because the dataset has no gloss supervision, so the joint loss is

$L = \lambda_R L_R + \lambda_T L_T,$

with

$\lambda_R = 0, \qquad \lambda_T = 1.$

Visual inputs are based on I3D features extracted via TSPNet sliding windows with window width 8 frames and stride 2 frames. Two frozen pretrained backbones are used: I3D-ASL2k, pretrained on WLASL’s 2000 isolated ASL signs, and I3D-BSL5k, an M+D+A model pretrained on BSL with 5,383 words. Frames are resized to $224 \times 224$ before feature extraction (Stassi et al., 7 Jul 2025).

The study defines four configurations: the whole dataset, and Source 1, Source 2, and Source 3 subsets. The default splits are random clip-level train/dev/test partitions at $0.8/0.1/0.1$ per configuration. The preprocessing pipeline also supports controlled experiments such as signer-disjoint splits or topic and genre filters. The authors explicitly recommend monitoring phrase duplication across train and test in order to avoid inflated scores from repeated phrases (Stassi et al., 7 Jul 2025).

Training runs for up to 100 epochs with batch size 128, except GASLT on the whole dataset, which uses batch size 64 due to RAM limits. For GASLT, cosine sentence similarity matrices are built per dataset following the official approach, and for large splits the matrix is computed in parts and reconstructed internally. The paper further notes that on iLSU-T, replacing BPE with word encoding improved GASLT performance (Stassi et al., 7 Jul 2025).

Evaluation uses BLEU-$N$ for $N=1,\dots,4$ and ROUGE-L on development and test sets, with SacreBLEU used for BLEU reproducibility. The definitions are explicitly given. For ROUGE-L, if $X$ is the reference of length $t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$0, $t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$1 the candidate of length $t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$2, and $t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$3 their longest common subsequence, then

$t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$4

and

$t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$5

with $t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$6. BLEU-$t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$7 is defined by modified precision

$t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$8

where

$t_1 \sim U(a_1,b_1), \qquad t_2 \sim U(a_2,b_2),$9

and

$$[a_1,b_1,a_2,b_2] = [0.4\,s,\,1.2\,s,\,2.1\,s,\,2.9\,s].$$0

with $$[a_1,b_1,a_2,b_2] = [0.4\,s,\,1.2\,s,\,2.1\,s,\,2.9\,s].$$1 and brevity penalty

$$[a_1,b_1,a_2,b_2] = [0.4\,s,\,1.2\,s,\,2.1\,s,\,2.9\,s].$$2

These metric definitions matter because iLSU-T includes short utterances, recurrent formulae, and overlapping clips, all of which can materially affect n-gram-based scores (Stassi et al., 7 Jul 2025).

5. Baseline performance and empirical behavior

On the whole-dataset test split, the best overall result is obtained by STLCU with I3D-ASL2k features, reaching BLEU-1 17.69, BLEU-2 8.17, BLEU-3 4.92, BLEU-4 3.43, and ROUGE-L 14.86. The second-best whole-dataset result reported in the summary is GASLT with I3D-ASL2k, which achieves BLEU-4 1.29 and ROUGE-L 11.57, while SLT with I3D-BSL5k reaches BLEU-4 1.24 and ROUGE-L 11.57 (Stassi et al., 7 Jul 2025).

The source-specific results show distinct difficulty profiles:

Configuration	Best reported test result	Additional note
Whole dataset	STLCU + I3D-ASL2k: BLEU-4 3.43, ROUGE-L 14.86	Best overall on whole
Source 3	STLCU + I3D-ASL2k: BLEU-4 3.82, ROUGE-L 16.05	Best on Parliament
Source 2	STLCU + I3D-ASL2k: BLEU-4 1.30	GASLT + I3D-ASL2k gives best ROUGE-L 10.74
Source 1	STLCU + I3D-BSL5k: BLEU-4 1.01	SLT + I3D-ASL2k gives best ROUGE-L 10.97

Source 3, the parliamentary subset, is the strongest source condition numerically, with STLCU + I3D-ASL2k reaching BLEU-4 3.82 and ROUGE-L 16.05 on test. GASLT + I3D-ASL2k is second-best there with BLEU-4 2.34 and ROUGE-L 14.80, followed by SLT + I3D-ASL2k with BLEU-4 2.26 and ROUGE-L 13.03. Source 2 is more difficult in BLEU-4 terms, where STLCU + I3D-ASL2k attains 1.30, while GASLT + I3D-ASL2k provides the best ROUGE-L at 10.74. Source 1 is lower still on BLEU-4, with STLCU + I3D-BSL5k at 1.01 and STLCU + I3D-ASL2k at 0.88, although SLT + I3D-ASL2k yields the best ROUGE-L of 10.97 (Stassi et al., 7 Jul 2025).

The qualitative analysis identifies an important structural factor: Source 3 contains frequent parliamentary formulas such as “se está votando” and “vamos a votar la solicitud de licencia...”. GASLT often produced near-exact variants of recurring phrases with high BLEU-1..4 and ROUGE-L, including “muchas gracias, señora presidenta.” with BLEU-4 77.88 and ROUGE-L 87.14. At the same time, multiple systems yielded semantically aligned outputs even when BLEU penalized short numeric phrases such as “24 en 24.” In such cases, BERTScore reflected higher semantic similarity than BLEU or ROUGE. This suggests that n-gram overlap metrics only partially capture translation quality on formulaic and short-span segments (Stassi et al., 7 Jul 2025).

The paper also reports that the models were robust to mixing 25 fps and 30 fps rates in Source 3. An explicit example is STLCU on the Source 3 test split, with BLEU-4 $$[a_1,b_1,a_2,b_2] = [0.4\,s,\,1.2\,s,\,2.1\,s,\,2.9\,s].$$3 under mixed 25/30 fps versus $$[a_1,b_1,a_2,b_2] = [0.4\,s,\,1.2\,s,\,2.1\,s,\,2.9\,s].$$4 using only 25 fps material. This does not eliminate frame-rate variability as a factor, but it indicates that, under the tested preprocessing and feature pipeline, frame-rate heterogeneity was not catastrophic (Stassi et al., 7 Jul 2025).

6. Position within sign language resources, limitations, and responsible use

iLSU-T is presented as a localized, gloss-free, multimodal LSU resource that complements several established datasets. The comparative context given in the paper includes PHOENIX-2014T, described as DGS-to-German with 10.5 hours, text and gloss, and TV domain; LSA-T, described as LSA-to-Spanish with 21.8 hours and 14,880 samples from the web, smaller but including keypoints and labels; CSL-Daily, described as a 23-hour Chinese Sign Language corpus with text and gloss in lab conditions; How2Sign, described as ASL-English with 79 hours in lab conditions; OpenASL, described as ASL-English with 288 hours from the web and emphasizing domain diversity and duplication effects; and BOBSL, described as BSL TV data at 1,467 hours with predominantly text annotations (Stassi et al., 7 Jul 2025).

Within that landscape, iLSU-T fills a gap for Uruguayan LSU by combining real broadcast-domain LSU video with Spanish audio and Spanish text while omitting gloss supervision. The dataset paper argues that localized datasets are essential because of language-specific lexicon, grammar, and cultural context. A plausible implication is that models trained on better-resourced sign languages or on English-target corpora cannot simply be transferred to LSU without significant domain and language mismatch.

Several limitations and sources of bias are explicitly documented. Interpreter variability is substantial: professional interpreters differ in strategies, including omissions, fingerspelling for proper names, and sign variants, and fingerspelling is not explicitly annotated. Subtitle-derived text inherits ASR errors from WhisperX and is affected by limited punctuation prediction, which can mis-segment or mislabel sentences and degrade clip alignment quality. The absence of glosses means that models must rely on video-text alignment and semantic-similarity approaches, and alignment remains an open problem. Source 3 contains protocolized, repeated phrases, so duplication across train and test can inflate metrics. Visual variability in background, interpreter scale, and frame rate is also present. Finally, only more than 20 hours have expert alignments; the majority of clips are obtained through heuristic random-delay clipping (Stassi et al., 7 Jul 2025).

These limitations also clarify several common misconceptions. “Open” does not mean unrestricted redistribution: the corpus is released under a restricted use license for research and educational purposes, with open but controlled access. “Gloss-free” does not mean annotation-free: the dataset includes sentence-level transcriptions, word-level timestamps from WhisperX, expert manual alignment for more than 20 hours, and topic and discourse-genre labels on the aligned subset. “Large-scale” does not imply that benchmark scores are immune to data leakage or formulaic overlap: the paper specifically recommends tracking phrase duplication across train and test and reporting split strategies transparently (Stassi et al., 7 Jul 2025).

The repository provides data, code, predefined configurations, and licensing information at https://github.com/ariel-e-stassi/iLSU-T. The FG 2025 paper is titled “iLSU-T: an Open Dataset for Uruguayan Sign Language Translation,” authored by Ariel E. Stassi, Yanina Boria, J. Matías Di Martino, and Gregory Randall, with Ariel E. Stassi listed as corresponding author. The tasks enabled by the resource include gloss-free SLT, sign spotting, segmentation and alignment research, multimodal MT leveraging audio and text, and topic or genre-conditioned SLT (Stassi et al., 7 Jul 2025).