Sign3D-WLASL: 3D Dataset & Recognition

• Sign3D-WLASL is a 3D keypoint‐augmented derivative of the WLASL corpus, enabling motion synthesis for realistic speech-to-sign animation.
• The approach applies dual-stream spatio-temporal modeling to disentangle hand morphology from motion trajectory for improved recognition.
• It bridges animation and recognition by offering a structured skeletal representation that enhances temporal modeling and addresses data scarcity.

Searching arXiv for the cited works and topic variants to ground the article. I’m going to look up the relevant arXiv entries for “Sign3D-WLASL” and the associated papers. Sign3D-WLASL is a label used in recent sign-language research for 3D or skeleton-centric reformulations of the Word-Level ASL (WLASL) corpus. In one explicit usage, it denotes a 3D keypoint-augmented derivative of WLASL created for motion synthesis and speech-to-sign animation; in other works, it denotes applying skeleton-based recognition architectures to WLASL with 3D skeletal inputs, or, more loosely, a structural WLASL perspective centered on pose rather than RGB video (Rahman et al., 9 Jul 2025, Liu et al., 10 Sep 2025, Saad, 11 Dec 2025). Across these usages, the shared objective is to expose body, hand, and facial kinematics in a representation that is more directly usable for animation, graph-based recognition, metric learning, and geometry-aware temporal modeling than RGB-only video.

1. Terminological scope and relation to WLASL

The term does not designate a single uniformly standardized resource. Rather, the literature uses it in several related senses tied to WLASL-derived skeletal structure.

Usage	Description	Representative source
Dataset sense	3D keypoint-augmented derivative of WLASL for animation	(Rahman et al., 9 Jul 2025)
Recognition sense	DSLNet applied to 3D skeletal inputs on WLASL-100 and WLASL-300	(Liu et al., 10 Sep 2025)
Broader structural sense	Skeleton-centric WLASL representation, sometimes 2D but treated as 3D-compatible	(Saad, 11 Dec 2025)

In the dataset-centered formulation, Sign3D-WLASL is explicitly defined as a 3D keypoint-augmented derivative of WLASL, created because the original WLASL contains isolated ASL word videos but no skeletal motion annotations, which limits its direct use for 3D sign production (Rahman et al., 9 Jul 2025). In the recognition-centered formulation, the same label is used for applying a dual-stream spatio-temporal dynamic graph convolutional network to 3D skeletons on WLASL-100 and WLASL-300, with the central goal of resolving geometric ambiguity between hand morphology and motion trajectory (Liu et al., 10 Sep 2025). In the broader metric-learning formulation, the term functions as a perspective on WLASL as a skeleton-based recognition problem under data scarcity and long-tail class imbalance, even though the reported experiments use 2D pose rather than true 3D lifting (Saad, 11 Dec 2025).

This suggests that “Sign3D-WLASL” is best understood as a research nexus around skeletal WLASL representations, rather than as a single benchmark artifact with one canonical schema or protocol.

2. Sign3D-WLASL as a 3D keypoint dataset

In the most concrete dataset sense, Sign3D-WLASL is built from WLASL videos of isolated ASL words, signed clearly by native signers, and the paper states that “WLASL includes 1,983 carefully curated videos” used to construct the keypoint dataset (Rahman et al., 9 Jul 2025). Its purpose is to enrich RGB-only isolated-sign videos with 3D skeletal data so that they become suitable for motion synthesis and animation.

The construction pipeline is explicitly multi-model. RTMPose3D with RTMDet predicts full-body 3D keypoints using knowledge learned from the COCO WholeBody dataset; a ResNet50-based model trained on H3WB focuses on detailed upper-body and hand recognition; OpenPifPaf performs bottom-up multi-person 2D keypoint detection and JointFormer lifts those keypoints to 3D using spatiotemporal reasoning and refinement; MediaPipe contributes lightweight real-time facial, pose, and hand landmarks (Rahman et al., 9 Jul 2025). This ensemble is used to improve robustness under varying lighting, angles, and signer appearance.

The resulting representation contains 133 anatomical keypoints covering the upper body, facial expressions, and hand articulations. The paper does not specify the per-part counts or exact landmark names, but it states that the use of COCO WholeBody implies alignment to its 133-keypoint schema (Rahman et al., 9 Jul 2025). Temporal extraction is sparse rather than frame-dense: frames are sampled at regular intervals, such as every 4th or 8th frame, to reduce data volume while preserving sign structure. The paper does not specify normalization, coordinate system, units, scaling, storage format, file organization, metadata schema, or train/validation/test splits (Rahman et al., 9 Jul 2025).

The representation is therefore defined more by its semantic coverage than by a published serialization standard. Body, hand, and face landmarks are jointly present, and 3D structure is obtained through a combination of direct estimation and 2D-to-3D lifting. This makes the dataset usable as a motion prior, but also means that its 3D geometry is estimator-derived rather than motion-capture-ground-truth.

3. Role in speech-to-sign animation

Within Speak2Sign3D, Sign3D-WLASL is the motion backbone that turns translated ASL gloss into lifelike 3D sign animations (Rahman et al., 9 Jul 2025). The end-to-end pipeline begins with Whisper for speech-to-text, then applies MarianMT for English-to-ASL gloss translation using BookGlossCorpus-CG, a parallel corpus of approximately $1.3$ million English sentences converted into ASL gloss via rule-based grammar. Word2Vec and FastText are used to improve semantic understanding, and a separate word-mapping stage uses a BERT tokenizer and cosine similarity against common ASL glosses to normalize vocabulary (Rahman et al., 9 Jul 2025).

The animation stage maps predicted gloss tokens to stored sign motions represented as 3D keypoint sequences, then concatenates and smooths them. MarianMT is reported with BLEU-1 $= 0.7714$ and BLEU-2 $= 0.8923$, where BLEU is defined as

$$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$

These scores quantify n-gram overlap between predicted gloss and reference gloss in the grammar-aware translation stage (Rahman et al., 9 Jul 2025).

Temporal continuity is enforced with cubic spline interpolation. For time knots $t_0 < t_1 < \dots < t_n$ and 3D points $y_k$, the spline on $[t_k, t_{k+1}]$ is

$$S_k(t) = a_k + b_k(t-t_k) + c_k(t-t_k)^2 + d_k(t-t_k)^3,$$

with continuity of position and first and second derivatives across knots, and natural boundary conditions such as $S''(t_0)=S''(t_n)=0$ (Rahman et al., 9 Jul 2025). In the reported evaluation, Mean Per Joint Position Error is used,

$$\mathrm{MPJPE} = \frac{1}{M}\sum_{j=1}^{M}\sqrt{(\hat{x}_j-x_j)^2+(\hat{y}_j-y_j)^2+(\hat{z}_j-z_j)^2},$$

and cubic interpolation achieves $= 0.7714$0, slightly lower than the baseline LSTM value of $= 0.7714$1 (Rahman et al., 9 Jul 2025).

This usage situates Sign3D-WLASL less as a recognition benchmark than as an intermediate motion lexicon: isolated WLASL signs become reusable 3D motion clips that can be composed into continuous ASL animation.

4. Recognition-oriented Sign3D-WLASL: dual-reference 3D skeleton modeling

In recognition work, Sign3D-WLASL has been used to denote 3D skeleton-based isolated sign language recognition on WLASL through DSLNet, a dual-stream spatio-temporal dynamic graph convolutional network (Liu et al., 10 Sep 2025). The motivating claim is that many isolated signs are morphologically similar yet semantically distinct because their meaning depends jointly on hand shape and on motion relative to the face or body. A single reference frame is described as insufficient because it cannot simultaneously deliver viewpoint invariance for morphology and semantic spatial sensitivity for trajectory.

DSLNet resolves this by decoupling hand morphology and motion trajectory into complementary frames. For hand joints $= 0.7714$2 with wrist $= 0.7714$3, the wrist-centric shape frame is

$= 0.7714$4

optionally scale-normalized by a hand-span term. For facial landmarks $= 0.7714$5, with face centroid $= 0.7714$6 and scale $= 0.7714$7, the face-centric trajectory frame is

$= 0.7714$8

The shape stream then applies topology-aware spatio-temporal dynamic graph convolution, temporal convolution, BiLSTM, and multi-head attention to produce a morphological descriptor, while the trajectory stream applies a Finsler geometry-based encoder in which a learnable Finsler metric emphasizes direction-sensitive motion events (Liu et al., 10 Sep 2025).

Fusion is handled by cross-attention followed by entropic optimal transport. With a cosine-distance cost between projected morphology and trajectory features, a Sinkhorn-solved coupling aligns the global morphology node with temporally distributed trajectory evidence before an MLP classifier predicts the class (Liu et al., 10 Sep 2025). Training uses a cross-entropy classification term plus a geometric consistency loss,

$= 0.7714$9

The reported input modality uses 21 hand keypoints plus 5 facial keypoints from MediaPipe, normalized to $= 0.8923$0, and the approach is defined in $= 0.8923$1. Parameters are $= 0.8923$2M, FLOPs are $= 0.8923$3G, and average inference is $= 0.8923$4 ms per sample on an RTX 4090 (Liu et al., 10 Sep 2025). On standard isolated recognition, DSLNet reports $= 0.8923$5 on WLASL-100, $= 0.8923$6 on WLASL-300, and $= 0.8923$7 on LSA64. Ablations indicate that the dual-reference design improves over global-only, wrist-only, and face-only variants, and that geometry-driven OT fusion outperforms simple concatenation and cross-attention alone (Liu et al., 10 Sep 2025).

In this formulation, Sign3D-WLASL is not principally a dataset-construction exercise; it is a geometric modeling problem over structured 3D sign skeletons.

5. Comparative methodological landscape

Several neighboring lines of work clarify what Sign3D-WLASL includes and what it does not include. P3D uses an “expressive 3D pose” consisting of 2D positional pose, 3D positional pose, 3D rotational pose, and a 10-dimensional facial expression vector, and alternates a part-wise encoding Transformer with a whole-body encoding Transformer. On WLASL, P3D reports $= 0.8923$8 Top-1 on WLASL100, $= 0.8923$9 on WLASL300, and $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$0 on WLASL2000 using 2D+3D pose, rising to $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$1, $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$2, and $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$3 when face expression is included; early fusion is both more accurate and more efficient than middle or late fusion, at $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$4 GFLOPs and $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$5M parameters (Lee et al., 2023). LA-Sign, by contrast, does not use 3D on WLASL: it operates on 2D skeletons with confidence from RTM-Pose, partitioned into left hand, right hand, upper body, and face, then processed by part-wise ST-GCN encoders and a looped mT5-initialized transformer with geometry-aware hyperbolic alignment. It reports $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$6 per-instance and $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$7 per-class Top-1 on WLASL2000, and $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$8 per-instance and $$\mathrm{BLEU} = \mathrm{BP} \times \exp\!\left(\sum_{n=1}^{N} w_n \log p_n\right).$$9 per-class on WLASL300, while remaining explicitly compatible with 3D inputs through channel substitution and joint remapping (Pu et al., 30 Mar 2026).

Other methods broaden the space of skeletal WLASL formulations. A few-shot prototypical network with an ST-GCN backbone and Multi-Scale Temporal Aggregation uses whole-body 2D pose from RTMPose-l and episodic metric learning rather than fixed classifier weights; it reports $t_0 < t_1 < \dots < t_n$0 Top-1, $t_0 < t_1 < \dots < t_n$1 Top-5, and $t_0 < t_1 < \dots < t_n$2 Top-10 on WLASL test, outperforming a matched standard classifier by $t_0 < t_1 < \dots < t_n$3 Top-1 points, and reaches $t_0 < t_1 < \dots < t_n$4 Top-1 in zero-shot transfer to SignASL without fine-tuning (Saad, 11 Dec 2025). TSSI-based recognition converts skeleton sequences into a Tree Structure Skeleton Image $t_0 < t_1 < \dots < t_n$5, where columns follow a depth-first traversal of a skeleton tree and channels encode $t_0 < t_1 < \dots < t_n$6; on WLASL-100, SL-TSSI-DenseNet reaches $t_0 < t_1 < \dots < t_n$7 Top-1 with augmentation, but the WLASL experiments effectively remain 2D because the MediaPipe $t_0 < t_1 < \dots < t_n$8 channel is set to zero due to unreliability (Laines et al., 2023).

Adjacent work further shows that skeletal WLASL can be enriched by complementary cues rather than by 3D alone. MSNN combines a global I3D stream, local hand and face I3D streams, and a 2D ST-GCN skeleton stream, and reports $t_0 < t_1 < \dots < t_n$9 Top-1 on WLASL100, $y_k$0 on WLASL300, $y_k$1 on WLASL1000, and $y_k$2 on WLASL2000 (Maruyama et al., 2021). Phonology-aware multitask training adds auxiliary prediction heads for ASL phonological features such as handshape and minor location; for SL-GCN on WLASL2000, Top-1 increases from $y_k$3 to $y_k$4, with corresponding gains in Top-3 and MRR (Kezar et al., 2023). At the efficiency extreme, a bidirectional reservoir-computing system using MediaPipe hand landmarks reports $y_k$5 Top-1 on WLASL100 with a CPU training time of $y_k$6 seconds, compared with $y_k$7 minutes and $y_k$8 seconds for a Bi-GRU baseline (Singh et al., 30 Nov 2025).

Taken together, these results show that Sign3D-WLASL is methodologically heterogeneous. Some systems depend on explicit 3D geometry, some are 2D but structurally compatible with 3D, and some obtain large gains through linguistic supervision, local appearance streams, or metric-learning objectives rather than through depth modeling alone.

6. Limitations, misconceptions, and open directions

A common misconception is that Sign3D-WLASL always denotes a single public 3D benchmark with fixed storage, metadata, joint schema, and evaluation protocol. The literature summarized here does not support that view. The Speak2Sign3D dataset paper defines a specific 3D keypoint derivative of WLASL but does not specify file formats, coordinate systems, units, scaling, retargeting details, signer counts, or train/validation/test splits (Rahman et al., 9 Jul 2025). Recognition papers often use their own skeletal schemas, preprocessing, and evaluation conventions, and some related works remain 2D or pseudo-3D even when discussed as Sign3D-compatible (Laines et al., 2023, Singh et al., 30 Nov 2025, Pu et al., 30 Mar 2026).

Technical limitations are also recurrent. In the animation-oriented dataset, all 3D data are inferred by pose estimators and lifting because WLASL lacks annotated motion; downsampling every 4th or 8th frame introduces temporal gaps that must be repaired by interpolation, and the small MPJPE difference between an LSTM baseline and cubic interpolation indicates sensitivity to smoothing choices (Rahman et al., 9 Jul 2025). In DSLNet, extremely subtle finger shapes with heavy occlusion or rapid non-manual cues beyond the current facial subset may still cause errors; the paper identifies richer facial landmarks and explicit 3D rotation alignment as plausible improvements (Liu et al., 10 Sep 2025). In few-shot recognition, long-tail class imbalance remains fundamental, and even prototype-based metric learning still struggles under extreme scarcity, near-synonyms, and kinematic homophenes (Saad, 11 Dec 2025).

The broader literature indicates that future Sign3D-WLASL work is likely to revolve around three pressures. First, richer non-manual modeling remains necessary, since hands alone do not capture the full semantic load of ASL, and several papers explicitly note the value of face, mouth, and phonological structure (Kezar et al., 2023, Maruyama et al., 2021). Second, 3D consistency remains unresolved: some systems use true lifted or estimated 3D pose, some use normalized image-space pseudo-3D, and others remain purely 2D while advertising straightforward 3D extensibility (Singh et al., 30 Nov 2025, Pu et al., 30 Mar 2026). Third, applications diverge between recognition and production. The animation line emphasizes gloss-to-motion continuity, retargetability, and realistic kinematics, whereas the recognition line emphasizes discriminative geometry, robustness, and long-tail generalization (Rahman et al., 9 Jul 2025, Liu et al., 10 Sep 2025).

Sign3D-WLASL therefore occupies a transitional position in the WLASL ecosystem. It is simultaneously a concrete 3D keypoint dataset for speech-to-sign animation, a recognition setting for geometric skeleton models, and a broader conceptual shift from RGB video toward structured sign kinematics.