EGSample: Example-based Gesture Sampling

• EGSample is a neural framework for generating full-body gestures from speech, incorporating zero-shot style control via short example motion clips.
• It integrates a speech encoder, style encoder, and conditional variational formulation to produce diverse and probabilistic gesture outputs.
• The architecture, tested on the ZeroEGGS dataset, achieves state-of-the-art performance in naturalness, style accuracy, and variability of generated gestures.

Searching arXiv for the primary paper and closely related gesture-generation context. EGSample, short for “Example-based Gesture Sampling,” is an implementation-oriented formulation of zero-shot, example-conditioned speech-driven gesture generation built on the ZeroEGGS framework. It is designed to generate full-body gesture motion from speech while controlling style from only a short example motion clip, including motion styles unseen during training. The system combines a speech encoder, a style encoder, a conditional variational formulation, and a gesture decoder so that style can be transferred, blended, or sampled probabilistically, yielding multiple plausible gestures for the same speech input (Ghorbani et al., 2022).

1. Conceptual definition and scope

EGSample is framed as a neural network system for speech-driven gesture generation with zero-shot style control by example. In this setting, speech is the primary conditioning signal, while a short reference motion clip provides style information. The defining claim is that style control can be achieved from an arbitrary short motion example, even when the motion style was unseen during training. This places EGSample within example-based and zero-shot style transfer paradigms rather than class-conditioned gesture synthesis (Ghorbani et al., 2022).

The approach also treats gesture generation as intrinsically stochastic. Rather than producing a single deterministic output for a given utterance, it uses a probabilistic latent variable to represent variability in gesture motion. This supports multiple distinct outputs for the same speech and style conditions. The underlying paper further states that the model’s variational structure makes style embedding manipulation straightforward, including latent-space modification, blending, and scaling of style embeddings (Ghorbani et al., 2022).

A plausible implication is that EGSample is intended not only as a reproduction recipe for ZeroEGGS, but also as a practical basis for extending example-conditioned gesture generation to new styles or modalities. The data block explicitly presents it in that role by specifying architecture, losses, preprocessing, hyperparameters, inference, and evaluation in implementation-oriented form.

2. Architectural organization

The model is organized as a left-to-right pipeline: speech $x$ is encoded into frame-level speech features $f_x$, a style example $s$ is encoded into a style embedding $e_s$, a conditional variational core defines posterior and prior distributions over a latent variable $z$, and a gesture decoder generates the motion sequence $\hat y$ conditioned on $f_x$ and $z$ (Ghorbani et al., 2022).

Component	Input	Configuration
Speech Encoder	Log-mel spectrogram frames, $T_s \times 80$	3 layers 1D CNN, channels $[64,128,256]$, kernel $f_x$0, stride $f_x$1; 2-layer Bi-LSTM, hidden $f_x$2 per direction; linear projection to $f_x$3
Style Encoder	Example motion clip as joint-angle trajectories	$f_x$4 1D conv, channels $f_x$5, kernel $f_x$6, stride $f_x$7; global average pooling; MLP $f_x$8 with ReLU to $f_x$9
Gesture Decoder	$s$0 and global latent $s$1	Concatenate $s$2; 2-layer LSTM, hidden $s$3; MLP head $s$4

The speech encoder takes log-mel spectrogram frames computed with 80 mel bins, a 25 ms window, and a 10 ms hop. After temporal convolution and bidirectional recurrence, it produces $s$5 with $s$6. The style encoder operates on an example motion clip represented as joint-angle trajectories for $s$7 joints, each with 3D rotation, giving per-frame dimension $s$8. The clip length is approximately 2 seconds, or 120 frames at 60 Hz (Ghorbani et al., 2022).

The gesture decoder receives the per-frame speech features together with a global latent code. At each time step $s$9, the model concatenates $e_s$0 and $e_s$1, yielding a 384-D input to a 2-layer LSTM. The decoder head then predicts the next frame joint angles, producing an output sequence $e_s$2. The use of a single global latent $e_s$3 means that stochastic variation is injected sequence-wide rather than as a frame-local disturbance (Ghorbani et al., 2022).

3. Conditional variational formulation and style representation

EGSample is formulated as a conditional VAE. Let $e_s$4 denote the speech sequence, $e_s$5 the example motion clip, and $e_s$6 the ground-truth gesture sequence. The recognition network, prior network, and decoder are defined as

$e_s$7

$e_s$8

$e_s$9

The loss is the sum of a reconstruction term and a KL term:

$z$0

$z$1

$z$2

In practice, the likelihood on $z$3 is Gaussian:

$z$4

and optimization uses the reparameterization trick (Ghorbani et al., 2022).

The style embedding is defined as $z$5 with $z$6. During training, $z$7 is one clip from a style class. At test time, the system accepts an arbitrary short motion clip, including one from an unseen style, and maps it to $z$8. This embedding supports explicit interpolation between styles. The implementation recipe specifies linear interpolation

$z$9

as well as spherical interpolation

$\hat y$0

This suggests that the style space is intended to be operationally navigable rather than merely discriminative. In EGSample, style is not only inferred from exemplars but also treated as a manipulable latent control signal (Ghorbani et al., 2022).

4. Data model and preprocessing pipeline

The data source is the public ZeroEGGS dataset. It contains 19 distinct gestural styles, full-body motion capture at 60 Hz including finger articulation across 21 joints, and aligned speech audio at 44.1 kHz with 10 minutes per speaker. The abstract characterizes the release as a high-quality dataset spanning 19 different styles (Ghorbani et al., 2022).

The preprocessing pipeline is specified stepwise:

1. Motion retarget to a canonical skeleton.
2. Zero-center root, normalize bone lengths, remove global translation/rotation.
3. Downsample to 60 Hz.
4. Extract a 63-D joint angle vector per frame, then smooth with a 3-frame median.
5. Use timestamps from MoCap for speech-motion alignment.
6. Extract speech features.

The speech feature specification includes log-mel spectrograms with 80 bins, $\hat y$1 derivatives for a total of 240-D per frame, and a $\hat y$2-frame stack giving $\hat y$3-D context. The architecture description separately states that the speech encoder takes log-mel spectrogram frames with 80 mel bins, 25 ms windowing, and 10 ms hop. Taken together, the recipe specifies both the low-level acoustic frontend and an expanded contextual speech representation (Ghorbani et al., 2022).

The motion representation is equally explicit. The example style clip is represented as joint-angle trajectories, with each of the 21 joints carrying a 3D rotation, either in axis-angle or quaternion form, for per-frame dimensionality $\hat y$4. The model thus operates on pose parameters rather than Cartesian keypoints or global trajectory coordinates. Because global translation and rotation are removed during preprocessing, the generated gestures are modeled in a canonicalized body-centered space (Ghorbani et al., 2022).

5. Training regime and inference procedure

The training recipe specifies Adam with learning rate $\hat y$5, $\hat y$6, $\hat y$7, and weight decay $\hat y$8. Training uses batch size 64, sequence chunk length 4 seconds or 240 frames, and 200 epochs, approximately 100k iterations. A $\hat y$9-scheduler linearly warms up from 0 to 1 in the first 20k steps. The latent dimension is $f_x$0, the style dimension is $f_x$1, the speech feature dimension is $f_x$2, and dropout 0.1 is applied on all MLP layers (Ghorbani et al., 2022).

Inference follows a fixed sequence. First, target speech $f_x$3 is recorded or loaded. A short style example clip $f_x$4 is then chosen or recorded. After preprocessing, the system computes the style embedding $f_x$5, obtains prior parameters $f_x$6, samples $f_x$7, computes speech features $f_x$8, and runs the gesture decoder to produce $f_x$9. Post-processing blends to $z$0-length and adds small Gaussian noise for realism (Ghorbani et al., 2022).

The sampling mechanism is central to the “sampling” aspect of EGSample. At inference, the prior distribution is conditioned only on style. The latent is sampled as

$z$1

where $z$2 is a temperature or variance-scaling factor, with the recipe giving $z$3 as an example range. Each independent sample of $z$4 yields a different gesture $z$5. This explicitly operationalizes diversity under fixed speech and style conditions, in contrast to deterministic regressors (Ghorbani et al., 2022).

6. Evaluation protocol and reported findings

The evaluation protocol combines automatic metrics and a user study. The automatic metrics are FGD (Fréchet Gesture Distance) against real motions per style, Beat Coverage measuring how many speech emphasis points are mirrored in motion, and Style Classification Accuracy obtained by training a style classifier on generated versus real motion (Ghorbani et al., 2022).

The user study uses $z$6 participants in a randomized forced-choice design. Each participant sees 10 pairs, drawn from three comparison types: “ours vs baseline,” “ours vs real,” and “ours-diverse vs ours-mono.” Ratings are collected on 5-point Likert scales for Naturalness, Appropriateness to speech, and Style matching. Statistical analysis uses paired $z$7-tests with $z$8 (Ghorbani et al., 2022).

The reported findings are specific. Zero-shot styles produced with no retraining achieve 85% of “seen” style FGD. In user preference, the method is preferred over a GAN-based baseline by $z$9 Likert points on naturalness and $T_s \times 80$0 on style. The abstract additionally states that the model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal, and that experiments demonstrate flexibility and generalizability to new speakers and styles (Ghorbani et al., 2022).

These results position EGSample as an overview framework whose claims rest on three linked properties: zero-shot style control, probabilistic output diversity, and example-based conditioning. A plausible implication is that its contribution is not confined to style transfer alone; it also addresses the long-standing mismatch between deterministic speech-to-gesture mapping and the stochastic character of human gesture motion.