---
title: 'EGSample: Example-based Gesture Sampling'
url: https://www.emergentmind.com/topics/egsample
type: topic
---

# EGSample: Example-based Gesture Sampling

Searching arXiv for the primary paper and closely related gesture-generation context.
EGSample, short for “Example-based Gesture Sampling,” is an implementation-oriented formulation of zero-shot, example-conditioned speech-driven gesture generation built on the ZeroEGGS framework. It is designed to generate full-body gesture motion from speech while controlling style from only a short example motion clip, including motion styles unseen during training. The system combines a speech encoder, a style encoder, a conditional variational formulation, and a gesture decoder so that style can be transferred, blended, or sampled probabilistically, yielding multiple plausible gestures for the same speech input [2209.07556].

## 1. Conceptual definition and scope

EGSample is framed as a neural network system for speech-driven gesture generation with zero-shot style control by example. In this setting, speech is the primary conditioning signal, while a short reference motion clip provides style information. The defining claim is that style control can be achieved from an arbitrary short motion example, even when the motion style was unseen during training. This places EGSample within example-based and zero-shot style transfer paradigms rather than class-conditioned gesture synthesis [2209.07556].

The approach also treats gesture generation as intrinsically stochastic. Rather than producing a single deterministic output for a given utterance, it uses a probabilistic latent variable to represent variability in gesture motion. This supports multiple distinct outputs for the same speech and style conditions. The underlying paper further states that the model’s variational structure makes style embedding manipulation straightforward, including latent-space modification, blending, and scaling of style embeddings [2209.07556].

A plausible implication is that EGSample is intended not only as a reproduction recipe for ZeroEGGS, but also as a practical basis for extending example-conditioned gesture generation to new styles or modalities. The data block explicitly presents it in that role by specifying architecture, losses, preprocessing, hyperparameters, inference, and evaluation in implementation-oriented form.

## 2. Architectural organization

The model is organized as a left-to-right pipeline: speech \(x\) is encoded into frame-level speech features \(f_x\), a style example \(s\) is encoded into a style embedding \(e_s\), a conditional variational core defines posterior and prior distributions over a latent variable \(z\), and a gesture decoder generates the motion sequence \(\hat y\) conditioned on \(f_x\) and \(z\) [2209.07556].

| Component | Input | Configuration |
|---|---|---|
| Speech Encoder | Log-mel spectrogram frames, \(T_s \times 80\) | 3 layers 1D CNN, channels \([64,128,256]\), kernel \(5\), stride \(2\); 2-layer Bi-LSTM, hidden \(256\) per direction; linear projection to \(f_x \in \mathbb{R}^{T \times 256}\) |
| Style Encoder | Example motion clip as joint-angle trajectories | \(2\times\) 1D conv, channels \([64,128]\), kernel \(3\), stride \(1\); global average pooling; MLP \(128 \rightarrow 256\) with ReLU to \(e_s \in \mathbb{R}^{256}\) |
| Gesture Decoder | \(f_x\) and global latent \(z \in \mathbb{R}^{128}\) | Concatenate \(f_x[t] \oplus z\); 2-layer LSTM, hidden \(256\); MLP head \(256 \rightarrow 128 \rightarrow D_j\) |

The speech encoder takes log-mel spectrogram frames computed with 80 mel bins, a 25 ms window, and a 10 ms hop. After temporal convolution and bidirectional recurrence, it produces \(f_x \in \mathbb{R}^{T \times D_f}\) with \(D_f = 256\). The style encoder operates on an example motion clip represented as joint-angle trajectories for \(J = 21\) joints, each with 3D rotation, giving per-frame dimension \(D_j = 63\). The clip length is approximately 2 seconds, or 120 frames at 60 Hz [2209.07556].

The gesture decoder receives the per-frame speech features together with a global latent code. At each time step \(t\), the model concatenates \(f_x[t]\) and \(z\), yielding a 384-D input to a 2-layer LSTM. The decoder head then predicts the next frame joint angles, producing an output sequence \(\hat y \in \mathbb{R}^{T \times D_j}\). The use of a single global latent \(z\) means that stochastic variation is injected sequence-wide rather than as a frame-local disturbance [2209.07556].

## 3. Conditional variational formulation and style representation

EGSample is formulated as a conditional VAE. Let \(x\) denote the speech sequence, \(s\) the example motion clip, and \(y\) the ground-truth gesture sequence. The recognition network, prior network, and decoder are defined as

$$
q_{\phi}(z \mid x, s) = \mathcal{N}\bigl(z; \mu_q(x,s), \Sigma_q(x,s)\bigr),
$$

$$
p_{\psi}(z \mid s) = \mathcal{N}\bigl(z; \mu_p(s), \Sigma_p(s)\bigr),
$$

$$
p_{\theta}(y \mid x, z).
$$

The loss is the sum of a reconstruction term and a KL term:

$$
L_{\text{rec}} = -\,\mathbb{E}_{z \sim q_{\phi}(z \mid x,s)}\bigl[\log p_{\theta}(y \mid x,z)\bigr],
$$

$$
L_{\text{KL}} = D_{\mathrm{KL}}\bigl(q_{\phi}(z \mid x,s)\,\|\,p_{\psi}(z \mid s)\bigr),
$$

$$
L = L_{\text{rec}} + \beta L_{\text{KL}}.
$$

In practice, the likelihood on \(y\) is Gaussian:

$$
\log p_{\theta}(y \mid x,z)
= -\frac{1}{2\sigma^2}\|y - \hat y_\theta(x,z)\|^2 + C,
$$

and optimization uses the reparameterization trick [2209.07556].

The style embedding is defined as \(e_s = E_{\psi}(s) \in \mathbb{R}^{D_s}\) with \(D_s = 256\). During training, \(s\) is one clip from a style class. At test time, the system accepts an arbitrary short motion clip, including one from an unseen style, and maps it to \(e_s\). This embedding supports explicit interpolation between styles. The implementation recipe specifies linear interpolation

$$
e_\lambda = (1-\lambda)e_{s_1} + \lambda e_{s_2},
$$

as well as spherical interpolation

$$
\mathrm{slerp}(e_{s_1}, e_{s_2}; \lambda)
= \frac{\sin((1-\lambda)\Omega)}{\sin\Omega}e_{s_1}
+ \frac{\sin(\lambda \Omega)}{\sin\Omega}e_{s_2},
\quad
\Omega = \arccos\bigl(\frac{\langle e_{s_1}, e_{s_2}\rangle}{\|e_{s_1}\|\|e_{s_2}\|}\bigr).
$$

This suggests that the style space is intended to be operationally navigable rather than merely discriminative. In EGSample, style is not only inferred from exemplars but also treated as a manipulable latent control signal [2209.07556].

## 4. Data model and preprocessing pipeline

The data source is the public ZeroEGGS dataset. It contains 19 distinct gestural styles, full-body motion capture at 60 Hz including finger articulation across 21 joints, and aligned speech audio at 44.1 kHz with 10 minutes per speaker. The abstract characterizes the release as a high-quality dataset spanning 19 different styles [2209.07556].

The preprocessing pipeline is specified stepwise:

1. Motion retarget to a canonical skeleton.
2. Zero-center root, normalize bone lengths, remove global translation/rotation.
3. Downsample to 60 Hz.
4. Extract a 63-D joint angle vector per frame, then smooth with a 3-frame median.
5. Use timestamps from MoCap for speech-motion alignment.
6. Extract speech features.

The speech feature specification includes log-mel spectrograms with 80 bins, \(\Delta + \Delta\Delta\) derivatives for a total of 240-D per frame, and a \(\pm 2\)-frame stack giving \(5 \times 240 = 1{,}200\)-D context. The architecture description separately states that the speech encoder takes log-mel spectrogram frames with 80 mel bins, 25 ms windowing, and 10 ms hop. Taken together, the recipe specifies both the low-level acoustic frontend and an expanded contextual speech representation [2209.07556].

The motion representation is equally explicit. The example style clip is represented as joint-angle trajectories, with each of the 21 joints carrying a 3D rotation, either in axis-angle or quaternion form, for per-frame dimensionality \(D_j = 63\). The model thus operates on pose parameters rather than Cartesian keypoints or global trajectory coordinates. Because global translation and rotation are removed during preprocessing, the generated gestures are modeled in a canonicalized body-centered space [2209.07556].

## 5. Training regime and inference procedure

The training recipe specifies Adam with learning rate \(10^{-4}\), \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), and weight decay \(10^{-5}\). Training uses batch size 64, sequence chunk length 4 seconds or 240 frames, and 200 epochs, approximately 100k iterations. A \(\beta\)-scheduler linearly warms up from 0 to 1 in the first 20k steps. The latent dimension is \(D_z = 128\), the style dimension is \(D_s = 256\), the speech feature dimension is \(D_f = 256\), and dropout 0.1 is applied on all MLP layers [2209.07556].

Inference follows a fixed sequence. First, target speech \(x_{1:T}\) is recorded or loaded. A short style example clip \(s\) is then chosen or recorded. After preprocessing, the system computes the style embedding \(e_s = \mathrm{StyleEncoder}(s)\), obtains prior parameters \(\mu_p,\Sigma_p = \mathrm{PriorNet}(e_s)\), samples \(z \sim \mathcal{N}(\mu_p,\tau \Sigma_p)\), computes speech features \(f_x = \mathrm{SpeechEncoder}(x)\), and runs the gesture decoder to produce \(\hat y_{1:T}\). Post-processing blends to \(T\)-length and adds small Gaussian noise for realism [2209.07556].

The sampling mechanism is central to the “sampling” aspect of EGSample. At inference, the prior distribution is conditioned only on style. The latent is sampled as

$$
z \sim \mathcal{N}(\mu_p,\tau \Sigma_p),
$$

where \(\tau > 0\) is a temperature or variance-scaling factor, with the recipe giving \(\tau \in [0.5,1.5]\) as an example range. Each independent sample of \(z\) yields a different gesture \(\hat y\). This explicitly operationalizes diversity under fixed speech and style conditions, in contrast to deterministic regressors [2209.07556].

## 6. Evaluation protocol and reported findings

The evaluation protocol combines automatic metrics and a user study. The automatic metrics are FGD (Fréchet Gesture Distance) against real motions per style, Beat Coverage measuring how many speech emphasis points are mirrored in motion, and Style Classification Accuracy obtained by training a style classifier on generated versus real motion [2209.07556].

The user study uses \(N = 30\) participants in a randomized forced-choice design. Each participant sees 10 pairs, drawn from three comparison types: “ours vs baseline,” “ours vs real,” and “ours-diverse vs ours-mono.” Ratings are collected on 5-point Likert scales for Naturalness, Appropriateness to speech, and Style matching. Statistical analysis uses paired \(t\)-tests with \(p < 0.05\) [2209.07556].

The reported findings are specific. Zero-shot styles produced with no retraining achieve 85% of “seen” style FGD. In user preference, the method is preferred over a GAN-based baseline by \(+1.2\) Likert points on naturalness and \(+0.8\) on style. The abstract additionally states that the model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal, and that experiments demonstrate flexibility and generalizability to new speakers and styles [2209.07556].

These results position EGSample as a synthesis framework whose claims rest on three linked properties: zero-shot style control, probabilistic output diversity, and example-based conditioning. A plausible implication is that its contribution is not confined to style transfer alone; it also addresses the long-standing mismatch between deterministic speech-to-gesture mapping and the stochastic character of human gesture motion.

Source: https://www.emergentmind.com/topics/egsample