---
title: Anticipatory Music Transformer (AMT)
url: https://www.emergentmind.com/topics/anticipatory-music-transformer-amt
type: topic
---

# Anticipatory Music Transformer (AMT)

The Anticipatory Music Transformer (AMT) is a class of autoregressive generative models for symbolic music that generalize the standard Transformer architecture to permit asynchronous control via a formally defined mechanism called "anticipation." This framework enables tractable, controllable generation of event sequences (such as musical notes) conditioned on arbitrary subsets of future events, a critical capability for musical infilling and accompaniment. AMT interleaves event and control tokens in a single causal stream using a precise stopping-time criterion, thus preserving both model tractability and causal generation. This approach is implemented via minor data-level modifications to a GPT-2 style Transformer, allowing standard architectures and optimization regimes to be used without core changes [2306.08620].

## 1. Anticipation and Temporal Point Process Modeling

AMT models two marked temporal point processes:
- The event process $\{e_j=(t_j, m_j)\}_{j=1}^N$ comprises arrival times $0 \leq t_1 \leq \dotsc \leq t_N$ and marks (e.g., pitch, velocity) $m_j$ from a finite vocabulary $\mathcal{V}$.
- The control process $\{u_k = (s_k, c_k)\}_{k=1}^K$ includes control times $0 \leq s_1 \leq \dotsc \leq s_K$ and control-marks $c_k$ in $\mathcal{U}$, often $\mathcal{U} = \mathcal{V}$ for infilling.

Anticipation enables the modeling of the joint conditional distribution $p(e_{1:N} \mid u_{1:K})$ by interleaving the event and control sequences using the operator $\mathrm{interleave}_\delta(e_{1:N}, u_{1:K})$, producing a single sequence $a_{1:N+K}$. Each control $u_k$ at time $s_k$ is “anticipated” into the sequence at index
$$
\tau_{u_k} = k + \min \{j: t_j \ge s_k - \delta \}
$$
and each event $e_j$ is inserted at
$$
\tau_{e_j} = j + \max \{k: t_{j-1} \ge s_k - \delta \}.
$$
Here, $\delta > 0$ is the anticipation interval—a hyperparameter determining how far in advance controls manifest in the causal stream. The interleaving preserves the stopping-time property: $\tau_{u_k} - 1$ depends only on the prefix $a_{1:i-1}$, ensuring causal, tractable sampling.

After interleaving, the joint is factorized autoregressively:
$$
p(a_{1:N+K}) = \prod_{i=1}^{N+K} p(a_i \mid a_{1:i-1}),
$$
and, for conditional generation, the model masks the loss on control indices during training [2306.08620].

## 2. Model Architecture and Token Sequence Construction

AMT employs a decoder-only (causally masked) Transformer architecture closely aligned with GPT-2, but introduces critical data-side modifications:
- **Arrival-time tokenization:** Each event $e_j = (t_j, d_j, n_j)$ is quantized into three tokens—onset $\tilde t_j = \mathrm{floor}(t_j/0.01)$, duration $d_j$, and note $n_j$. This encoding yields context-free triplet tokens (3N total) that are permutation-compatible for interleaving.
- **Vocabulary doubling (control vs event):** Control and event tokens are assigned disjoint ID ranges, enabling the Transformer to distinguish “anticipated” (future control) tokens from historical event tokens of the same type. Every sequence is prepended with a single global control code $z \in \{\mathrm{AR}, \mathrm{AAR}\}$, indicating unconditional or anticipatory generation mode.
- **Position embeddings:** Uses standard absolute positional embeddings over sequence positions $1\dotsc 1024$. No special embeddings for anticipation or stopping-times are required.
- **Attention masking:** Utilizes causal (decoder-only) masking; no need for bidirectional or custom sparse mechanisms, as asynchronous controls are already woven into the causal stream.

Other architecture components—multi-head self-attention, layer normalization, GeLU activations, and residual connections—are standard [2306.08620].

## 3. Maximum Likelihood Training and Evaluation Metrics

Training proceeds with a maximum-likelihood objective. Given a packed example $x_{1:M}$ (sequence of interleaved tokens plus control code),
$$
\max_{\theta} \sum_{i=1}^M \log p_{\theta}(x_i \mid z, x_{1:i-1})
$$
with per-token cross-entropy loss
$$
L_{\mathrm{token}} = - \sum_{i=1}^M \log p(x_i \mid x_{<i}).
$$
For event-wise analysis, tokens are grouped into triplets (onset, duration, note), and the event loss is:
$$
L_{\mathrm{event}} = -\sum_{j=1}^{N} [\log p(\tilde t_j | \cdots) + \log p(d_j | \cdots) + \log p(n_j | \cdots)]
$$
with corresponding event, onset, duration, and note perplexities reported as $ppl(e) = \exp(L_{\mathrm{event}}/N)$, alongside $ppl(\tilde t), ppl(d), ppl(n)$. No auxiliary losses or discriminators are used—plain autoregressive cross-entropy suffices [2306.08620].

## 4. Dataset Processing, Model Scaling, and Training Regimes

AMT is trained on the filtered Lakh MIDI dataset (Raffel et al. 2016): 164,747 MIDI sequences, comprising 7,827h training, 555h validation, and 561h test data. Preprocessing steps include:
- Exclusion of sequences $<$10s or $>$1h and those with $>$16 instruments.
- **Sparsity correction:** REST tokens are inserted every 1s between sparse events to prevent the anticipation interval ($\delta$ up to 5s) from overshooting the target time.
- **Tokenization:** Arrival-time encoding yields a 27,512-token vocabulary, including REST/SEP/control tokens.

Model configurations are as follows:

| Scale   | Params    | Layers (L) | Heads (H) | Dim (d) |
|---------|-----------|------------|-----------|---------|
| Small   | 128 M     | 12         | 12        | 768     |
| Medium  | 360 M     | 24         | 16        | 1024    |
| Large   | 780 M     | 36         | 20        | 1280    |

Optimization uses AdamW ($\beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 1\text{e}{-8}$), weight decay $0.1$, dropout $0.1$, gradient clipping ($\|g\| \leq 1$), with a linear warmup followed by cosine decay on the learning rate ($l_{\max} = 6\text{e}{-4}, 3\text{e}{-4}, 2\text{e}{-4}$ for Small, Medium, Large). Training utilizes 512 sequences per step (context length 1024), on TPU v3–32 slices. Throughput is $\sim$0.7M tok/s (Small), 0.26M (Medium), 0.14M (Large). Training steps vary by model and experiment; refer to Table 5.1 in [2306.08620].

## 5. Quantitative and Qualitative Evaluation

**Quantitative evaluation** on the test set without anticipation ($z=\text{AR}$) uses bits/sec (b/s) and event perplexity. Representative results:

| Config             | Params | Encoding      | AM? | b/s  | ppl(e) | ppl($\tilde t$) | ppl(d) | ppl(n) |
|--------------------|--------|--------------|-----|------|--------|-----------------|--------|--------|
| Small interarrival | 112 M  | interarrival | —   | 85.9 | —      | —               | —      | —      |
| Small arrival (100k) | 128 M | arrival      | AAR | 80.7 | 15.0   | 1.58            | 3.98   | 2.40   |
| Small arrival (800k) | 128 M | arrival      | AAR | 75.0 | 12.4   | 1.52            | 3.64   | 2.24   |
| Medium (800k)      | 360 M  | arrival      | AAR | 69.7 | 10.4   | 1.49            | 3.29   | 2.12   |

Findings:
- Arrival-time encoding consistently outperforms interarrival encoding.
- Anticipation (AAR) introduces negligible overhead on unconditional log-loss; overhead vanishes with longer training.
- Scaling model size and training duration reduces perplexity.

**Human evaluation** via pairwise judgments was conducted on Amazon Mechanical Turk with 20 s synthesized audio clips (nucleus sampling $p=0.95$):

*Prompt continuation experiments (3-bar prompt + 14 s continuation)*:
- Human vs. Medium: humans slightly preferred ($p=0.0027$).
- Medium vs. Figaro baseline: Medium strongly preferred ($p < 1\text{e}{-8}$).
- Small arrival vs. Small interarrival: Small arrival preferred ($p=0.007$).

*Accompaniment experiments (5 s melody prompt + 15 s accompaniment)*:
- Human vs. AAR: no significant preference ($p=0.19$).
- Human vs. AR baseline: human overwhelmingly preferred ($p < 1\text{e}{-8}$).
- AAR vs. AR baseline: AAR vastly preferred ($p < 1\text{e}{-7}$).
- AAR vs. random retrieval: AAR strongly preferred ($p < 1\text{e}{-8}$).

These results indicate that AMT can produce accompaniments with musicality rivalling human-composed samples over 20-second timescales [2306.08620].

## 6. Functional Advantages and Significance

AMT’s key contribution is a principled method—anticipation—for $\delta$-second-ahead conditioning on asynchronous controls, achieved via data-side modifications alone. No structural or algorithmic changes to Transformer architectures are necessary, retaining compatibility with standard frameworks and optimization techniques. This construction enables tractable infilling and accompaniment tasks in symbolic music generation, matching or exceeding standard autoregressive baselines in both unconditional and conditional scenarios. Empirical results demonstrate negligible performance penalty for anticipation and strong human preference for AMT-generated accompaniments when compared to baselines.

A plausible implication is that anticipation as a general mechanism may extend to controllable sequence modeling outside music, whenever asynchronous or infilling control is desired within causal autoregressive models [2306.08620].

Source: https://www.emergentmind.com/topics/anticipatory-music-transformer-amt