RIPO Transformer: Symbolic Music Modeling

• RIPO Transformer is an architecture that integrates bias-adjusted sinusoidal embeddings and multi-relational attention to capture musical invariants in symbolic music.
• It uses the Fundamental Music Embedding (FME) to preserve absolute and relative musical attributes, ensuring translational invariance and transposability.
• Empirical evaluations show that the model outperforms standard approaches in melody prediction and generation, aligning closely with human musical perception.

The RIPO Transformer is an architecture for symbolic music modeling that integrates a mathematically principled embedding space and a novel multi-relational attention mechanism. Designed to address the limitations of standard Transformer models in capturing domain-specific invariances and relational structures inherent in symbolic music, the RIPO Transformer leverages domain knowledge to enhance both predictive and generative tasks. Its principal innovations are the Fundamental Music Embedding (FME)—a bias-adjusted sinusoidal representation preserving musical invariants—and the RIPO attention mechanism, which jointly conditions attention weights on relative index, pitch, and onset relationships. Empirical evaluation demonstrates state-of-the-art performance in melody modeling and generation, with marked improvements in the avoidance of degenerative patterns and alignment with human musical perception (Guo et al., 2022).

1. Motivation and Conceptual Distinctions

Symbolic music differs fundamentally from natural language in that its tokens carry both absolute (e.g., pitch, duration, onset) and relative (e.g., pitch interval, time shift) information. These relative attributes, such as the intervallic relationships between pitches and onset-time differences, underpin human perception of musical motifs and patterns. Standard input encodings—either learned token embeddings or text-inspired positional methods—fail to preserve critical musical invariants, such as interval-based transposition and translational invariance. The RIPO Transformer is motivated by the need for an embedding and attention framework that renders both absolute and relative information explicitly, maintaining properties essential for musically meaningful sequence modeling (Guo et al., 2022).

2. Fundamental Music Embedding (FME)

FME is a bias-adjusted sinusoidal encoding for symbolic music tokens, designed to preserve both absolute and relative musical invariants within the embedding space. Its mathematical construction ensures translational invariance, transposability, and separability among feature types.

Given a sequence $F$ of scalar music tokens (e.g., pitches $P$, durations $D$, onsets $O$), FME maps each $f \in F$ into a $d$-dimensional vector by concatenating $d/2$ sin-cos components:

1. Define $w_k = B^{-2k/d}$, with $B$ a type-specific base (e.g., $B_P$ for pitch, $P$0 for duration).
2. For an absolute token $P$1,

$P$2

where $P$3 are trainable biases.

1. The embedding is:

$P$4

Relative embeddings, termed Fundamental Music Shift (FMS), are defined for any interval $P$5 without bias:

$P$6

Key invariances are:

• Translational invariance: $P$7, a function of the absolute interval.
• Transposability: Existence of a block-diagonal rotation matrix $P$8, derived from $P$9, such that $D$0.
• Separability: Trainable biases $D$1 and $D$2 enforce distinctive subspaces for pitch, duration, and onset.

3. RIPO Attention Mechanism

The RIPO attention mechanism extends standard self-attention by incorporating three distinct forms of relational information:

• Relative index ($D$3, as in Shaw et al. “skewing”)
• Relative pitch intervals ($D$4)
• Relative onset-time intervals ($D$5)

Absolute positional encodings are constructed as:

$D$6

with $D$7 and $D$8 in the same sinusoidal form as FME, but with independent base values $D$9. The periodic metrical structure is modeled by the $O$0 component.

Input projection is given by:

$O$1

with $O$2 derived as standard.

The attention computation for each head (dimension $O$3) combines:

• Relative-pitch logits: $O$4
• Relative-onset logits: $O$5
• Relative-index logits $O$6, inherited from Music Transformer skewing.

The final attention scores:

$O$7

4. Architectural Organization

The RIPO Transformer comprises:

• Two layers, each with self-attention and feedforward subblocks.
• Eight attention heads per layer.
• Hidden and embedding dimensionality of 256 ($O$8 per head).
• Pitch and duration inputs are embedded using FME; onset is either implicit (as cumulative duration) or encoded by $O$9.

Principal architectural divergences from canonical Transformer implementations include the substitution of FME for trainable embedding tables, RIPO attention for vanilla multi-head attention or index-only relative attention, and the utilization of onset and beat index positional encodings. The current experimental setup supports monophonic sequence generation.

5. Empirical Evaluation

Dataset and Tasks

• Data: 10,199 monophonic 4/4 pieces from the TheoryTab and Wikifonia corpora, transposed to C major/A minor; 90%/10% train/test split.
• Melody completion (Task 1): next-token pitch and duration prediction with cross-entropy loss.
• Downstream generation (Task 2): conditioned on a two-bar seed, generate sixteen bars autoregressively.

Hyperparameters

• Batch size: 16. Optimizer: Adam, initial learning rate $f \in F$0 with decay.
• Hidden/embedding size: 256. Two layers. Eight heads.
• Duration quantized to 16th-note grid; vocab size 17 (including sustain/pad).
• Sampling: top-$f \in F$1 ($f \in F$2) and top-$f \in F$3 ($f \in F$4) strategies with temperature scaling.

Results

Model	Melody Completion (CE$f \in F$5+CE$f \in F$6)	KLD (Pitch/Duration)	In-scale Ratio (ISR)	Arpeggio Ratio (AR)	Seq-rep-4
Music Transformer (one-hot)	2.405	0.014–0.041	—	—	0.50–0.71
Linear Transformer (one-hot)	2.880	0.033–0.040	—	—	0.50–0.71
RIPO Transformer + FME	2.367	0.011/0.024	0.981	0.049	0.29–0.54

Additional findings:

• RIPO Transformer exhibits step-wise sampling probabilities and variance closely matching human data, whereas baselines display overconfident, low-variance predictions.
• In listening tests with 25 participants (5-point Likert scale): RIPO outperforms baselines in overall enjoyment (3.57 vs. 2.80), pitch correctness (3.43 vs. 2.97), duration correctness (3.37 vs. 2.95), and interestingness (3.52 vs. 2.94).

6. Ablation and Component Contribution

Ablation experiments reveal the performance impact of RIPO’s three relational terms:

Component Removed	Test Loss
None (Full Model)	2.367
$f \in F$7	2.381
$f \in F$8	2.398
$f \in F$9	2.686
Onset PE or Beat PE	+0.03–0.05

Each relational term and positional encoding contributes noticeably to modeling efficacy, with index (relative-position) ablation exhibiting the largest negative effect.

7. Significance and Implications

The RIPO Transformer advances symbolic music modeling by explicitly embedding musically meaningful invariants and jointly conditioning attention computation on index, pitch interval, and onset interval relations. This dual mechanism supports superior sequence prediction and generative tasks, as evidenced by notable improvements in cross-entropy loss, music-structural metrics (KLD, ISR, AR), sequence degeneracy metrics (seq-rep-4), and human evaluation. The approach addresses critical limitations of prior models such as uninformative embedding spaces and the absence of domain-relational inductive biases, yielding outputs that are both statistically and perceptually aligned with human-created music (Guo et al., 2022).

A plausible implication is that architectures employing bias-adjusted sinusoidal encodings and multi-faceted relational attention may provide similar benefits in other structured sequence domains where domain-specific invariances and relational structure play a critical role.