ARIMA: Reconstruction-Grounded Predictive Representation Learning for Symbolic Music

Abstract: Self-supervised learning for symbolic music has advanced largely through token-level pretraining, but such representations remain tied to tokenizer-specific sequences and often provide time-span-level embeddings only indirectly. In this paper, we propose ARIMA, a reconstruction-grounded latent predictive framework for symbolic music that learns compact window-based representations directly from data. ARIMA encodes each fixed-duration window into a continuous latent representation, trains a causal predictor with contrastive next-latent prediction, and grounds the encoder through structured reconstruction of music elements. This design preserves local musical details while modeling temporal progression across windows. We evaluate ARIMA on downstream tasks spanning various levels of music understanding. Results show that ARIMA is particularly efficient and effective on tasks involving harmonic, timing, and cross-performance retrieval, while remaining competitive with much larger baselines on other tasks. Ablations further show that next-latent prediction is essential for temporally integrated representations, and that structured reconstruction stabilizes latent learning without requiring explicit variance regularization. The code is at https://github.com/AndyWeasley2004/symbolic_music_wm.

• The paper presents ARIMA, which integrates structured reconstruction with contrastive next-latent prediction to capture both local musical details and global temporal progression.
• It employs a Transformer-based encoder to create fixed-duration window embeddings, achieving competitive performance across nine symbolic music tasks against larger models.
• The approach decouples local content encoding and global context integration, enhancing key estimation and temporal modeling for improved symbolic music analysis.

ARIMA: Reconstruction-Grounded Predictive Representation Learning for Symbolic Music

Introduction

ARIMA introduces a window-level self-supervised learning paradigm for symbolic music representation, combining structured reconstruction objectives with contrastive latent prediction. This formulation directly addresses the limitations of conventional token-based pretraining approaches in symbolic music, which are highly tokenizer-dependent and provide time-span-level embeddings only through indirect pooling or adaptation. By utilizing a compact, fixed-duration window encoding pipeline, ARIMA enforces local musical detail preservation and temporal progression modeling, enabling effective and efficient symbolic music analysis across diverse downstream tasks.

Model Architecture

ARIMA partitions symbolic music into fixed-duration windows, encoding each window into a content latent $z_t$ via a Transformer-based encoder. This representation is grounded through structured reconstruction of musical elements—onset and sustain pianorolls, chromaroll, and velocity map. Temporal modeling is dictated by a causal predictor, which receives the sequence of $z_t$ and outputs temporal hidden states $h_t$, trained via contrastive next-latent prediction with an InfoNCE objective against a momentum-averaged (EMA) target encoder. This setup encourages the predictor to integrate temporal context while the encoder remains locally faithful.

The input encoding is tailored to the symbolic music domain, including pitch, velocity, onset time, duration, IOI, and explicit flags for cross-boundary note ties. The training loss is a weighted combination of contrastive latent prediction, binary cross-entropy for pianoroll and chroma reconstruction, and MSE for velocity regression. Augmentation includes pitch transposition and velocity perturbation to enforce desired invariances.

Figure 1: The architecture and training dynamics of ARIMA, from the tokenization, window encoding (left), to structured pianoroll reconstruction and contrastive next latent prediction (right).

Experimental Evaluation

Task Benchmarking

ARIMA was evaluated on nine diverse symbolic music understanding tasks involving harmonic (key estimation), temporal (IOI regression, performer verification, cross-performance retrieval), style (composer, pianist, performer classification), emotion recognition, and score difficulty estimation. Comparisons were provided against established baselines including MidiBERT-Piano, PianoBART, M2BERT, and Aria, as well as ablated ARIMA variants.

Performance was assessed on both the content latent $z$ (local window encoding) and temporal predictor $h$ (integrated causal embedding), clarifying the effects of local versus contextually-aggregated representations.

Results

ARIMA (38M parameters) consistently matched or surpassed larger models (61–110M) on a majority of tasks. The temporal predictor $h$ achieved best-in-class results on composer classification, IOI regression, and performer verification, while $z$ exhibited superior performance on key estimation and cross-performance retrieval. Notably, the chroma reconstruction head in ARIMA improved key estimation significantly, paralleling the strong key estimation in M2BERT, also employing chroma-based objectives.

Tasks benefiting from context aggregation, such as style recognition and temporal abstraction, favored the predictor latent $h$, while more content-localized analyses, such as key estimation or content-based retrieval, were best captured in $z$. This clear delineation supports ARIMA's design in decoupling local reconstruction and global prediction.

Despite ARIMA's reduced parameter count, it remains competitive with models trained on larger, more diverse datasets, such as the public Aria-embedding reference (632M parameters), which, while dominant in some tasks, lags on key estimation, IOI regression, and cross-performance retrieval due to over-invariance to musically critical features.

Ablation Study

Ablations revealed that next-latent prediction is essential for integrating temporal dependencies, as its omission severely degraded both $z$ and (trivially) $z_t$0 representations. Disentangling onset/sustain during reconstruction produces nuanced effects: while joint reconstruction modestly uplifts similarity and timing metrics, disentanglement improves interpretability and style/performer detection, highlighting inductive bias alignment as a strategic design choice.

Variance regularization via VICReg substantially increases the per-dimension variance in $z_t$1 (rendering it more isotropic), but yields minimal gain in downstream accuracy, indicating the sufficiency of structured reconstruction as an anti-collapse signal in the latent space.

Figure 2: Descending sorted per-dimension variance of content latent $z_t$2 and predictor latent $z_t$3 after piece-level mean pooling on EMOPIA and ASAP.

Theoretical and Practical Implications

ARIMA demonstrates that window-level reconstruction-grounded predictive models can outperform or rival token/sequence-based baselines on a spectrum of symbolic MIR tasks, with improved efficiency. The explicit separation of local content encoding and global temporal abstraction affords diagnostic transparency and task-specific alignment, supporting applications in music analysis, MIR, and even generative tasks conditioned on compact, interpretable latents.

The architecture's emphasis on explicit music-theoretic features (e.g., chroma, IOI, velocity), and its inductive bias alignment, make it better suited for tasks sensitive to these dimensions as compared to monolithic contrastive pretraining over unstructured augmentations. The findings imply that purely large-scale foundation model scaling may neglect musically relevant details unless explicitly countered during pretraining objective design.

From a theoretical standpoint, ARIMA provides evidence challenging the direct transposition of JEPA-style latent prediction from vision to symbolic music, showing the necessity for representation grounding via structured reconstruction given the sparsity and multi-modality inherent in symbolic music data.

Future Directions

Scaling ARIMA on broader and more diverse symbolic corpora could further close gaps with foundation-scale models and extend its utility to complex symbolic music generation and multi-level editing tasks. Hierarchical or structure-aware variants, combining latent prediction at multiple temporal resolutions, could enhance long-term abstraction and enable more expressive downstream conditioning. Applications extending to interactive music analysis, performance transformation, and robust cross-domain symbolic audio transfer are plausible.

Conclusion

ARIMA establishes a new direction for symbolic music self-supervised learning, leveraging window-level reconstruction and causal next-latent prediction to produce both musically faithful and contextually aware representations. The empirical results solidify the advantages of structured, inductive-bias-aligned objectives over generic, token-focused approaches. The framework invites further research into scalable, musically informed self-supervised methods, with implications for both symbolic MIR and creative generative modeling (2607.10003).