Diffusion Autoencoder with Perceivers (DAEP)

• DAEP is an architectural paradigm that fuses advanced tokenization, cross-attention encoding, and iterative diffusion decoding to process long, irregular, and multimodal sequences.
• It utilizes a Perceiver-IO diffusion decoder for progressive denoising, ensuring high-fidelity reconstruction even with missing or corrupted measurements.
• Benchmarking shows DAEP outperforms VAE and masked autoencoder baselines, achieving lower reconstruction errors and richer latent representations for downstream classification.

Diffusion Autoencoder with Perceivers (daep) is an architectural paradigm designed to learn representations and reconstruct data for domains where inputs consist of long, irregular, and multimodal sequences. The framework combines advanced tokenization strategies, Perceiver-based encoders, and a Perceiver-IO-guided diffusion decoder to achieve scalable, high-fidelity reconstruction and discriminative latent spaces. daep has established state-of-the-art results on spectroscopic and photometric astronomical datasets, outperforming both variational autoencoders (VAE) and masked autoencoder baselines in reconstruction error and downstream classification metrics (Shen et al., 23 Oct 2025).

1. Architectural Principles and Data Constraints

daep is motivated by the need to process scientific data that depart from traditional image or video formats. Such data—common in astronomy, healthcare, finance, and sensor networks—arrive as long, irregularly sampled streams and often combine heterogeneous modalities. Standard architectures (e.g., CNNs, ViTs) fail to capture semantic relationships in these settings due to rigid input structures.

The daep architecture is composed of:

• Tokenizer: Embeds heterogeneous measurements (values, positions, and metadata) into tokens. This is accomplished by separately embedding measurement values (e.g., flux), continuous/categorical position information (e.g., wavelength, time, filter type), and metadata (e.g., instrument id). Continuous coordinates are sinusoidally embedded and optionally refined by a shallow MLP; categorical features receive learned embeddings.
• Perceiver Encoder: Utilizes cross-attention from a fixed bottleneck query sequence into the input token sequence, followed by repeated Perceiver blocks (cross-attention and bottleneck self-attention), yielding a fixed-size latent representation $z$ regardless of input length.
• Perceiver-IO Diffusion Decoder: Reconstructs the original input by iterative denoising. At each timestep, a noisy version of the tokenized input $x_t$ and the latent $z$ are used for predicting added noise. Noise and step embeddings are concatenated with $z$, and two stages of cross-attention refine the noise prediction. This module’s flexibility allows efficient handling of diverse sequence lengths and modalities.

2. Tokenization, Encoding, and Latent Compression

Raw input data $(v,s,m)$ are tokenized to capture value ($v$), position ($s$), and metadata ($m$) information. After projection, the token sequence is efficiently compressed by Perceiver-based encoding. The encoder applies cross-attention to relate variable-length input tokens to a manageable set of bottleneck queries, followed by bottleneck self-attention to capture contextual dependencies. In architectural terms, the Perceiver encoder generalizes to:

$z = \textrm{PerceiverEnc}_\theta(x_\text{tokens})$

A distinguishing feature is the handling of arbitrary input lengths (e.g., thousands of spectral measurements or observation times), ensuring scalability and eliminating the need for manual sequence truncation or padding.

3. Diffusion Decoding and Denoising Dynamics

The decoder is implemented as a Perceiver-IO diffusion transformer, which progressively reconstructs the input through score-based denoising:

• At each timestep $t$, the noisy tokens $x_t$, latent $z$, and time embedding are processed to predict noise $\epsilon_\theta(x_t, z, t)$.
• The loss is computed as the L2 distance between the predicted and true noise:

$$\mathcal{L} = \| \epsilon - \epsilon_\theta(x_t, z, t) \|^2$$

• The iterative process allows temporal refinement: coarse structure is recovered in early denoising steps, with fine details reconstructed as noise diminishes.

This decoding strategy is robust to missing or heavily corrupted measurements, critical in astronomical data where physical phenomena manifest across diverse locations, scales, and observational environments.

4. Comparative Benchmarking: VAE and Masked Autoencoder Baselines

daep’s effectiveness is established through direct comparison to:

• VAE Baseline: Employs identical Perceiver encoder and decoder components but is trained via a $\text{KL}$ divergence and L2 reconstruction loss. VAE struggles with fine detail reconstruction due to smooth latent distributions and direct mapping.
• maep (Masked Autoencoder with Perceiver): Extends masked autoencoder principles with a Perceiver encoder and masked reconstruction decoder. It relies on context from unmasked tokens but omits generative denoising, thus underperforming on fidelity and latent discriminability.

Extensive benchmarks on LAMOST spectra, ZTF light curves, and other datasets demonstrate that daep reliably yields:

• Lower per-measurement absolute reconstruction errors
• Latent spaces better suited for downstream classification (e.g., supernova, variable star classification via linear probing)
• Enhanced preservation of high-frequency spectral features absent in VAE or maep reconstructions

5. Handling Multimodal, Irregular, and Long Sequences

daep’s architecture is domain-agnostic and inherently scalable to multimodal and irregular sequence domains. The Perceiver encoder’s linear complexity in input length and its cross-modal tokenization strategy enable flexible handling of data with missing measurements, irregular sampling, and heterogeneous modalities.

Its design specifically enables recovery of fine-scale structure, which is vital for identification of physical phenomena in astronomical datasets (e.g., absorption lines in spectra, transient photometric events). The iterative denoising process makes daep resilient against common challenges in scientific data, such as sparse sampling or variable observation conditions.

6. Applications and Broader Implications

While developed in the context of astronomy, daep’s principles are broadly transferable. The robust representation and reconstruction of long, irregular, multimodal sequences allow its application to:

• Healthcare: longitudinal patient records, heterogeneous sensor readings
• Finance: acquisition and interpolation of irregular transaction histories
• Environmental monitoring: sensor arrays with variable-resolution temporal measurements

By enabling self-supervised latent space learning, daep facilitates effective compression for downstream tasks—classification, anomaly detection, simulation, and generative modeling.

7. Future Directions and Framework Extensions

The framework invites further investigation along several fronts:

• Cross-domain validation: Applying daep to other scientific disciplines and multimodal fusion tasks
• Hybrid objectives: Integrating predictive or contrastive losses to further regularize representation learning
• Efficiency optimization: Reducing the computational cost of diffusion decoding for practical deployment in large-scale scientific workflows
• Generative sampling: Developing latent distribution models for explicit simulation of physical systems
• Noise-aware modeling: Enhancing tokenization and embedding for measurement uncertainty, crucial in scientific and medical datasets

In summary, Diffusion Autoencoder with Perceivers (daep) synthesizes advanced tokenization, scalable attention-based encoding, and generative diffusion-based decoding, yielding state-of-the-art reconstruction quality and representation learning for irregular, long, and multimodal data streams, with clear potential across scientific, medical, and industrial domains (Shen et al., 23 Oct 2025).