Masked Transformer-Based Autoencoder

• Masked Transformer-Based Autoencoder is a self-supervised framework that uses an asymmetric encoder-decoder to reconstruct masked inputs from modalities like images, time series, and graphs.
• It employs domain-specific tokenization and masking strategies, adapting the masking ratio and reconstruction targets to the inherent characteristics of the data.
• Empirical results show that such models improve downstream task performance and sample efficiency by pretraining encoders that transfer effectively to diverse applications.

Masked transformer-based autoencoders are self-supervised models that learn representations by masking part of an input, encoding the visible subset with a Transformer or Transformer-based backbone, and reconstructing the missing content with a decoder. In the literature summarized here, this paradigm appears across image analysis, time series, graphs, point clouds, meshes, laboratory trajectories in electronic health records, transient imaging, human motion, depth completion, waveform modeling, and biomedical signal processing, with the masking operator, tokenization scheme, reconstruction target, and downstream transfer protocol adapted to each modality (Zhang et al., 2022, Li et al., 2023, Restrepo et al., 5 Jan 2025, Shen et al., 10 Jun 2025).

1. Core formulation and architectural pattern

A recurrent design in this literature is an asymmetric encoder-decoder. The encoder is typically deeper or stronger, while the decoder is deliberately shallow or lightweight. GMAE uses a deep Graphormer encoder and a shallow transformer decoder to reconstruct masked node features (Zhang et al., 2022). Ti-MAE adopts an asymmetric encoder-decoder Transformer in which only visible timestamps are encoded and a lighter decoder reconstructs masked values at the point level (Li et al., 2023). Social-MAE similarly uses a transformer as the MAE encoder and a lighter-weight transformer as the decoder for masked multi-person joint trajectories (Ehsanpour et al., 2024). DMAE follows the same asymmetric template for images, with a stronger encoder and smaller decoder, but reconstructs a clean image from an input that is both Gaussian-corrupted and patch-masked (Wu et al., 2022).

Another recurrent pattern is that the encoder processes only the visible tokens or visible regions. This is explicit in GMAE, where masking reduces the number of nodes seen by the encoder and improves memory efficiency (Zhang et al., 2022). The same visible-only encoding principle is used in Ti-MAE for time-series tokens, in MAE3D for visible point-cloud patches, and in AstroMAE for visible image patches (Li et al., 2023, Jiang et al., 2022, Fathkouhi et al., 2024). A learnable mask token is then inserted at masked positions before decoding in several systems, including Lab-MAE, GMAE, MeshMAE, DMAE, and indoor depth completion (Restrepo et al., 5 Jan 2025, Zhang et al., 2022, Liang et al., 2022, Wu et al., 2022, Sun et al., 2024).

This suggests that the defining mechanism is not a single backbone family, but a training geometry: mask visible input, encode sparsely, decode densely. What changes across domains is the representation of a token and the semantics of reconstruction.

2. Masking operators and reconstruction targets

The masking operator is highly domain-specific. Image-oriented systems often mask a large fraction of regular patches. DMAE uses a masking ratio of 0.75 after adding Gaussian noise to the image, while AstroMAE also uses a 75% masking ratio for SDSS image patches (Wu et al., 2022, Fathkouhi et al., 2024). Ti-MAE reports that around 75% masking is the best-performing setting for embedded time-series tokens (Li et al., 2023). By contrast, Lab-MAE uses random masking during training with a mask ratio of 0.25, and Social-MAE uses Tube Masking with a masking ratio of 50% over entire joint trajectories (Restrepo et al., 5 Jan 2025, Ehsanpour et al., 2024). In sparse outdoor LiDAR, Voxel-MAE masks 70% of non-empty voxels and also samples about 10% empty voxels so that the decoder must learn occupancy as well as geometry (Hess et al., 2022). MeshMAE reports that 50% masking works best overall, while MAE3D uses a masking ratio of 0.7 and finds block masking better than random masking (Liang et al., 2022, Jiang et al., 2022). MARMOT introduces a scanning pattern mask (SPM) in which the unmasked subset is functionally equivalent to arbitrary sampling for transient measurements (Shen et al., 10 Jun 2025).

The target of reconstruction also varies substantially.

Paper	Masked unit	Reconstruction target
Lab-MAE (Restrepo et al., 5 Jan 2025)	Sequential lab values	Masked continuous lab values
Ti-MAE (Li et al., 2023)	Embedded timestamps	Point-level time-series values
Social-MAE (Ehsanpour et al., 2024)	Entire joint trajectories	Ground-truth joints trajectory
DMAE (Wu et al., 2022)	Image patches after Gaussian corruption	Original clean image on all patches
Voxel-MAE (Hess et al., 2022)	Non-empty and sampled empty voxels	Geometry, point density, occupancy
MeshMAE (Liang et al., 2022)	Mesh patches	Vertex coordinates and face features
MAE3D (Jiang et al., 2022)	Point-cloud patches	Coarse centers and full point cloud
Indoor depth completion (Sun et al., 2024)	RGB-D patches	Depth, with RMSE over non-zero pixels

These differences are not cosmetic. Voxel-MAE uses a composite loss with Chamfer distance, smooth L1 for point counts, and binary cross entropy for occupancy because sparse automotive LiDAR contains many empty voxels and strongly varying density (Hess et al., 2022). MeshMAE combines MSE on face-wise features with a Chamfer-distance term on relative vertex coordinates (Liang et al., 2022). MAE3D uses a multi-task Chamfer-distance objective over coarse patch centers and the reconstructed full point cloud (Jiang et al., 2022). Indoor depth completion computes pretraining RMSE only over non-zero pixels in the true depth image, explicitly ignoring invalid depth values (Sun et al., 2024). DMAE computes reconstruction loss on all patches, not only masked ones, because visible patches are noisy and must be denoised as well as inpainted (Wu et al., 2022).

This suggests that “masked reconstruction” is best understood as a family of pretext tasks rather than a single loss template.

3. Transformer backbones and modality-specific adaptation

A second recurrent theme is that the Transformer is rarely used in a completely modality-agnostic way. Instead, the encoder-decoder is coupled to a domain-specific tokenizer or backbone.

In vision, ConvMAE replaces the pure ViT encoder with a three-stage hybrid convolution-transformer encoder, using masked convolution and block-wise masking to prevent information leakage and reduce the pretraining-finetuning discrepancy (Gao et al., 2022). For low-level image restoration, MAEIP is built around CSformer, a hierarchical U-shaped Transformer with channel attention, shifted-window self-attention, and a gated convolutional feed-forward network (Duan et al., 2023). For LDCT denoising, the masked autoencoder is built on SwinIR, using window-based multi-head attention and a denoising-oriented encoder-decoder redesign (Wang et al., 2022). MAST also uses a Swin-based design, but in a three-path Swin-Unet that jointly processes time-domain, frequency-domain, and magnitude-based HD-sEMG representations under four masking strategies (Laamerad et al., 2024).

Outside images, tokenization becomes more structured. Lab-MAE jointly models continuous lab values and their timestamps through a structured encoding scheme and a missing-value attention mask (Restrepo et al., 5 Jan 2025). Social-MAE represents multi-person motion in the frequency domain using the Discrete Cosine Transform and masks complete joint trajectories rather than scattered time steps (Ehsanpour et al., 2024). GMAE keeps the graph structure available through Graphormer structural encodings while masking only node features (Zhang et al., 2022). Voxel-MAE uses the Single-stride Sparse Transformer (SST) over non-empty voxel tokens, with regional grouping and regional shift to control attention cost (Hess et al., 2022). MeshMAE constructs non-overlapping local patches of equal face count through remeshing and subdivision, then uses 3D patch-center coordinates as positional embeddings (Liang et al., 2022). MAE3D partitions a point cloud into patches with farthest point sampling and k-nearest neighbors before applying patch-wise transformer modeling (Jiang et al., 2022). MARMOT uses a Transformer-based encoder-decoder specialized for transient imaging and pretrains on diverse NLOS transients (Shen et al., 10 Jun 2025).

A plausible implication is that masked autoencoding does not remove the need for inductive bias; rather, it relocates that bias into token construction, positional encoding, masking geometry, and reconstruction heads.

4. Pretraining, fine-tuning, and transfer protocols

These systems are usually presented as pretraining frameworks rather than task-isolated autoencoders. MARMOT is pretrained on TransVerse-a, described as a synthesized transient dataset of 500K 3D models, and then adapts to downstream imaging tasks through direct feature transfer or decoder finetuning (Shen et al., 10 Jun 2025). Social-MAE replaces the pretraining decoder with task-specific decoders for pose forecasting, social grouping, and social action understanding (Ehsanpour et al., 2024). GMAE discards the decoder after pretraining and fine-tunes the encoder for downstream graph classification, node classification, or molecular regression (Zhang et al., 2022). AstroMAE pretrains a ViT encoder on unlabeled SDSS images and then fine-tunes it inside a specialized redshift-prediction architecture that concatenates pretrained transformer features, an Inception branch, and a magnitude block (Fathkouhi et al., 2024).

Several papers emphasize sample efficiency. Voxel-MAE reports that, with only 40% of the labeled data, a pretrained model already matches or exceeds the full-data random-initialization baseline on nuScenes (Hess et al., 2022). The HPGe waveform study reports that MAE pre-training improves sample efficiency, reducing labeled-data requirements by factors of 2–4 in low-label regimes (Babicz et al., 6 Mar 2026). Lab-MAE is explicitly described as a foundation laboratory imputation model and is evaluated as a single multi-feature model rather than one model per lab (Restrepo et al., 5 Jan 2025). Ti-MAE is positioned as a pretraining framework that can support both forecasting and classification, and the paper emphasizes that the model can be pretrained once and reused across different forecast horizons (Li et al., 2023).

This transfer-oriented framing also changes the role of the decoder. In most of these works, the decoder is instrumental during pretraining and disposable during downstream deployment. Exceptions exist, especially in image restoration or depth completion, where the same or closely related encoder-decoder structure is retained for supervised reconstruction (Duan et al., 2023, Sun et al., 2024, Wang et al., 2022).

5. Representative empirical results

Across the cited papers, masked transformer-based autoencoders are reported to improve downstream performance over supervised training from scratch or over non-masked baselines, but the gains are modality-dependent.

In vision pretraining, ConvMAE reports that ConvMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared with MAE-Base. On object detection, ConvMAE-Base finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP (Gao et al., 2022). For certified robustness, DMAE reports a large gap between plain MAE and denoising masked autoencoding: at noise level $\sigma = 0.25$, certified accuracy at radius $r = 0.5$ is 13.3% for MAE and 45.3% for DMAE (Wu et al., 2022).

In sparse 3D perception, Voxel-MAE improves SST on nuScenes from 53.39 mAP / 62.95 NDS to 55.14 mAP / 64.00 NDS when intensity information is used, and from 49.08 mAP / 60.75 NDS to 51.95 mAP / 62.16 NDS with 10 sweeps (Hess et al., 2022). MAE3D reports 93.4% classification accuracy on ModelNet40 and 86.2% on ScanObjectNN (PB_T50_RS) after pretraining on ShapeNet55 (Jiang et al., 2022). MeshMAE reports 92.5 accuracy on Manifold40 when pretrained on ShapeNet, compared with 91.5 for the Transformer without pre-training (Liang et al., 2022).

In structured sequential data, Lab-MAE is better than XGBoost on RMSE in 89 out of 100 lab values, on $R^2$ in 89 out of 100 lab values, and on Wasserstein Distance in 79 out of 100 lab values on MIMIC-IV (Restrepo et al., 5 Jan 2025). Ti-MAE reports an Avg. Accuracy of 0.8231 and Avg. Rank of 2.054 on the UCR Archive, while also improving forecasting metrics across ETT, Weather, Exchange, and ILI (Li et al., 2023). GMAE reports state-of-the-art performance on 5 of 7 graph classification datasets and 5 of 6 node classification datasets, and reaches 0.120 ± 0.003 MAE on ZINC versus 0.122 ± 0.006 for Graphormer (Zhang et al., 2022).

In human motion and healthcare-related sensing, Social-MAE reports 48.6 overall VIM on SoMoF, compared with 50.4 for the supervised baseline of the same architecture, and improves JRDB-Act grouping and action-understanding mAP on both validation and test splits (Ehsanpour et al., 2024). MAST reports 0.973 ± 0.0008 on CapgMyo DB-a and 0.562 ± 0.007 on DB-b inter-session, with MAE pretraining improving accuracy by 4% on DB-a and 6% on DB-b (Laamerad et al., 2024). In HPGe waveform modeling, the fine-tuned transformer reports combined PSD-pass accuracy 0.9551, F1 0.9415, and AUROC 0.9918, compared with 0.8967, 0.8733, and 0.9598 for the GBDT baseline (Babicz et al., 6 Mar 2026).

Results are not uniformly monotonic on every metric. In indoor depth completion, fine-tuning without pre-training gives RMSE = 0.660, while fine-tuning with pre-training gives RMSE = 0.690 but stronger ME = 0.206, SSIM = 0.765, $\delta_{1.25} = 0.852$, and $\delta_{1.25^2} = 0.912$ than the no-pretraining variant (Sun et al., 2024). This indicates that masked pretraining can improve structural or threshold-based measures even when a single error measure moves differently.

6. Misconceptions, limitations, and unresolved design questions

One common misconception is that masked autoencoding is primarily an image-pretraining method. The papers summarized here contradict that view directly by adapting the paradigm to graphs, time series, laboratory trajectories, LiDAR voxels, meshes, point-cloud patches, transient imaging, motion trajectories, waveform windows, and RGB-D depth completion (Zhang et al., 2022, Li et al., 2023, Restrepo et al., 5 Jan 2025, Hess et al., 2022, Liang et al., 2022, Shen et al., 10 Jun 2025).

A second misconception is that a larger decoder is necessarily better. Several studies argue the opposite for their regimes. SDMAE states that the standard MAE decoder is too complex for small datasets and selects decoder depth = 1 and decoder embedding dimension = 128 as the preferred configuration (Mao et al., 2022). Ti-MAE reports that a small decoder is sufficient and that 2 decoder blocks with decoder hidden size around 32 work best (Li et al., 2023). GMAE and Social-MAE also retain the standard asymmetry of a deep encoder and shallow decoder (Zhang et al., 2022, Ehsanpour et al., 2024).

A third unresolved question concerns whether masked reconstruction alone is enough. SDMAE adds location prediction and contrastive learning to introduce localization, invariance, and class-token training on small datasets (Mao et al., 2022). DMAE adds Gaussian corruption and reconstructs the clean image to learn robust semantics for randomized smoothing (Wu et al., 2022). Voxel-MAE shows that a naive MAE-style reconstruction using only Chamfer loss can hurt detection performance, and that occupancy-aware and density-aware objectives are important in sparse LiDAR (Hess et al., 2022). These cases do not refute masked autoencoding; rather, they suggest that the pretext task may need to reflect the failure modes of a modality.

Several papers also state explicit limitations. Lab-MAE is evaluated on a single academic medical center dataset and notes limited generalizability and the need for intersectional fairness analysis (Restrepo et al., 5 Jan 2025). Social-MAE relies only on pose and motion, not image or video appearance, and reports failures when people move similarly and are close in 2D (Ehsanpour et al., 2024). MeshMAE notes that segmentation gains are smaller because patch embeddings are less suitable for dense per-face prediction (Liang et al., 2022). The HPGe waveform study observes a small common underestimation bias in energy regression and states that gains in PSD should not yet be interpreted as a quantified half-life sensitivity improvement for $0\nu\beta\beta$ searches (Babicz et al., 6 Mar 2026). In indoor depth completion, pretraining improves several structural metrics but does not dominate RMSE (Sun et al., 2024).

Taken together, these papers depict masked transformer-based autoencoding as a broad self-supervised design pattern rather than a fixed model class. Its central invariants are sparse visible-token encoding, masked reconstruction, and transfer of pretrained encoders; its principal variables are tokenization, masking geometry, reconstruction target, auxiliary objectives, and the degree of domain-specific inductive bias incorporated into the Transformer pipeline.