Local Masked Autoencoder (LMAE)

• Local Masked Autoencoder (LMAE) is a paradigm where masked reconstruction is driven solely by local context, using nearby patches or tokens for prediction.
• It is implemented in various modalities such as LoMaR for images, MAE3D for 3D point clouds, and GL-MAE for volumetric medical imaging, each adapting local masking to their domain structure.
• By constraining attention to local windows or patches, LMAE reduces computational cost while strengthening fine-grained feature learning and preserving essential spatial relationships.

Local Masked Autoencoder (LMAE) denotes a class of masked autoencoding methods in which reconstruction is constrained by locality: masked content is predicted from neighborhood context, local windows, local surface patches, or local sub-volumes rather than from unrestricted global context. The term is not fully standardized across the literature. In vision, it functions primarily as a conceptual umbrella for local or window-based masked image modeling; in 3D point clouds, MAE3D is an explicit instantiation built on masked local surface patches; in volumetric medical imaging, local masked reconstruction serves as the local component later extended by Global-Local MAE. A distinct acronym, L-MAE, refers to Label Mask AutoEncoder for semantic label completion and is not a Local Masked Autoencoder (Zhang et al., 2022, Jiang et al., 2022, Zhuang et al., 2023, Jia et al., 2022).

1. Definition, scope, and locality as a design principle

Within masked image modeling and related self-supervised paradigms, locality is enforced by modifying the masking pattern, the encoder’s attention field, the reconstruction pathway, or the tokenization scheme itself. The survey literature identifies local/window-based variants such as LoMaR, group window attention for hierarchical vision transformers, UM-MAE, HiViT, ObjMAE, block-wise masking in methods such as BEiT and SimMIM, and tube or cube masking in video masked modeling (Zhang et al., 2022).

This framing places LMAE closer to a family resemblance than to a single canonical architecture. In some works, locality means reconstructing only within small sampled windows, as in LoMaR. In others, it means masking contiguous local regions, as in MAE3D block masking over point-cloud patches. In volumetric medical imaging, locality is tied to cropped local views of a 3D scan. The common premise is that reconstruction should be driven by nearby structure when the modality itself is spatially local, irregular, or computationally prohibitive for global attention (Chen et al., 2022, Jiang et al., 2022, Zhuang et al., 2023).

The motivation is twofold. First, locality reduces compute and memory by restricting token interactions. Second, it can strengthen feature learning when discriminative information is concentrated in local neighborhoods, as with surface geometry in point clouds, organ boundaries in medical volumes, or patch-level texture and shape cues in images. The survey also emphasizes compatibility with hierarchical vision transformers whose inductive bias is already local window attention, making locality-aware masking a practical adaptation of MAE-style pretraining rather than a purely conceptual variation (Zhang et al., 2022).

2. Core architectural pattern and reconstruction objectives

A generic masked autoencoding pipeline patchifies an input into tokens, applies a binary mask, encodes visible content, and reconstructs targets for masked positions. Local variants augment this formulation by partitioning tokens into windows or neighborhoods $W=\{W_k\}$ and restricting encoding or decoding to those subsets. The survey summarizes the local reconstruction objective as

$$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$

where $M_k=M \cap W_k$ denotes masked indices within each local region (Zhang et al., 2022).

Concrete LMAE systems diverge in how they realize this template. LoMaR samples several square windows of size $w \times w$ patches, masks a large fraction within each window, feeds both visible and masked tokens from those windows into a ViT encoder, and reconstructs masked pixels with a single-layer MLP head. Its complexity is reported as $O(hw + v m^4)$, in contrast to the $O(P^2)$ global attention cost of standard MAE and BEiT, with fixed local window size making scaling substantially more resolution-friendly (Chen et al., 2022).

By contrast, MAE3D follows the asymmetric visible-only encoder pattern more closely. It partitions a point cloud into patch-wise tokens, excludes masked patches from the encoder, and restores them only in the decoder through learnable mask tokens enriched with positional embeddings. In that setting, locality arises from surface-patch tokenization, block masking, and patch-wise reconstruction rather than from 2D window attention (Jiang et al., 2022).

This variation in design implies that locality is not identical to any single encoder–decoder topology. Some local methods keep the MAE asymmetry; others feed masked and visible tokens together to a single encoder; some use transformer decoders, while others use an MLP reconstruction head. The survey’s synthesis suggests that locality often permits lighter decoders because the reconstruction problem is conditioned on stronger neighborhood priors than in global masked modeling (Zhang et al., 2022).

3. MAE3D as a local masked autoencoder for 3D point clouds

MAE3D is a direct instantiation of local masked autoencoding for irregular 3D point sets. Its premise is that point clouds are sparse and unordered, and that the most discriminative signal lies in local surface neighborhoods. The method therefore masks entire local patches rather than globally regular tokens, forcing the encoder to learn fine-grained geometric priors and the decoder to infer missing local geometry from nearby context (Jiang et al., 2022).

The preprocessing pipeline begins with a point cloud $P \in \mathbb{R}^{N \times 3}$. For pretraining on ShapeNet55, $N=2{,}048$. Farthest Point Sampling selects $S$ patch centers $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$ with $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$0, and KNN retrieves a local neighborhood $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$1 of $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$2 points around each center, with $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$3. A trainable MLP $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$4 maps each center coordinate to a positional embedding $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$5. Visible patches are embedded by a patch feature extractor $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$6, instantiated by early PointNet or DGCNN layers before global pooling, and the patch token is formed as $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$7 (Jiang et al., 2022).

Two masking strategies are evaluated: random masking and block masking. The latter chooses a random center and removes its nearest centers’ patches, producing a contiguous blank region. Pretraining uses a masking ratio of $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$8, so with $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$9 only about $M_k=M \cap W_k$0 patches remain visible. For masked patches, MAE3D uses learnable mask tokens $M_k=M \cap W_k$1, and the true masked points are not used anywhere in encoding or decoding. This design encourages recovery from local context rather than direct leakage of masked geometry (Jiang et al., 2022).

The encoder processes only visible tokens, together with a class token $M_k=M \cap W_k$2. The decoder receives visible latents plus mask tokens, restored to the original patch order, and completes the latent representation of masked patches. Reconstruction proceeds in two stages. First, MLPs regress a coarse point cloud $M_k=M \cap W_k$3 of $M_k=M \cap W_k$4 predicted patch centers. Second, for each center, a folding decoder deforms a regular $M_k=M \cap W_k$5 2D grid into a $M_k=M \cap W_k$6-point surface patch. The reconstructed patches are concatenated into the full point cloud $M_k=M \cap W_k$7, with folding used to preserve local surface continuity and reduce redundancy relative to naive per-point MLP regression (Jiang et al., 2022).

The training objective is a multi-task Chamfer Distance loss,

$M_k=M \cap W_k$8

The first term aligns predicted and true patch centers; the second aligns the dense reconstruction. Ablations show that including the center term lowers Chamfer Distance from $M_k=M \cap W_k$9 to $w \times w$0. Block masking at ratio $w \times w$1 outperforms random masking both for reconstruction and downstream classification, with CD $w \times w$2 versus $w \times w$3 and top-1 accuracy $w \times w$4 versus $w \times w$5. The reported downstream results are $w \times w$6 on ModelNet40 and $w \times w$7 on ScanObjectNN PB_T50_RS, and pretraining with MAE3D transformers improves downstream accuracy by about $w \times w$8 over removing them. Fine-tuning initializes only the patch feature extractor $w \times w$9, not the heavy transformer encoder, which the paper reports as more lightweight and sometimes similar or better in accuracy than loading a heavy encoder (Jiang et al., 2022).

4. Local-window image pretraining and the local contrastive interpretation

In 2D vision, LoMaR is a concrete realization of LMAE in which masked reconstruction is performed only inside randomly sampled local windows. For an image token grid, LoMaR samples several square windows $O(hw + v m^4)$0 of size $O(hw + v m^4)$1, with the best setting reported as $O(hw + v m^4)$2, masks a large fraction of patches in each window, with the best ratio reported as $O(hw + v m^4)$3, and reconstructs only those masked patches from visible patches in the same window. Unlike canonical MAE, LoMaR feeds both visible and masked tokens from sampled windows to a single ViT encoder, uses contextual relative positional encoding, and replaces the heavy decoder with a single-layer MLP reconstruction head (Chen et al., 2022).

The efficiency argument is central. Global self-attention in MAE and BEiT scales as $O(hw + v m^4)$4 in the total number of image patches, whereas LoMaR reports end-to-end complexity $O(hw + v m^4)$5 for fixed local windows. On ImageNet-1K with ViT-B/16 at $O(hw + v m^4)$6, LoMaR reaches $O(hw + v m^4)$7 top-1 at $O(hw + v m^4)$8 epochs, $O(hw + v m^4)$9 at $O(P^2)$0, and $O(P^2)$1 at $O(P^2)$2, compared with MAE at $O(P^2)$3, $O(P^2)$4, and $O(P^2)$5. After fine-tuning on $O(P^2)$6 images, LoMaR reaches $O(P^2)$7 top-1, surpassing MAE by $O(P^2)$8. On MS COCO, LoMaR improves over a reproduced MAE baseline by up to $O(P^2)$9 and $P \in \mathbb{R}^{N \times 3}$0, and on high-resolution pretraining it is reported as $P \in \mathbb{R}^{N \times 3}$1 faster than MAE at $P \in \mathbb{R}^{N \times 3}$2 and $P \in \mathbb{R}^{N \times 3}$3 faster at $P \in \mathbb{R}^{N \times 3}$4, while maintaining slightly better classification accuracy (Chen et al., 2022).

A later analytical work, LC-MAE, reinterprets masked autoencoding itself from a local contrastive perspective. It derives a patch-level view in which the reconstructive objective implicitly promotes cross-mask alignment of the same patch index and preservation of patch-to-patch relations. LC-MAE makes this explicit by combining the MAE reconstruction loss with a cross-view local alignment loss

$P \in \mathbb{R}^{N \times 3}$5

and an in-view distribution-consistency loss

$P \in \mathbb{R}^{N \times 3}$6

On ImageNet-1K with ViT-B/16 and $P \in \mathbb{R}^{N \times 3}$7-epoch pretraining, MAE reaches $P \in \mathbb{R}^{N \times 3}$8 finetune accuracy, LC-MAE reaches $P \in \mathbb{R}^{N \times 3}$9, MAE plus cross-view reaches $N=2{,}048$0, and cross-view alone collapses. The same paper reports that a weighted-average local decoder with an MLP head reaches $N=2{,}048$1, a single $N=2{,}048$2 convolutional decoder reaches $N=2{,}048$3, and a transformer decoder also reaches $N=2{,}048$4, supporting the claim that MAE’s decoder primarily exploits local neighborhoods rather than requiring strong global context (Yue et al., 2023).

Taken together, these results support a broad interpretation of LMAE in image pretraining: locality can be imposed architecturally, as in LoMaR’s local windows, or revealed analytically, as in LC-MAE’s patch-level alignment and effective receptive field restriction.

5. Local masked autoencoding in volumetric medical imaging

In volumetric medical imaging, a local masked autoencoder is defined on cropped local sub-volumes of a 3D scan. A high-resolution local view $N=2{,}048$5 is tokenized into 3D patches, masking is applied patch-wise, visible tokens are encoded, and a decoder reconstructs the missing voxels, typically with MSE in voxel space. The paper on GL-MAE identifies two limitations of such pure local masked reconstruction: lack of global clinical context and instability when heavy masking is applied to small local fields of view (Zhuang et al., 2023).

GL-MAE extends this local-only regime by adding masked global reconstruction and two consistency terms anchored by a complete unmasked global view. Local views are sampled at random scale in $N=2{,}048$6 and resized to $N=2{,}048$7; global views are sampled at scale in $N=2{,}048$8 and resized to $N=2{,}048$9. The method uses $S$0 local views and $S$1 global views per input volume. A student encoder processes masked local and masked global views, while a momentum teacher encodes the unmasked global anchor. Projection heads map class-token features into a shared embedding space of dimension $S$2, and the total loss combines local reconstruction, global reconstruction, global-to-global consistency, and global-to-local consistency with weights $S$3 (Zhuang et al., 2023).

The reported objective is

$S$4

Ablations identify mask ratio $S$5 as best, while $S$6 is described as too high for stable reconstruction. Pretraining uses AdamW for $S$7 epochs with initial learning rate $S$8, batch size $S$9, and $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$0 GPUs (Zhuang et al., 2023).

Quantitatively, the local-only baseline in this study is MAE3D-style local masked reconstruction. With a ViT-B backbone and end-to-end fine-tuning, Dice on BTCV rises from $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$1 for the supervised UNETR baseline to $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$2 for MAE3D and $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$3 for GL-MAE; on MSD Spleen, from $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$4 to $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$5 to $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$6; and on MM-WHS, from $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$7 to $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$8 to $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$9. On BTCV with a ViT-T backbone, linear evaluation improves from $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$00 for MAE3D to $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$01 for GL-MAE, and end-to-end Dice improves from $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$02 to $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$03. The paper therefore positions pure LMAE as effective for fine-grained local detail, but incomplete for tasks requiring coherent anatomical context across large 3D extents (Zhuang et al., 2023).

6. Related variants, nomenclature collisions, and common misconceptions

A common misconception is that “LMAE” names a single universally accepted model. The survey literature does not standardize the exact term, but it consistently groups local/window-based masked modeling methods as a recognizable design family. These include LoMaR, group window attention in Green Hierarchical Transformer, UM-MAE, HiViT, ObjMAE, block-wise masking strategies in SimMIM and BEiT, and tube or cube masking in video methods such as VideoMAE, MaskFeat, BEVT, and VIMPAC (Zhang et al., 2022). This suggests that “Local Masked Autoencoder” is best understood as a locality-constrained masked autoencoding paradigm rather than the proper name of one model.

A second source of confusion is the acronym L-MAE. In the semantic segmentation paper “L-MAE: Masked Autoencoders are Semantic Segmentation Datasets Augmenter,” L-MAE stands for Label Mask AutoEncoder or Label Masked Autoencoder, not Local Masked Autoencoder. That method masks patches on a fused image-and-label representation, reconstructs labels rather than pixels, re-inserts image information through Image Patch Supplement, and reports an average $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$04 PA-mIoU improvement from IPS and roughly $$L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,$$05 downstream improvement when completed labels are used to retrain conventional segmentation models on a degraded Pascal VOC setup (Jia et al., 2022).

A third misconception is that locality necessarily means the loss of all global semantics. The literature instead presents trade-offs. MAE3D combines local surface reconstruction with patch-to-patch contextual learning. LoMaR restricts attention to local windows but maintains competitive or بہتر downstream performance relative to global MAE. GL-MAE explicitly augments local masked reconstruction with global views and consistency constraints when global anatomical context is indispensable. The recurring pattern is not rejection of global information, but selective control over when and how global context is introduced (Jiang et al., 2022, Chen et al., 2022, Zhuang et al., 2023).

In that sense, LMAE occupies a specific place within the broader masked autoencoding landscape: it emphasizes neighborhood-conditioned reconstruction, local geometric or spatial priors, and often lighter or more compute-efficient reconstruction pathways. The exact implementation varies by modality, but the underlying principle remains the same—masked prediction is organized around locality because the signal, the inductive bias, or the computational constraint of the problem demands it.