---
title: Local Masked Autoencoder (LMAE)
url: https://www.emergentmind.com/topics/local-masked-autoencoder-lmae
type: topic
---

# Local Masked Autoencoder (LMAE)

Local Masked Autoencoder (LMAE) denotes a class of masked autoencoding methods in which reconstruction is constrained by locality: masked content is predicted from neighborhood context, local windows, local surface patches, or local sub-volumes rather than from unrestricted global context. The term is not fully standardized across the literature. In vision, it functions primarily as a conceptual umbrella for local or window-based masked image modeling; in 3D point clouds, MAE3D is an explicit instantiation built on masked local surface patches; in volumetric medical imaging, local masked reconstruction serves as the local component later extended by Global-Local MAE. A distinct acronym, L-MAE, refers to Label Mask AutoEncoder for semantic label completion and is not a Local Masked Autoencoder [2208.00173] [2207.01545] [2306.08913] [2211.11242].

## 1. Definition, scope, and locality as a design principle

Within masked image modeling and related self-supervised paradigms, locality is enforced by modifying the masking pattern, the encoder’s attention field, the reconstruction pathway, or the tokenization scheme itself. The survey literature identifies local/window-based variants such as LoMaR, group window attention for hierarchical vision transformers, UM-MAE, HiViT, ObjMAE, block-wise masking in methods such as BEiT and SimMIM, and tube or cube masking in video masked modeling [2208.00173].

This framing places LMAE closer to a family resemblance than to a single canonical architecture. In some works, locality means reconstructing only within small sampled windows, as in LoMaR. In others, it means masking contiguous local regions, as in MAE3D block masking over point-cloud patches. In volumetric medical imaging, locality is tied to cropped local views of a 3D scan. The common premise is that reconstruction should be driven by nearby structure when the modality itself is spatially local, irregular, or computationally prohibitive for global attention [2206.00790] [2207.01545] [2306.08913].

The motivation is twofold. First, locality reduces compute and memory by restricting token interactions. Second, it can strengthen feature learning when discriminative information is concentrated in local neighborhoods, as with surface geometry in point clouds, organ boundaries in medical volumes, or patch-level texture and shape cues in images. The survey also emphasizes compatibility with hierarchical vision transformers whose inductive bias is already local window attention, making locality-aware masking a practical adaptation of MAE-style pretraining rather than a purely conceptual variation [2208.00173].

## 2. Core architectural pattern and reconstruction objectives

A generic masked autoencoding pipeline patchifies an input into tokens, applies a binary mask, encodes visible content, and reconstructs targets for masked positions. Local variants augment this formulation by partitioning tokens into windows or neighborhoods $W=\{W_k\}$ and restricting encoding or decoding to those subsets. The survey summarizes the local reconstruction objective as
$$
L_{\text{local}}=\frac{1}{\sum_k |M_k|}\sum_k \sum_{i \in (M \cap W_k)} \left\| f_{\text{dec}}(f_{\text{enc}}(W_k); i)-t_i \right\|_2^2,
$$
where $M_k=M \cap W_k$ denotes masked indices within each local region [2208.00173].

Concrete LMAE systems diverge in how they realize this template. LoMaR samples several square windows of size $w \times w$ patches, masks a large fraction within each window, feeds both visible and masked tokens from those windows into a ViT encoder, and reconstructs masked pixels with a single-layer MLP head. Its complexity is reported as $O(hw + v m^4)$, in contrast to the $O(P^2)$ global attention cost of standard MAE and BEiT, with fixed local window size making scaling substantially more resolution-friendly [2206.00790].

By contrast, MAE3D follows the asymmetric visible-only encoder pattern more closely. It partitions a point cloud into patch-wise tokens, excludes masked patches from the encoder, and restores them only in the decoder through learnable mask tokens enriched with positional embeddings. In that setting, locality arises from surface-patch tokenization, block masking, and patch-wise reconstruction rather than from 2D window attention [2207.01545].

This variation in design implies that locality is not identical to any single encoder–decoder topology. Some local methods keep the MAE asymmetry; others feed masked and visible tokens together to a single encoder; some use transformer decoders, while others use an MLP reconstruction head. The survey’s synthesis suggests that locality often permits lighter decoders because the reconstruction problem is conditioned on stronger neighborhood priors than in global masked modeling [2208.00173].

## 3. MAE3D as a local masked autoencoder for 3D point clouds

MAE3D is a direct instantiation of local masked autoencoding for irregular 3D point sets. Its premise is that point clouds are sparse and unordered, and that the most discriminative signal lies in local surface neighborhoods. The method therefore masks entire local patches rather than globally regular tokens, forcing the encoder to learn fine-grained geometric priors and the decoder to infer missing local geometry from nearby context [2207.01545].

The preprocessing pipeline begins with a point cloud $P \in \mathbb{R}^{N \times 3}$. For pretraining on ShapeNet55, $N=2{,}048$. Farthest Point Sampling selects $S$ patch centers $P_c=\{c_i\} \in \mathbb{R}^{S \times 3}$ with $S=32$, and KNN retrieves a local neighborhood $P_i$ of $K$ points around each center, with $K=64$. A trainable MLP $\Phi$ maps each center coordinate to a positional embedding $\phi_i$. Visible patches are embedded by a patch feature extractor $\Psi$, instantiated by early PointNet or DGCNN layers before global pooling, and the patch token is formed as $x_i=\text{concat}(\psi_i,\phi_i)$ [2207.01545].

Two masking strategies are evaluated: random masking and block masking. The latter chooses a random center and removes its nearest centers’ patches, producing a contiguous blank region. Pretraining uses a masking ratio of $0.7$, so with $S=32$ only about $10$ patches remain visible. For masked patches, MAE3D uses learnable mask tokens $\widetilde{\xi}_i=\text{concat}(\widetilde{\psi}_i,\phi_i)$, and the true masked points are not used anywhere in encoding or decoding. This design encourages recovery from local context rather than direct leakage of masked geometry [2207.01545].

The encoder processes only visible tokens, together with a class token $\Theta_0$. The decoder receives visible latents plus mask tokens, restored to the original patch order, and completes the latent representation of masked patches. Reconstruction proceeds in two stages. First, MLPs regress a coarse point cloud $\widetilde{P}_c$ of $S=32$ predicted patch centers. Second, for each center, a folding decoder deforms a regular $8 \times 8$ 2D grid into a $64$-point surface patch. The reconstructed patches are concatenated into the full point cloud $\widetilde{P}$, with folding used to preserve local surface continuity and reduce redundancy relative to naive per-point MLP regression [2207.01545].

The training objective is a multi-task Chamfer Distance loss,
$$
L = CD(\widetilde{P}_c, P_c) + \alpha \cdot CD(\widetilde{P}, P).
$$
The first term aligns predicted and true patch centers; the second aligns the dense reconstruction. Ablations show that including the center term lowers Chamfer Distance from $4.034 \times 10^{-3}$ to $3.127 \times 10^{-3}$. Block masking at ratio $0.7$ outperforms random masking both for reconstruction and downstream classification, with CD $3.127 \times 10^{-3}$ versus $3.260 \times 10^{-3}$ and top-1 accuracy $93.4\%$ versus $93.2\%$. The reported downstream results are $93.4\%$ on ModelNet40 and $86.2\%$ on ScanObjectNN PB\_T50\_RS, and pretraining with MAE3D transformers improves downstream accuracy by about $1\%$ over removing them. Fine-tuning initializes only the patch feature extractor $\Psi$, not the heavy transformer encoder, which the paper reports as more lightweight and sometimes similar or better in accuracy than loading a heavy encoder [2207.01545].

## 4. Local-window image pretraining and the local contrastive interpretation

In 2D vision, LoMaR is a concrete realization of LMAE in which masked reconstruction is performed only inside randomly sampled local windows. For an image token grid, LoMaR samples several square windows $W_k$ of size $w \times w$, with the best setting reported as $w=7$, masks a large fraction of patches in each window, with the best ratio reported as $r=80\%$, and reconstructs only those masked patches from visible patches in the same window. Unlike canonical MAE, LoMaR feeds both visible and masked tokens from sampled windows to a single ViT encoder, uses contextual relative positional encoding, and replaces the heavy decoder with a single-layer MLP reconstruction head [2206.00790].

The efficiency argument is central. Global self-attention in MAE and BEiT scales as $O(P^2)$ in the total number of image patches, whereas LoMaR reports end-to-end complexity $O(hw + v m^4)$ for fixed local windows. On ImageNet-1K with ViT-B/16 at $224 \times 224$, LoMaR reaches $83.6\%$ top-1 at $400$ epochs, $83.8\%$ at $800$, and $84.1\%$ at $1600$, compared with MAE at $83.1\%$, $83.3\%$, and $83.6\%$. After fine-tuning on $384 \times 384$ images, LoMaR reaches $85.4\%$ top-1, surpassing MAE by $0.6\%$. On MS COCO, LoMaR improves over a reproduced MAE baseline by up to $0.5 \, \text{AP}^{\text{box}}$ and $0.5 \, \text{AP}^{\text{mask}}$, and on high-resolution pretraining it is reported as $2.5\times$ faster than MAE at $384 \times 384$ and $3.1\times$ faster at $448 \times 448$, while maintaining slightly better classification accuracy [2206.00790].

A later analytical work, LC-MAE, reinterprets masked autoencoding itself from a local contrastive perspective. It derives a patch-level view in which the reconstructive objective implicitly promotes cross-mask alignment of the same patch index and preservation of patch-to-patch relations. LC-MAE makes this explicit by combining the MAE reconstruction loss with a cross-view local alignment loss
$$
L_{\text{cross}} = \sum_{i=1}^n (m_{1,i} \cdot m_{2,i}) \cdot \left(1-\cos(v_i^{m_1}, v_i^{m_2})\right)
$$
and an in-view distribution-consistency loss
$$
L_{\text{in}} = \sum_{i \neq j} (m_i \cdot m_j)\cdot \left|\cos(v_i, v_j)-\cos(x_i, x_j)\right|.
$$
On ImageNet-1K with ViT-B/16 and $100$-epoch pretraining, MAE reaches $82.9\%$ finetune accuracy, LC-MAE reaches $83.0\%$, MAE plus cross-view reaches $83.1\%$, and cross-view alone collapses. The same paper reports that a weighted-average local decoder with an MLP head reaches $82.5\%$, a single $5 \times 5$ convolutional decoder reaches $82.9\%$, and a transformer decoder also reaches $82.9\%$, supporting the claim that MAE’s decoder primarily exploits local neighborhoods rather than requiring strong global context [2310.01994].

Taken together, these results support a broad interpretation of LMAE in image pretraining: locality can be imposed architecturally, as in LoMaR’s local windows, or revealed analytically, as in LC-MAE’s patch-level alignment and effective receptive field restriction.

## 5. Local masked autoencoding in volumetric medical imaging

In volumetric medical imaging, a local masked autoencoder is defined on cropped local sub-volumes of a 3D scan. A high-resolution local view $v_l \in \mathbb{R}^{C \times H \times W \times D}$ is tokenized into 3D patches, masking is applied patch-wise, visible tokens are encoded, and a decoder reconstructs the missing voxels, typically with MSE in voxel space. The paper on GL-MAE identifies two limitations of such pure local masked reconstruction: lack of global clinical context and instability when heavy masking is applied to small local fields of view [2306.08913].

GL-MAE extends this local-only regime by adding masked global reconstruction and two consistency terms anchored by a complete unmasked global view. Local views are sampled at random scale in $[0.25, 0.5]$ and resized to $[96,96,96]$; global views are sampled at scale in $[0.5,1]$ and resized to $[160,160,160]$. The method uses $q=8$ local views and $p=2$ global views per input volume. A student encoder processes masked local and masked global views, while a momentum teacher encodes the unmasked global anchor. Projection heads map class-token features into a shared embedding space of dimension $K=512$, and the total loss combines local reconstruction, global reconstruction, global-to-global consistency, and global-to-local consistency with weights $\beta_1=\beta_2=\beta_3=1.0$ [2306.08913].

The reported objective is
$$
\mathcal{L}_{\text{total}}=\mathcal{L}_{R}^{l}+\beta_1 \mathcal{L}_{R}^{g}+\beta_2 \mathcal{L}_{C}^{gg}+\beta_3 \mathcal{L}_{C}^{gl}.
$$
Ablations identify mask ratio $0.6$ as best, while $0.7$ is described as too high for stable reconstruction. Pretraining uses AdamW for $1600$ epochs with initial learning rate $1 \times 10^{-2}$, batch size $256$, and $4 \times 3090\text{Ti}$ GPUs [2306.08913].

Quantitatively, the local-only baseline in this study is MAE3D-style local masked reconstruction. With a ViT-B backbone and end-to-end fine-tuning, Dice on BTCV rises from $79.61$ for the supervised UNETR baseline to $81.74$ for MAE3D and $82.33$ for GL-MAE; on MSD Spleen, from $94.20$ to $95.20$ to $95.72$; and on MM-WHS, from $83.85$ to $86.03$ to $88.88$. On BTCV with a ViT-T backbone, linear evaluation improves from $78.36$ for MAE3D to $80.22$ for GL-MAE, and end-to-end Dice improves from $80.21$ to $81.02$. The paper therefore positions pure LMAE as effective for fine-grained local detail, but incomplete for tasks requiring coherent anatomical context across large 3D extents [2306.08913].

## 6. Related variants, nomenclature collisions, and common misconceptions

A common misconception is that “LMAE” names a single universally accepted model. The survey literature does not standardize the exact term, but it consistently groups local/window-based masked modeling methods as a recognizable design family. These include LoMaR, group window attention in Green Hierarchical Transformer, UM-MAE, HiViT, ObjMAE, block-wise masking strategies in SimMIM and BEiT, and tube or cube masking in video methods such as VideoMAE, MaskFeat, BEVT, and VIMPAC [2208.00173]. This suggests that “Local Masked Autoencoder” is best understood as a locality-constrained masked autoencoding paradigm rather than the proper name of one model.

A second source of confusion is the acronym L-MAE. In the semantic segmentation paper “L-MAE: Masked Autoencoders are Semantic Segmentation Datasets Augmenter,” L-MAE stands for Label Mask AutoEncoder or Label Masked Autoencoder, not Local Masked Autoencoder. That method masks patches on a fused image-and-label representation, reconstructs labels rather than pixels, re-inserts image information through Image Patch Supplement, and reports an average $4.1\%$ PA-mIoU improvement from IPS and roughly $13.5\%$ downstream improvement when completed labels are used to retrain conventional segmentation models on a degraded Pascal VOC setup [2211.11242].

A third misconception is that locality necessarily means the loss of all global semantics. The literature instead presents trade-offs. MAE3D combines local surface reconstruction with patch-to-patch contextual learning. LoMaR restricts attention to local windows but maintains competitive or بہتر downstream performance relative to global MAE. GL-MAE explicitly augments local masked reconstruction with global views and consistency constraints when global anatomical context is indispensable. The recurring pattern is not rejection of global information, but selective control over when and how global context is introduced [2207.01545] [2206.00790] [2306.08913].

In that sense, LMAE occupies a specific place within the broader masked autoencoding landscape: it emphasizes neighborhood-conditioned reconstruction, local geometric or spatial priors, and often lighter or more compute-efficient reconstruction pathways. The exact implementation varies by modality, but the underlying principle remains the same—masked prediction is organized around locality because the signal, the inductive bias, or the computational constraint of the problem demands it.

Source: https://www.emergentmind.com/topics/local-masked-autoencoder-lmae