---
title: Multi-Scale Geometry-Aware Tokenizer (MAGNO)
url: https://www.emergentmind.com/topics/multi-scale-geometry-aware-tokenizer-magno
type: topic
---

# Multi-Scale Geometry-Aware Tokenizer (MAGNO)

Searching arXiv for the cited MAGNO-related papers to ground the article in current literature.
Multi-Scale Geometry-Aware Tokenizer (MAGNO) denotes a tokenizer architecture that converts variable-size geometric inputs into a fixed or bounded latent token set while preserving multi-scale spatial structure and explicit geometric cues. In current usage, the term is most directly instantiated in Shape, where MAGNO converts a normalized CAD surface mesh into a structured \(24 \times 24 \times 24\) latent token grid, and it is also articulated as a tokenizer design pattern in GeoTransolver, where multi-scale ball-query encoders build geometry, global-parameter, and boundary-condition context that is reused across transformer blocks on irregular domains [2604.22826][2512.20399].

## 1. Conceptual lineage and scope

In Shape, MAGNO is the front-end “geometry interface” that turns an arbitrary-resolution CAD surface mesh into a fixed, structured 3D grid of tokens. Shape states that this tokenizer is inherited directly from the Multiscale Attentional Graph Neural Operator encoder of GAOT, but repurposed for self-supervised representation learning rather than PDE surrogate modeling. The resulting pipeline is
\[
\text{surface mesh} \xrightarrow{\text{MAGNO tokenizer}} \text{latent 3D token grid } \mathbf{Z} \xrightarrow{\text{Transformer processor}} \text{embeddings / reconstructions},
\]
with a released configuration of \(T=24^3=13{,}824\) tokens and token dimension \(C=128\) [2604.22826].

GeoTransolver provides a complementary, tokenizer-centric formulation. There, the central design is to separate physical state tokens from geometry, global-parameter, and boundary-condition context; precompute this context via multi-scale ball queries; augment local state tokens with geometry-aware features; and reuse the context in every transformer block through Geometry-Aware Latent Embedding (GALE). Reinterpreted in tokenizer language, MAGNO becomes a mechanism for turning meshes, field samples, and operating conditions into geometry-aware state tokens plus a compact context token set in a shared latent “physical state” space [2512.20399].

Earlier and adjacent work shows that the MAGNO label names a broader design family rather than a single canonical block. The 2023 “Multi-scale Geometry-aware Transformer” for point cloud classification divided point clouds into multi-scale patches, used a sphere-mapping local feature extractor, and applied geodesic-based self-attention, thereby establishing an explicit multi-scale and geometry-aware tokenization logic for unordered 3D data [2304.05694]. This suggests that MAGNO is best understood as a research program centered on geometry-aware token formation, not merely a single implementation.

## 2. Representational primitives and token types

A MAGNO-style tokenizer typically begins by distinguishing local signals from geometric context. In GeoTransolver, for slice \(m\), the local inputs are
\[
X_m = \{(x_{m,i}, f_{m,i})\}_{i=1}^{N_m}, \qquad x_{m,i}\in\mathbb{R}^3,\quad f_{m,i}\in\mathbb{R}^{d_x},
\]
while geometry is represented as
\[
\mathcal{G} = \{(g_j,\gamma_j)\}_{j=1}^{M_g}, \qquad g_j\in\mathbb{R}^3,\quad \gamma_j\in\mathbb{R}^{d_g},
\]
and global parameters are
\[
p \in \mathbb{R}^{d_p}.
\]
In tokenizer terms, these correspond respectively to state tokens, geometry tokens, and global or boundary-condition tokens [2512.20399].

Shape specializes this abstraction to CAD surface meshes. Each mesh is canonicalized to a unit bounding box, sampled at \(N=8{,}192\) surface points, and equipped with coordinates \(\mathbf{P}\in\mathbb{R}^{N\times 3}\), normals \(\mathbf{n}\in\mathbb{R}^{N\times 3}\), and scalar curvature \(\kappa\in\mathbb{R}^{N}\). For each latent grid cell \(t\), MAGNO computes a 28-dimensional raw geometric signature \(\mathbf{Y}_t\) from mean, standard deviation, minimum, and maximum over three groups within a physical neighborhood: relative positions \(\mathbf{P}_i-\mathbf{Q}_t\), normals \(\mathbf{n}_i\), and curvature \(\kappa_i\). The dimensional decomposition is \(12+12+4=28\), and the paper identifies this vector both as an initializer or enrichment for token embeddings and as the masked-token reconstruction target during pretraining [2604.22826].

These two formulations reveal two recurrent token types. The first is the dense set of local tokens tied to spatial support, such as latent grid cells in Shape or slice-wise state tokens in GeoTransolver. The second is a compact set of contextual tokens encoding global geometry, regime variables, or cross-scale summaries. A plausible implication is that MAGNO’s distinctive feature is not simply geometry-aware local embedding, but the coexistence of local and contextual token channels.

## 3. Multi-scale geometry extraction

The defining MAGNO operation is multi-scale geometry aggregation. In GeoTransolver, a set of scales
\[
\mathcal{S} = \{(r_s,k_s)\}_{s=1}^{S}
\]
specifies radii \(r_s\) and maximum neighbor counts \(k_s\). For each state point \(x_{m,i}\), geometry-to-state ball queries collect nearby geometry points \(g_j\) and aggregate scale-specific features from \([\gamma_j,\; g_j-x_{m,i}]\). Concatenating the outputs across scales yields a multi-scale geometry encoding \(U_{m,i}\), which is then projected and appended to the local latent state. In the reverse direction, state-to-geometry ball queries aggregate nearby state features around each geometry point and pool them into scale-wise summaries \(E_s\). Together with pooled geometry-only features \(c_{\mathrm{geom}}\) and global parameters \(p\), these form a reusable context vector
\[
C = [p,\; c_{\mathrm{geom}},\; E_1,\dots,E_S].
\]
Ablations on DrivAerML varied single-scale and multi-scale radii, including \(0.01, 0.05, 0.25, 1.0, 2.5, 5.0\), and varied kernel size up to \(k=32\); the reported finding is that multi-scale settings consistently lower errors and that larger kernels improve performance [2512.20399].

Shape instantiates multi-scale geometry aggregation on a structured latent grid. For each grid cell \(t\), neighborhoods
\[
\mathcal{N}_r(t)=\{i:\|\mathbf{P}_i-\mathbf{Q}_t\|_2\le r\}
\]
are built at radii \(r\in\{0.05, 0.1, 0.2\}\). The tokenizer runs three AGNO layers at successively coarser radii, combining them by concatenation and linear projection. Attention from grid cells to nearby surface points uses cosine-style logits with a learned temperature:
\[
\alpha_{t,i}=\mathrm{softmax}_{i\in\mathcal{N}_r(t)}\!\left(\frac{\langle \mathbf{q}_t,\mathbf{k}_i\rangle}{\|\mathbf{q}_t\|\,\|\mathbf{k}_i\|\,\tau}\right),
\]
and the scale-specific aggregated representation is
\[
h_t^{(r)}=\sum_{i\in\mathcal{N}_r(t)} \alpha_{t,i} V_i.
\]
The three scale outputs are concatenated and projected into the final token embedding \(\mathbf{Z}_t\in\mathbb{R}^{128}\) [2604.22826].

Across these systems, multi-scale is realized by neighborhood radii rather than by a single pyramid of uniformly downsampled grids. The literature also shows other valid realizations. GPSToken uses entropy-driven region partitioning and parameterizes each token as a 2D Gaussian with learnable center, covariance, and texture features, producing non-uniform image tokenization in which small tokens appear in complex regions and large tokens in homogeneous ones [2509.01109]. This suggests that MAGNO’s “multi-scale” qualifier is architectural rather than tied to one discretization scheme.

## 4. Interfaces to transformers and downstream reasoning

GeoTransolver’s GALE formalizes how geometry-aware tokens are consumed. Within each slice \(m\), self-attention operates only over the slice’s latent state tokens, while cross-attention uses the shared context \(C\) as keys and values. An adaptive gate
\[
\alpha_m^{(\ell)}=\sigma\!\left(\eta^{(\ell,m)}(\mathrm{Pool}(\mathrm{SA}_m^{(\ell)}),\mathrm{Pool}(C))\right)
\]
blends self-attention and context-attention outputs at each layer. The effect is persistent geometry and regime conditioning at every depth, rather than one-time injection at the input [2512.20399].

Shape consumes MAGNO outputs with a transformer processor using grouped-query attention and RMSNorm. The \(24^3\) latent grid is patchified with patch size \(p=6\), yielding \(4^3=64\) patches, then processed by \(L=3\) transformer blocks with 4 attention heads and 2 KV heads. Absolute positional encoding derived from grid coordinates is used in the released configuration. A learned attention-pooling module then forms a global embedding \(\bm{z}_{\text{pool}}\in\mathbb{R}^{128}\), while per-token embeddings feed a reconstruction head [2604.22826].

A third architectural pattern appears in NDTokenizer3D, where raw point clouds are converted into multi-scale Normal Distributions Transform cells, encoded scale-wise, and then fused by a Multi-Scale NDT Decoder (MSDec). Each cell stores a 15-dimensional descriptor \([\mu_r^j;\Sigma_r^j;c_r^j]\) consisting of mean, covariance, and projected RGB color. MSDec uses query tokens that cross-attend to one scale per decoder layer, producing a bounded set of holistic scene tokens; ablations found that about 400–850 queries are effective, and three scales \(r=\{2,3,4\}\) gave the best trade-off [2511.21191]. Unlike GALE, where context remains external to the main state stream, MSDec directly produces the scene-token sequence that an LLM consumes.

## 5. Objectives, calibration, and interpretability

MAGNO systems have been trained under both self-supervised and task-conditioned objectives. Shape combines masked-token reconstruction of normalized geometric signatures with multi-resolution contrastive consistency. Fifty percent of latent grid tokens are masked, the reconstruction head predicts \(\hat{\mathbf{Y}}_t\in\mathbb{R}^{28}\), and the target is the z-scored signature
\[
\tilde{\mathbf{Y}}_t=\frac{\mathbf{Y}_t-\bm{\mu}}{\bm{\sigma}+\epsilon}.
\]
The reconstruction loss is Smooth-L1 on masked positions, and the global contrastive objective is symmetric InfoNCE with temperature \(\tau_c=0.07\). On a held-out split of 2,983 meshes, Shape reports reconstruction \(R^2=0.729\) and 98.1% top-1 retrieval under the Wang–Isola protocol. Its \(2\times 2\) ablation on loss type and target-space normalization shows that per-dimension normalization is critical: without it, performance collapses to \(R^2<0.14\) and top-1 \(<88\%\), whereas with normalization both MSE and Smooth-L1 exceed \(R^2>0.70\) and top-1 \(>96\%\), with Smooth-L1 offering secondary stability [2604.22826].

Shape also demonstrates a direct interpretability mechanism. Because each latent grid token is associated with a spatial cell and a geometric signature, one can run the reconstruction head without masking and compute per-token residuals
\[
e_t=\sum_{d=1}^{28}(\tilde{Y}_{t,d}-\hat{Y}_{t,d})^2.
\]
Mapping \(e_t\) back to the corresponding surface neighborhoods yields heatmaps that highlight unusual fillets, defects, or sharp edges, while typical regions remain low-error. In this formulation, explainability is not an auxiliary attribution method but a by-product of the reconstruction prior [2604.22826].

Task-conditioned tokenizers show a related but different supervision strategy. The \(\mu^2\)Tokenizer for radiology report generation uses relative positional encoding, differentiable soft top-\(k\) token selection with \(k=1024\), and dynamic multi-scale pooling over scales \(\{1,2,4\}\); its full system improves GREEN from 0.339 under supervised fine-tuning to 0.400 after DPO [2507.00316]. This suggests that MAGNO-like tokenizers can also be optimized through downstream preference objectives rather than solely through geometric pretext tasks.

## 6. Variants, naming ambiguity, and broader research directions

The term MAGNO carries a mild naming ambiguity. In Shape, it explicitly refers to a “multi-scale geometry-aware tokenizer,” while the implementation is inherited from the Multiscale Attentional Graph Neural Operator encoder of GAOT [2604.22826]. GeoTransolver, by contrast, presents MAGNO more as a reconstructed design recipe based on multi-scale ball queries, shared geometry/global context, and persistent cross-attentional conditioning [2512.20399]. The literature therefore does not define a single mandatory MAGNO block.

Related work expands the design space in several directions. Hi-SAM’s Disentangled Semantic Tokenizer performs geometry-aware alignment on a shared hypersphere using a Gram-matrix volume objective, then applies coarse-to-fine residual quantization with three shared and three modality-specific codes per item; on its Industrial dataset, removing Cross-Modal Geometric Alignment drops GAUC from 0.6410 to 0.6166, underscoring the importance of geometry-aware alignment before tokenization [2602.11799]. Galaxy Walker creates geometry tokens from Euclidean, spherical, and hyperbolic galaxy graphs and processes them with a geometry-adapter mixture-of-experts, reaching \(R^2\) up to \(0.91\) and up to \(+0.17\) F1 improvement in challenging morphology features [2503.18578]. GPSToken shows that geometry-aware tokenization can be continuous rather than grid-based, achieving rec.FID \(0.65\) and FID \(1.50\) on image reconstruction and generation with 128 Gaussian tokens [2509.01109]. NDTokenizer3D demonstrates that scene tokenization can also be built from Gaussian cell statistics in a voxel pyramid, outperforming downsampling-based alternatives on 3D VQA and referring segmentation [2511.21191].

Several misconceptions are clarified by this record. MAGNO is not synonymous with voxelization: GeoTransolver relies on ball-query context construction, Shape uses latent-grid queries over surface points, GPSToken uses Gaussian regions, and Hi-SAM operates in a quantized semantic space. Nor is geometry awareness reducible to positional encoding alone: the cited systems encode normals, curvature, covariance, boundary conditions, global parameters, or non-Euclidean manifold structure in addition to spatial position. Finally, “multi-scale” is not implemented uniformly across the literature; it may refer to query radii, entropy-driven partitions, pooling kernels, residual-quantization depth, or NDT voxel pyramids. This suggests that MAGNO is best treated as a unifying tokenizer principle for geometry-conditioned token formation across irregular domains, CAD, 3D scenes, medical volumes, recommendation, and multimodal scientific modeling.

Source: https://www.emergentmind.com/topics/multi-scale-geometry-aware-tokenizer-magno