VariViT: Adaptive Vision Transformer Models

• VariViT is a design pattern for adapting Vision Transformers to non-canonical inputs by modifying tokenization, positional encoding, and fusion to handle variable image sizes, modalities, or segmentation masks.
• Architectural variants tailor specific mechanisms—such as center-and-select embeddings for 3D MRI or split-sequence projections for radiomics—to preserve the canonical ViT pipeline while addressing input heterogeneity.
• Empirical results across domains, including brain tumor classification, automotive lidar segmentation, and multimodal clinical modeling, demonstrate performance gains and efficiency improvements using these adaptive strategies.

Across papers, the label VariViT has been applied to several Vision Transformer variants that address input heterogeneity while preserving most of the canonical ViT pipeline. The most explicit use denotes a 3D ViT for variable image sizes in brain MRI (Varma et al., 16 Feb 2026). Closely related uses include variable-length segmented inputs for radiomics and clinical data under the name variable Vision Transformer (vViT, sometimes called VariViT) (Usuzaki, 2022), variable-input 3D MR segmentation with changing contrast sets under the name VIViT (Das et al., 13 May 2025), a plain non-hierarchical ViT for automotive point-cloud semantic segmentation referred to in the provided description as VariViT / VaViT (Puy et al., 29 May 2026), and multimodal clinical modeling framed as a “VariViT”-style family in which arbitrary data are rendered as images before Swin-based processing (Tölle et al., 30 Jan 2025). In this broader usage, VariViT denotes not a single standardized architecture but a recurring design pattern: adapt tokenization, positional encoding, batching, or fusion so that a largely standard transformer can operate on inputs that violate the fixed-size, fixed-modality assumptions of vanilla ViT.

1. Nomenclature and defining variants

The term appears in several technically distinct forms. In "VariViT: A Vision Transformer for Variable Image Sizes" (Varma et al., 16 Feb 2026), the problem is direct handling of variable 3D crop sizes with a fixed patch size. In "Splitting expands the application range of Vision Transformer -- variable Vision Transformer (vViT)" (Usuzaki, 2022), the central mechanism is splitting a 1D feature sequence into segments of different lengths and embedding each segment separately. In "VIViT: Variable-Input Vision Transformer Framework for 3D MR Image Segmentation" (Das et al., 13 May 2025), the variable object is the set of available MR contrasts per study. In the lidar segmentation work "Vanilla ViT for Automotive Point Cloud Semantic Segmentation" (Puy et al., 29 May 2026), the detailed description explicitly uses the label VariViT / VaViT for a model whose token sequence length varies with the number of occupied BEV pillars. In "Arbitrary Data as Images" (Tölle et al., 30 Jan 2025), the relation is more conceptual: the paper positions its multimodal visualization strategy in the language of a “VariViT”-style family.

Variant name	Variable factor	Core mechanism
VariViT	3D image size	Center-and-select positional embeddings and variable-size batching
vViT	Segment dimensionality	Split sequence with segment-specific linear embeddings
VIViT	MR contrast set	Dynamic patch tokenizer, modality embeddings, modality fusion decoder
VaViT / “VariViT / VaViT”	Occupied BEV token count	3D-aware tokenizer, vanilla ViT backbone, gated point-level head
ViTiMM as “VariViT”-style	Modality representation	Visualization-based tokenization and late multimodal fusion

This terminological dispersion matters because the shared name can obscure the fact that the proposals intervene at different points of the ViT stack. Some alter positional encoding, some alter token formation, and some alter fusion across modalities, while leaving transformer blocks themselves largely unchanged.

2. VariViT for variable image sizes in 3D MRI

The most literal instantiation of VariViT is the 2026 model for variable 3D image sizes in brain MRI classification (Varma et al., 16 Feb 2026). Its starting point is the limitation that conventional ViTs require a fixed input resolution, forcing resizing, padding, or cropping. In the reported medical-imaging setting, this is problematic because tumors are small relative to the brain, highly variable in size and shape, and embedded in large amounts of background. A fixed bounding box crop can therefore induce highly variable foreground-to-background ratios, while resizing can introduce artefacts.

Architecturally, this model is a 3D ViT-S/16 based on 3DViT, with a fixed patch size of $16 \times 16 \times 16$ mm$^3$, embedding dimension $d = 384$, 12 encoder blocks, and 6 attention heads. Variable 3D crops are allowed, so the number of tokens depends on crop size: a 64$^3$ crop yields 64 patches, an 80$^3$ crop yields 125 patches, and a 96$^3$ crop yields 216 patches. The classification head is standard: the final CLS token is mapped to class logits.

Its distinctive contribution is the center-and-select positional embedding scheme. A large sinusoidal 3D positional embedding grid is initialized for the largest crop size, corresponding to a $6 \times 6 \times 6$ patch grid. For a smaller crop with patch grid $G'_l \times G'_h \times G'_w$, a centered sub-grid is selected rather than interpolated: $$start_k = C_k - \left\lfloor \frac{G'_k}{2} \right\rfloor,\qquad end_k = start_k + G'_k$$ for $k \in \{l,h,w\}$. The motivation is domain-specific: tumor crops are center-aligned using the tumor center of mass, so the semantic center is stable across crop sizes. This removes the need for trilinear interpolation of 3D positional embeddings and preserves the exact sinusoidal lattice around the region of interest.

A second contribution is the batching strategy for variable token counts. The paper introduces Custom Batch Sampler (CBS), which groups samples of the same image size into a batch, and Gradient Accumulation (GA), which uses mini-batch size 1 with accumulated gradients to emulate a larger effective batch. This is tied directly to transformer complexity, since self-attention scales roughly as $^3$0, and the jump from $^3$1 to $^3$2 corresponds to approximately $^3$3 more attention cost.

The reported evaluations are on two 3D brain MRI datasets. For IDH mutation status prediction in glioma, the best model is VariViT-GA with AUC $^3$4, F1 $^3$5, and MCC $^3$6. For primary versus metastatic tumor classification, VariViT-GA reaches AUC $^3$7, F1 $^3$8, and MCC $^3$9. The paper summarizes these as F1-scores of 75.5% and 76.3%, respectively. It also reports that the batching strategy reduces computation time by up to ~30% compared to conventional fixed-size architectures. Ablations show that center-and-select slightly outperforms interpolation-based positional embedding strategies, and that absolute positional encoding performs better than the tested relative alternatives in this setting.

The limitations are equally specific. The method assumes availability of tumor segmentation masks to derive centered crops, retains a fixed patch size despite variable image size, and is evaluated only on 3D brain MRI tumor tasks. These constraints make the proposal specialized rather than universal, but within its target regime it is a minimal-change adaptation of ViT to variable spatial extent.

3. Variable Vision Transformer for heterogeneous feature segments

A different meaning of VariViT appears in the 2022 variable Vision Transformer (vViT, sometimes called VariViT) (Usuzaki, 2022). Here the variable object is not image size but the dimensionality of semantically distinct feature groups. Standard ViT assumes that all patches share the same dimension and can be embedded by a single projection matrix. vViT replaces this with a split sequence $d = 384$0 that partitions a 1D input sequence into $d = 384$1 segments of lengths $d = 384$2, followed by segment-specific linear projections.

If the input is divided into sub-sequences $d = 384$3, each segment $d = 384$4 is projected by its own matrix $d = 384$5, producing

$d = 384$6

Each segment embedding is then mapped to the transformer hidden dimension $d = 384$7 using its own $d = 384$8, a learnable class token is prepended, positional embeddings are added, and a standard transformer encoder with pre-LayerNorm, MSA, residual connections, and GELU MLP blocks is applied. The class token at the last layer is used for classification.

The demonstration task is glioma survival prediction from radiomics derived from T1-weighted MRI. Using PyRadiomics, the study extracts 104 radiomic features per slice, organized into seven heterogeneous groups: 19 First Order, 10 Shape2D, 24 GLCM, 16 GLSZM, 16 GLRLM, 5 NGTDM, and 14 GLDM. These group lengths instantiate the split-sequence design directly. The dataset is UCSF-PDGM from TCIA, with 501 patients and 10,538 2D slices remaining after exclusion of masks with 16 pixels or fewer.

Training uses binary cross-entropy, Adam with learning rate $d = 384$9, $^3$0, $^3$1, a multiplicative learning-rate decay of 0.99 per epoch, 600 training epochs, and no reported pretraining. The reported best results for 365-day survival prediction are sensitivity 0.83, specificity 0.82, accuracy 0.81, and AUC-ROC 0.76. The authors state that, to their knowledge, this is state-of-the-art for direct prediction of survival among glioma patients in that setting.

The importance of vViT lies less in a new attention mechanism than in preserving group structure among heterogeneous inputs. The paper does not report ablations against a concatenated MLP or a standard ViT without variable splitting, but its architectural claim is clear: heterogeneous medical inputs can be jointly modeled without pretending that all groups are interchangeable patches. Its principal limitations are that the experiments remain slice-based rather than patient-based, do not integrate clinical information or raw MRI images, and do not provide extensive ablation or interpretability analysis.

4. Variable-input multimodal extensions

A more expansive form of the same design pressure appears in VIViT, a Variable-Input Vision Transformer for 3D MR segmentation (Das et al., 13 May 2025). The motivating problem is that real-world MR studies contain variable sets of contrasts per study, with each study having its own subset $^3$2. Instead of assuming a fixed channel layout, VIViT accepts any subset of available contrasts during self-supervised pretraining and downstream finetuning or inference, and can also handle new contrasts at finetuning time that were not present in pretraining.

The architecture consists of a two-stage dynamic patch tokenizer, a Dynamic Transformer–Convolution Encoder, and a Modality Fusion Decoder. Each input contrast $^3$3 is processed by dynamic 3D convolutions conditioned on a learned modality vector $^3$4. Tokens from all available contrasts are concatenated into one variable-length sequence, augmented with 3D positional embeddings and modality embeddings, and processed by global self-attention interleaved with dynamic 3D convolutions. For segmentation, the decoder first applies modality-specific self-attention at each encoder level and then averages across available modalities: $^3$5 This symmetry over the available modalities is the key to its variable-input behavior.

For self-supervised learning, VIViT uses a masked autoencoder-style objective with 70% masking, which the paper reports as the best masking ratio. Pretraining is performed on 45,374 MR studies with arbitrary subsets from ADC, TraceW, T2, GRE, SWI, T1, T1CE, FLAIR, using 500 epochs, batch size 4, Weighted Adam, cosine annealing, and initial learning rate $^3$6. The reported downstream segmentation results are mean Dice 0.624 for brain infarct segmentation and mean Dice 0.883 for brain tumor segmentation with SSL. These outperform the compared UNET, UNETR, SegResNet, SwinUNETR, and AdaptiveUNETR variants. Importantly, the paper also reports that SSL remains beneficial even when some downstream contrasts were unseen during pretraining.

A related but differently named extension is ViTiMM, which treats irregular clinical measurements, medications, ECG waveforms, and images as synthetic images and processes them with separate Swin-Large encoders plus a RoBERTa text encoder, followed by late fusion (Tölle et al., 30 Jan 2025). The paper explicitly frames this as a ViT variant whose key feature is input tokenization via visualization rather than modality-specific numeric embeddings. On 6,125 patients from MIMIC-IV, the full multimodal configuration C|M|X|E reaches AUROC 0.922, AUPRC 0.764, and balanced accuracy 0.847 for in-hospital mortality, and AUROC 0.784, AUPRC 0.549, and balanced accuracy 0.659 for phenotyping.

Taken together, these works indicate that the variable-input theme generalizes far beyond image resizing. It encompasses changing channel sets, changing modality inventories, and even the choice to convert non-image data into image-like representations so that a largely standard vision transformer can be reused.

5. VaViT and the extension of the name to point clouds

The 2026 lidar segmentation paper introduces VaViT, short for Vanilla ViT, and the provided description explicitly labels it VariViT / VaViT (Puy et al., 29 May 2026). Its problem setting is large-scale automotive lidar semantic segmentation on nuScenes, SemanticKITTI, and Waymo Open Dataset, where raw point counts can reach the order of $^3$7 to $^3$8 per frame and full self-attention on points is infeasible.

The architecture pushes nearly all domain-specific processing into a 3D-aware tokenizer and a lightweight segmentation head, while keeping the backbone as a plain, non-hierarchical ViT with global self-attention over BEV tokens. Each point begins with features such as

$^3$9

A shared point MLP with PointNet++/DGCNN-style local aggregation and neighbor gating is repeated $^3$0 times by default. Final point embeddings are then aggregated into BEV pillars by max pooling, producing one token per occupied pillar. Only occupied pillars yield tokens, so the sequence length $^3$1 varies by frame and is padded within a batch using a learnable padding token. These tokens are processed by a standard ViT with 2D RoPE positional encoding. The segmentation head then lifts pillar tokens back to points and fuses them with point embeddings using gated fusion before point-level classification.

The model family includes VaViT-S/B/L, corresponding to ViT-S/B/L-style backbones. The tokenizer feature dimension scales with model size: 64 for VaViT-S, 128 for VaViT-B, and 192 for VaViT-L. Ablations show that max pooling is slightly better than mean pooling for BEV projection, that gating in the tokenizer and segmentation head both improve performance, and that using both gates yields the best results, with +2–2.3 mIoU on WOD/SemanticKITTI versus no gates. The optimal tokenization pillar size in the reported ablation is 0.50 m, although this increases memory cost.

The paper also introduces a pillar-based mixing augmentation tailored to ViT training on lidar. The best setting mixes 3 point clouds using mixing pillars of size 7.5 m, outperforming both PillarMix and PolarMix in the reported comparisons. Training uses AdamW, 4 epochs of warmup to 0.001, cosine decay, weight decay 0.003, 45 epochs, batch size 8, and stochastic depth with default probability 0.3.

The main results show that a plain ViT can be competitive with specialized point-cloud backbones. On nuScenes validation, VaViT-B reaches 81.3% mIoU without TTA. On SemanticKITTI validation, VaViT-B* reaches 68.0% mIoU. On Waymo Open Dataset validation, VaViT-B* reaches 70.9% mIoU, with VaViT-B at 70.5%. The paper also reports strong efficiency: on nuScenes validation on an NVIDIA H100, VaViT-B requires 20 ms per cloud, compared with 42 ms for LitePT without chunking and 295 ms with chunking. TTA gains are small—+0.3 on nuScenes, +0.7 on SemanticKITTI, and +0.5 on WOD for VaViT-B*—which the authors interpret as evidence of stronger invariance to global transformations.

Although this model is semantically distant from the medical-imaging uses of VariViT, the naming overlap is instructive. Here again, the backbone remains largely standard, and variability is absorbed by the tokenizer, the token count, and the fusion head.

6. Comparative interpretation, limitations, and scope

Several architectural regularities recur across these models. First, the transformer blocks are usually close to canonical ViT: standard MSA, standard MLP blocks, pre-LayerNorm, and residual connections remain intact in the 3D image-size VariViT, the radiomics vViT, the variable-contrast VIViT encoder, and the BEV-token VaViT (Varma et al., 16 Feb 2026, Usuzaki, 2022, Das et al., 13 May 2025, Puy et al., 29 May 2026). The modifications concentrate on how tokens are formed, how positional information is assigned, and how heterogeneous sources are fused.

Second, the notion of “variable” differs by domain. In one line, variability means a changing number of spatial patches, handled through center-selected positional embeddings and size-aware batching. In another, it means changing segment dimensionalities, handled through per-segment embeddings. In another, it means changing modality subsets, handled through dynamic convolution, modality embeddings, and symmetric fusion across the available inputs. In the lidar case, it means changing numbers of occupied BEV pillars and thus changing sequence length. This suggests that the stable conceptual core of VariViT is not a specific block-level innovation but a family of mechanisms for making ViTs tolerant to non-canonical input structure.

Third, these methods differ in how much prior structure they assume. The 3D MRI VariViT relies on tumor center alignment and segmentation-derived crops. The radiomics vViT presumes a fixed semantic partition of features into groups such as First Order or GLCM. VIViT depends on explicit contrast identity through modality vectors and embeddings. VaViT assumes that automotive point clouds can be efficiently summarized on a BEV pillar grid. ViTiMM places substantial weight on visual prompt engineering, including panel layout, color, and plotting conventions (Tölle et al., 30 Jan 2025). In each case, variability is handled not by removing inductive bias, but by relocating it into the interface between raw data and transformer tokens.

The limitations are equally systematic. The 3D image-size VariViT remains tied to brain tumor classification and requires tumor masks. vViT lacks patient-level aggregation, external validation, and multimodal integration. VIViT is computationally intensive and relies on a very large heterogeneous pretraining corpus. VaViT discards vertical structure within a pillar after BEV pooling, exhibits a performance plateau when scaling from B to L, and remains a single-frame model. ViTiMM is design-dependent, expensive at multimodal resolution, and evaluated only on MIMIC-IV. A plausible implication is that “VariViT” is best understood as a research direction rather than a settled architecture: a recurring attempt to preserve the simplicity and transfer properties of ViTs while re-engineering the tokenization boundary for variable, irregular, or multimodal inputs.