FlexViT: Adaptable Vision Transformer

Updated 4 July 2026

FlexViT is a design objective aimed at making Vision Transformers adaptable across input modalities, model scales, and hardware constraints.
It spans methods such as sparse video tube tokenization, backbone stitching, and retraining-free pruning to optimize performance and efficiency.
Hardware implementations like FPGA-based accelerators demonstrate FlexViT’s capability to dynamically optimize ViT inference for edge devices.

Searching arXiv for papers related to “FlexViT” and adjacent usages of the term. {"query":"FlexViT vision transformer flexible backbone video tubes FPGA arXiv","max_results":10,"sort_by":"submittedDate","sort_order":"descending"} Searching more specifically for the cited arXiv records to anchor the article. {"query":"(Piergiovanni et al., 2022, Pan et al., 2023, Dymarkowski et al., 30 Jun 2026) FlexViT arXiv","max_results":10,"sort_by":"relevance","sort_order":"descending"} FlexViT designates a cluster of research directions centered on making Vision Transformers flexible under changing modalities, compute budgets, and deployment substrates. In the materials considered here, the term appears in at least four related senses: as a label sometimes applied to TubeViT’s sparse video-tube tokenization for joint image–video learning, as shorthand for a single stitched ViT backbone spanning multiple performance–efficiency trade-offs, as an elastic inference regime obtained by post-pretraining structured pruning, and as the title of a reconfigurable FPGA accelerator for edge ViT inference (Piergiovanni et al., 2022, Pan et al., 2023, Simoncini et al., 20 Oct 2025, Dymarkowski et al., 30 Jun 2026). Taken together, these usages indicate that “FlexViT” is less a single canonical architecture than a recurring design objective: preserving ViT performance while relaxing fixed assumptions about input structure, model scale, or hardware execution.

1. Terminological scope

Within the records surveyed here, “FlexViT” does not refer to one unique method. Instead, it names or informally describes several mechanisms for making ViTs adaptable at different layers of the stack.

Usage	Mechanism	Representative paper
TubeViT / sometimes referred to as “FlexViT”	Sparse video tubes for joint image and video learning	"Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning" (Piergiovanni et al., 2022)
Flexible backbone	Two-way stitching of pretrained ViTs into many subnetworks	"Stitched ViTs are Flexible Vision Backbones" (Pan et al., 2023)
Elastic inference	Single-shot structured pruning with one global ranking	"Elastic ViTs from Pretrained Models without Retraining" (Simoncini et al., 20 Oct 2025)
FPGA accelerator	Reconfigurable INT8 GEMM-based edge accelerator	"FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers" (Dymarkowski et al., 30 Jun 2026)

This terminological multiplicity matters because the underlying technical objects differ substantially. One line of work modifies tokenization while keeping the ViT backbone unchanged; another stitches together pretrained model families; a third derives a continuum of sparse subnetworks from one pretrained model; and the hardware-oriented work reconfigures dataflow and tiling for heterogeneous ViT layers. A common misconception is to treat these as interchangeable variants of a single system. The available evidence instead supports a layered interpretation in which “flexibility” can target input representation, backbone composition, post-pretraining compression, or accelerator execution.

2. Sparse video tubes and joint image–video ViTs

TubeViT begins from a standard ViT encoder with input sequence $z^0=\{z_i^0\}_{i=1}^N$ , $z_i^0\in\mathbb R^d$ , followed by $L$ layers of MSA and MLP blocks with residual connections and LayerNorm. To support video, it replaces the single 2D patch tokenizer by a small family of 3D tubes, including a conventional 2D-only tube $(T=1,H=16,W=16)$ with a large temporal stride and several spatio-temporal tubes such as $(8\times 8\times 8)$ , $(16\times 4\times 4)$ , and $(4\times 12\times 12)$ , each applied with its own 3D convolution. All resulting tube tokens are concatenated into one sequence while the ViT backbone itself is kept exactly the same (Piergiovanni et al., 2022).

A video tube $i$ is specified by kernel size $(T_i\times H_i\times W_i)$ , stride $(T_i^s,H_i^s,W_i^s)$ , and optional offset $z_i^0\in\mathbb R^d$ 0. For video $z_i^0\in\mathbb R^d$ 1, the number of tokens produced by tube $z_i^0\in\mathbb R^d$ 2 is

$z_i^0\in\mathbb R^d$ 3

Each token is a flattened 3D block linearly projected to $z_i^0\in\mathbb R^d$ 4. With $z_i^0\in\mathbb R^d$ 5 tubes, the total token count is

$z_i^0\in\mathbb R^d$ 6

with $z_i^0\in\mathbb R^d$ 7 and $z_i^0\in\mathbb R^d$ 8 as the typical regime described in the paper. The sparse sampling strategy uses large temporal strides such as $z_i^0\in\mathbb R^d$ 9 and mixes long–narrow and short–wide tube shapes to cover space–time sparsely but diversely.

The positional encoding is fixed sine–cosine and aligned to the true center of each token in $L$ 0 space. For hidden size $L$ 1 and $L$ 2,

$L$ 3

and the positional embedding $L$ 4 is constructed as

$L$ 5

Image-only inputs are handled by the special $L$ 6 tube, so the same encoder can be trained jointly on images and video clips. The training objective uses a shared ViT encoder and separate classification heads, for example one for ImageNet-1k and one for Kinetics-600, with cross-entropy computed only on the corresponding head:

$L$ 7

For adaptation from pretrained image ViTs, the method injects the new tube tokenizers with linear weights taken from a small jointly trained TubeViT, freezes the first $L$ 8 layers, and finetunes only layers $L$ 9 plus classification heads. An optional gated skip is added from the raw input tokens:

$(T=1,H=16,W=16)$ 0

where $(T=1,H=16,W=16)$ 1 is a learned scalar initialized to $(T=1,H=16,W=16)$ 2.

Empirically, the sparse-tube construction is presented as a way to retain full self-attention while avoiding dense token counts. For a dense $(T=1,H=16,W=16)$ 3 video, the description gives $(T=1,H=16,W=16)$ 4 tokens, versus $(T=1,H=16,W=16)$ 5 for TubeViT, implying $(T=1,H=16,W=16)$ 6 fewer attention operations. The reported benchmark summary includes 88.6% top-1 at $(T=1,H=16,W=16)$ 7 TFLOPs for TubeViT-B and 90.2% at $(T=1,H=16,W=16)$ 8 TFLOPs for TubeViT-L on Kinetics-400 with $(T=1,H=16,W=16)$ 9 crops, compared with 89.9% at $(8\times 8\times 8)$ 0 TFLOPs for MTV-H; 91.5% for TubeViT-L on Kinetics-600; 83.8% on Kinetics-700; and 76.1% on Something-Something-V2. This suggests that, in this usage of the term, FlexViT denotes flexibility across visual modality rather than across model size or hardware.

3. Flexible backbones through stitching

SN-Netv2 defines flexibility at the backbone level by turning a family of pretrained plain ViTs into a single model that covers many subnetworks with different performance–efficiency trade-offs. The method is built on stitchable neural networks and introduces a systematically improved stitching framework for downstream task adaptation. The base models, or anchors, are standard plain ViTs of different sizes such as DeiT3-S, DeiT3-B, and DeiT3-L, each containing a patch embedding, $(8\times 8\times 8)$ 1 transformer blocks, and a classification head (Pan et al., 2023).

Let $(8\times 8\times 8)$ 2 be a small ViT and $(8\times 8\times 8)$ 3 a large ViT, with activations $(8\times 8\times 8)$ 4 after block $(8\times 8\times 8)$ 5 and $(8\times 8\times 8)$ 6 after block $(8\times 8\times 8)$ 7. Two learnable $(8\times 8\times 8)$ 8 convolutional stitching layers are introduced:

$(8\times 8\times 8)$ 9

This yields Fast→Slow, Slow→Fast, Fast→Slow→Fast, and Slow→Fast→Slow routes. Representative configurations include

$(16\times 4\times 4)$ 0

for Fast→Slow and

$(16\times 4\times 4)$ 1

for Slow→Fast. Allowing $(16\times 4\times 4)$ 2 to vary produces a stitching space described as approximately $(16\times 4\times 4)$ 3 that of SN-Netv1.

Training samples subnetworks in a FLOPs-aware manner. Total FLOPs are discretized into buckets $(16\times 4\times 4)$ 4, using a step size of $(16\times 4\times 4)$ 5 G for ImageNet and $(16\times 4\times 4)$ 6 G for dense tasks. At each iteration, a budget bucket is sampled uniformly and then a stitch is sampled uniformly among those with that FLOPs value. The summary characterizes this as a resource-constrained sampling strategy, or ROS, that ensures each FLOPs level is seen equally often and avoids under-training anchors. The expected FLOPs per minibatch is exactly $(16\times 4\times 4)$ 7.

A further ingredient is low-rank adaptation of the stitching layers. Initial stitching matrices are obtained by least-squares matching,

$(16\times 4\times 4)$ 8

and training freezes $(16\times 4\times 4)$ 9 while learning only a low-rank update

$(4\times 12\times 12)$ 0

According to the description, this steadies learning when both anchors must adapt to a new data domain and empirically produces a smoother, higher Pareto frontier.

At inference time, the model functions as a flexible backbone: a desired FLOPs budget is selected, one pre-evaluated stitch with approximately that budget is chosen, and the corresponding forward pass is executed with no retraining or weight loading. The switching overhead is described as negligible, since the extra modules are standard $(4\times 12\times 12)$ 1 convolutions and memory is only about $(4\times 12\times 12)$ 2 above the largest single ViT.

The reported results quantify this flexibility across several tasks. On ImageNet-1K, anchors DeiT3-S and DeiT3-L are given as $(4\times 12\times 12)$ 3 G FLOPs / $(4\times 12\times 12)$ 4 and $(4\times 12\times 12)$ 5 G / $(4\times 12\times 12)$ 6, while SN-Netv2 yields about $(4\times 12\times 12)$ 7 stitches covering $(4\times 12\times 12)$ 8 G; example accuracies include roughly $(4\times 12\times 12)$ 9 at $i$ 0 G and $i$ 1 at $i$ 2 G, improving on SN-Netv1. On ADE20K, selected SN-Netv2 operating points are 45.6 mIoU at $i$ 3 G, 46.8 at $i$ 4 G, 49.0 at $i$ 5 G, and 51.4 at $i$ 6 G, versus 40.4 for SN-Netv1 at $i$ 7 G. On COCO-Stuff-10K, the reported points are 40.1 at $i$ 8 G, 42.7 at $i$ 9 G, and 48.2 at $(T_i\times H_i\times W_i)$ 0 G. On NYUv2, RMSE decreases from approximately 0.48 at $(T_i\times H_i\times W_i)$ 1 G to 0.42 at $(T_i\times H_i\times W_i)$ 2 G. Training efficiency is also reported: on ADE20K, separate training of DeiT3-S, -B, and -L costs $(T_i\times H_i\times W_i)$ 3 A100-GPU hours, versus $(T_i\times H_i\times W_i)$ 4 hours for one SN-Netv2 $(T_i\times H_i\times W_i)$ 5 run; on COCO-Stuff, the comparison is $(T_i\times H_i\times W_i)$ 6 hours versus $(T_i\times H_i\times W_i)$ 7 hours. In this line of work, FlexViT denotes a unified backbone that spans a discrete but rich set of stitched subnetworks.

4. Elastic ViTs through retraining-free pruning

SnapViT addresses flexibility after pretraining rather than during backbone construction. The input is a pretrained ViT and an unlabeled dataset $(T_i\times H_i\times W_i)$ 8, and the output is a continuum of elastic submodels that can be adjusted to any computational budget without retraining. The method combines self-supervised gradient information, blockwise grouping, an evolutionary approximation of global Hessian structure, and a single global importance ranking (Simoncini et al., 20 Oct 2025).

The first stage computes self-supervised DINO gradients on a small batch of approximately $(T_i\times H_i\times W_i)$ 9 samples and forms a diagonal Hessian proxy per parameter:

$(T_i^s,H_i^s,W_i^s)$ 0

Parameters are then grouped into $(T_i^s,H_i^s,W_i^s)$ 1 functional blocks, such as FFN row–column groups or whole attention heads, with a binary membership matrix $(T_i^s,H_i^s,W_i^s)$ 2. To approximate global off-diagonal Hessian structure, the method runs xNES over block-scaling vectors $(T_i^s,H_i^s,W_i^s)$ 3. For each sampled $(T_i^s,H_i^s,W_i^s)$ 4, it constructs a trial prunability vector

$(T_i^s,H_i^s,W_i^s)$ 5

prunes the smallest entries at several target sparsities such as $(T_i^s,H_i^s,W_i^s)$ 6, and evaluates representation drift by cosine similarity between PCA-compressed embeddings of the original and pruned networks. After $(T_i^s,H_i^s,W_i^s)$ 7 iterations, $(T_i^s,H_i^s,W_i^s)$ 8 is interpreted as a proxy for the global Hessian component $(T_i^s,H_i^s,W_i^s)$ 9.

Final importance scores are

$z_i^0\in\mathbb R^d$ 00

and these are globally ranked once. Any desired sparsity $z_i^0\in\mathbb R^d$ 01 is then realized by zeroing the lowest-ranked scores, so that the subnetwork at sparsity $z_i^0\in\mathbb R^d$ 02 is determined directly by thresholding the fixed ranking. The stated objective is structured, single-shot pruning under a compute or parameter-count constraint while minimizing the induced second-order loss increase

$z_i^0\in\mathbb R^d$ 03

with $z_i^0\in\mathbb R^d$ 04 neglected.

This elastic inference mechanism is explicitly budget-driven. At inference, a user specifies sparsity $z_i^0\in\mathbb R^d$ 05, computes a threshold $z_i^0\in\mathbb R^d$ 06 such that $z_i^0\in\mathbb R^d$ 07, applies the binary mask $z_i^0\in\mathbb R^d$ 08, and runs $z_i^0\in\mathbb R^d$ 09. Because the ranking is computed once, there is no per-budget retraining or optimization.

The experimental scope covers seven image-classification datasets, including ImageNet-1k, FGVC Aircraft, Oxford-Pets, DTD, EuroSAT, and CIFAR-10/100, as well as Pascal VOC 2012 for linear semantic segmentation. Backbones include self-supervised models such as DINO, DINOv3, and SigLIPv2, and supervised models such as DeIT, DeIT-III, and AugReg. Reported results include less than 5% k-NN and linear-probe drop for DINO ViT-B/16 at 40% sparsity; +15.1% versus SNIP-Mag and +53.2% versus LAMP for the linear metric; +7.0% versus SNIP and +12.3% versus FPTP for AugReg/DeIT ViT-B/16 at 50% sparsity; and, for DeIT-III ViT-H/14 at 50% sparsity, an average drop of 3.7% in k-NN and 3.4% in linear accuracy over seven datasets, with recovery within $z_i^0\in\mathbb R^d$ 10 after one weight-correction step. The reported runtime is under five minutes on one A100 for models up to ViT-L/16, with longer xNES schedules needed for large self-supervised models such as DINOv3 and SigLIPv2. This suggests a notion of FlexViT in which elasticity is induced post hoc from a single pretrained model rather than encoded during initial architecture design.

5. FlexViT as an FPGA-based edge accelerator

In the most literal use of the name, FlexViT is a reconfigurable FPGA accelerator for efficient ViT inference on resource-constrained edge devices. It is built on SECDA-TFLite, which uses the TensorFlow-Lite delegate mechanism to intercept subgraphs in a TFLite model and redirect them to custom hardware kernels. FlexViT extends this framework with a custom delegate that recognizes both fully connected and convolutional operators at runtime, a lightweight host-side im2col transformation, and a compact control-packet protocol over AXI streams for layer configuration, including dimensions, mode flags, and quantization parameters (Dymarkowski et al., 30 Jun 2026).

The hardware–software co-design partitions responsibilities between an ARM host CPU and FPGA fabric. The ARM host loads and quantizes the TFLite model, performs im2col on convolution inputs, selects a dataflow mode by heuristic or DMA-traffic estimator, and streams activations, weights, and bias or scale parameters into the FPGA. The FPGA contains a scheduler, three GEMM cores with on-chip buffers and local accumulators, and a post-processing unit that applies INT8 requantization and packs results back to the host. Control is decoupled from compute, and AXI-stream handshakes are used to overlap DMA, computation, and post-processing.

The compute substrate is a unified INT8 GEMM engine. Each of the $z_i^0\in\mathbb R^d$ 11 GEMM cores has tile size $z_i^0\in\mathbb R^d$ 12 rows by $z_i^0\in\mathbb R^d$ 13 columns, reduction depth $z_i^0\in\mathbb R^d$ 14, and SIMD width $z_i^0\in\mathbb R^d$ 15. The arithmetic is hybrid: for each 8-bit multiplication, the lower 8 bits are implemented in LUT logic and the upper 8 bits use DSP slices. The dataflow is output-stationary, so partial sums remain in 32-bit accumulator registers until the full $z_i^0\in\mathbb R^d$ 16 span has been processed.

At $z_i^0\in\mathbb R^d$ 17 MHz, the theoretical MAC throughput is

$z_i^0\in\mathbb R^d$ 18

equivalent to

$z_i^0\in\mathbb R^d$ 19

A single $z_i^0\in\mathbb R^d$ 20 tile requires

$z_i^0\in\mathbb R^d$ 21

corresponding to a nominal compute latency of

$z_i^0\in\mathbb R^d$ 22

Both FC and convolutional layers are mapped to GEMM. For convolution, host-side im2col rearranges $z_i^0\in\mathbb R^d$ 23 into

$z_i^0\in\mathbb R^d$ 24

with kernels reshaped into

$z_i^0\in\mathbb R^d$ 25

so that

$z_i^0\in\mathbb R^d$ 26

FlexViT then selects between two runtime dataflows. Input-Broadcast broadcasts one input tile of size $z_i^0\in\mathbb R^d$ 27 to all three cores and partitions the weight space across cores; Weight-Broadcast broadcasts one weight tile of size $z_i^0\in\mathbb R^d$ 28 and partitions the input space. A simple criterion for FC layers is

$z_i^0\in\mathbb R^d$ 29

More generally, the mode is chosen by comparing estimated bytes transferred:

$z_i^0\in\mathbb R^d$ 30

$z_i^0\in\mathbb R^d$ 31

A further key mechanism is depth-first tiling. On-chip buffers are dimensioned to hold a full $z_i^0\in\mathbb R^d$ 32-depth tile,

$z_i^0\in\mathbb R^d$ 33

so all MACs for one $z_i^0\in\mathbb R^d$ 34 output tile can be accumulated locally in a single pass. The description contrasts this with a naive tiled GEMM, where partial sums would incur approximately

$z_i^0\in\mathbb R^d$ 35

bytes of 32-bit traffic; under the proposed strategy, only the final 8-bit outputs are streamed out, approximately

$z_i^0\in\mathbb R^d$ 36

Implementation is reported on a PYNQ-Z2 FPGA at 200 MHz. Resource utilization is 71.8% of 220 DSP slices, 83.9% of 140 BRAM18 blocks, 69.3% of 53,200 LUTs, and approximately 65% of flip-flops. The depth buffer $z_i^0\in\mathbb R^d$ 37 covers all but one layer across five ViT variants; the final $z_i^0\in\mathbb R^d$ 38 classification layer in EfficientViT-b1 has $z_i^0\in\mathbb R^d$ 39 and would require depth tiling if offloaded.

The evaluation includes ViT-T, DeiT-T, Swin-T, MobileViT-S, and EfficientViT-b1, all quantized to INT8 TFLite, with an ARM Cortex-A9 CPU baseline measured over 100 runs. The reported peak layer speedup is 2.74× and the peak end-to-end speedup is 1.40×. Representative results include ViT-T total latency dropping from 2.04 s on CPU to 1.46 s on CPU+accelerator, DeiT-T from 2.04 s to 1.46 s, and Swin-T from 7.67 s to 5.68 s. Off-chip traffic is reported to fall by up to 50% relative to a naive tiled GEMM with partial-sum exchanges. Under INT8 quantization with per-tensor scales for FC and per-channel scales for CONV, all five models achieve greater than 99% cosine similarity to the floating-point baseline. In this hardware sense, FlexViT is not a model family but a deployment substrate for heterogeneous ViT operators.

6. Comparative perspective

The surveyed works define flexibility at different abstraction levels. TubeViT changes tokenization while keeping the encoder unchanged; SN-Netv2 combines pretrained anchors into a large stitching space; SnapViT derives a continuum of sparse subnetworks from one pretrained model; and the FPGA FlexViT adapts execution strategy to operator heterogeneity and edge resource limits (Piergiovanni et al., 2022, Pan et al., 2023, Simoncini et al., 20 Oct 2025, Dymarkowski et al., 30 Jun 2026). This layered view is the most faithful way to reconcile the shared label.

Several distinctions follow. First, flexibility need not imply dynamic token sparsity at inference time: TubeViT uses fixed sparse tubes and still computes full self-attention over the resulting token set. Second, flexibility need not imply a single dense backbone: SN-Netv2 explicitly exposes many stitched routes, whereas SnapViT exposes many masked subnetworks through one global score ranking. Third, hardware FlexViT does not compress or retrain the model; it re-expresses FC and CONV as INT8 GEMM and switches dataflow modes at runtime.

A further misconception is that all forms of FlexViT address the same bottleneck. The bottleneck in TubeViT is quadratic attention over dense video tokens; in SN-Netv2 it is the cost of separately training and deploying multiple ViT scales; in SnapViT it is the rigidity of predetermined pretrained sizes and the need for label-free, retraining-free adaptation; in FPGA FlexViT it is the variation in tensor shapes and operator types across modern hybrid ViTs. The shared vocabulary therefore marks a common systems objective—adaptability—rather than a single invariant mechanism.

Taken together, these papers suggest that FlexViT has become a useful umbrella for research on configurable ViTs. The term spans modality-unified tokenization, FLOPs-conditioned backbone stitching, sparsity-conditioned elastic inference, and reconfigurable edge acceleration. For researchers, the principal value of the term lies not in naming one architecture but in identifying a design space in which ViTs are made adjustable without abandoning the core transformer formulation.