---
title: 'FlexViT: Adaptable Vision Transformer'
url: https://www.emergentmind.com/topics/flexvit
type: topic
---

# FlexViT: Adaptable Vision Transformer

Searching arXiv for recent papers related to “FlexViT” and adjacent usages of the term.
{"query":"FlexViT vision transformer flexible backbone video tubes FPGA arXiv","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}
Searching more specifically for the cited arXiv records to anchor the article.
{"query":"2212.03229 2307.00154 2606.31938 FlexViT arXiv","max_results":10,"sort_by":"relevance","sort_order":"descending"}
FlexViT designates a cluster of research directions centered on making Vision Transformers flexible under changing modalities, compute budgets, and deployment substrates. In the materials considered here, the term appears in at least four related senses: as a label sometimes applied to TubeViT’s sparse video-tube tokenization for joint image–video learning, as shorthand for a single stitched ViT backbone spanning multiple performance–efficiency trade-offs, as an elastic inference regime obtained by post-pretraining structured pruning, and as the title of a reconfigurable FPGA accelerator for edge ViT inference [2212.03229][2307.00154][2510.17700][2606.31938]. Taken together, these usages indicate that “FlexViT” is less a single canonical architecture than a recurring design objective: preserving ViT performance while relaxing fixed assumptions about input structure, model scale, or hardware execution.

## 1. Terminological scope

Within the records surveyed here, “FlexViT” does not refer to one unique method. Instead, it names or informally describes several mechanisms for making ViTs adaptable at different layers of the stack.

| Usage | Mechanism | Representative paper |
|---|---|---|
| TubeViT / sometimes referred to as “FlexViT” | Sparse video tubes for joint image and video learning | "Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning" [2212.03229] |
| Flexible backbone | Two-way stitching of pretrained ViTs into many subnetworks | "Stitched ViTs are Flexible Vision Backbones" [2307.00154] |
| Elastic inference | Single-shot structured pruning with one global ranking | "Elastic ViTs from Pretrained Models without Retraining" [2510.17700] |
| FPGA accelerator | Reconfigurable INT8 GEMM-based edge accelerator | "FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers" [2606.31938] |

This terminological multiplicity matters because the underlying technical objects differ substantially. One line of work modifies tokenization while keeping the ViT backbone unchanged; another stitches together pretrained model families; a third derives a continuum of sparse subnetworks from one pretrained model; and the hardware-oriented work reconfigures dataflow and tiling for heterogeneous ViT layers. A common misconception is to treat these as interchangeable variants of a single system. The available evidence instead supports a layered interpretation in which “flexibility” can target input representation, backbone composition, post-pretraining compression, or accelerator execution.

## 2. Sparse video tubes and joint image–video ViTs

TubeViT begins from a standard ViT encoder with input sequence \(z^0=\{z_i^0\}_{i=1}^N\), \(z_i^0\in\mathbb R^d\), followed by \(L\) layers of MSA and MLP blocks with residual connections and LayerNorm. To support video, it replaces the single 2D patch tokenizer by a small family of 3D tubes, including a conventional 2D-only tube \((T=1,H=16,W=16)\) with a large temporal stride and several spatio-temporal tubes such as \((8\times 8\times 8)\), \((16\times 4\times 4)\), and \((4\times 12\times 12)\), each applied with its own 3D convolution. All resulting tube tokens are concatenated into one sequence while the ViT backbone itself is kept exactly the same [2212.03229].

A video tube \(i\) is specified by kernel size \((T_i\times H_i\times W_i)\), stride \((T_i^s,H_i^s,W_i^s)\), and optional offset \((\tau_i,x_i,y_i)\). For video \(V\in\mathbb R^{T\times H\times W\times C}\), the number of tokens produced by tube \(i\) is

$$
N_i
=
\Big\lfloor\frac{T-T_i}{T_i^s}\Big\rfloor
\times
\Big\lfloor\frac{H-H_i}{H_i^s}\Big\rfloor
\times
\Big\lfloor\frac{W-W_i}{W_i^s}\Big\rfloor .
$$

Each token is a flattened 3D block linearly projected to \(\mathbb R^d\). With \(M\) tubes, the total token count is

$$
N_{\rm total}=\sum_{i=1}^M N_i,
$$

with \(M=4\) and \(N_{\rm total}\approx 400\!-\!700\) as the typical regime described in the paper. The sparse sampling strategy uses large temporal strides such as \(16\) and mixes long–narrow and short–wide tube shapes to cover space–time sparsely but diversely.

The positional encoding is fixed sine–cosine and aligned to the true center of each token in \((t,x,y)\) space. For hidden size \(d\) and \(j\in\{0,\ldots,d/6-1\}\),

$$
\omega_j = 1/(10{,}000^{\,j/(d/6)}),
$$

and the positional embedding \(p\in\mathbb R^d\) is constructed as

$$
p_{6j}=\sin(\omega_j t),\quad
p_{6j+1}=\cos(\omega_j t),\quad
p_{6j+2}=\sin(\omega_j x),\quad
p_{6j+3}=\cos(\omega_j x),\quad
p_{6j+4}=\sin(\omega_j y),\quad
p_{6j+5}=\cos(\omega_j y).
$$

Image-only inputs are handled by the special \(T=1,H=16,W=16\) tube, so the same encoder can be trained jointly on images and video clips. The training objective uses a shared ViT encoder and separate classification heads, for example one for ImageNet-1k and one for Kinetics-600, with cross-entropy computed only on the corresponding head:

$$
\mathcal L
=
\mathbb 1_{\rm img}\,CE(y_{\rm img},\hat y_{\rm img})
+
\mathbb 1_{\rm vid}\,CE(y_{\rm vid},\hat y_{\rm vid}).
$$

For adaptation from pretrained image ViTs, the method injects the new tube tokenizers with linear weights taken from a small jointly trained TubeViT, freezes the first \(S\) layers, and finetunes only layers \(S+1,\ldots,L\) plus classification heads. An optional gated skip is added from the raw input tokens:

$$
z^S
=
y^S+\mathrm{MLP}(\mathrm{LN}(y^S))+\tanh(\alpha)\,z^0,
$$

where \(\alpha\) is a learned scalar initialized to \(0\).

Empirically, the sparse-tube construction is presented as a way to retain full self-attention while avoiding dense token counts. For a dense \(32\times224\times224\) video, the description gives \(N_{\rm dense}\approx 32\times14\times14=6272\) tokens, versus \(N_{\rm sparse}\approx 559\) for TubeViT, implying \((6272/559)^2\approx125\times\) fewer attention operations. The reported benchmark summary includes 88.6% top-1 at \(0.87\) TFLOPs for TubeViT-B and 90.2% at \(9.53\) TFLOPs for TubeViT-L on Kinetics-400 with \(4\times3\) crops, compared with 89.9% at \(73.6\) TFLOPs for MTV-H; 91.5% for TubeViT-L on Kinetics-600; 83.8% on Kinetics-700; and 76.1% on Something-Something-V2. This suggests that, in this usage of the term, FlexViT denotes flexibility across visual modality rather than across model size or hardware.

## 3. Flexible backbones through stitching

SN-Netv2 defines flexibility at the backbone level by turning a family of pretrained plain ViTs into a single model that covers many subnetworks with different performance–efficiency trade-offs. The method is built on stitchable neural networks and introduces a systematically improved stitching framework for downstream task adaptation. The base models, or anchors, are standard plain ViTs of different sizes such as DeiT3-S, DeiT3-B, and DeiT3-L, each containing a patch embedding, \(N\) transformer blocks, and a classification head [2307.00154].

Let \(f_\theta\) be a small ViT and \(f_\phi\) a large ViT, with activations \(\mathcal A_{\theta,l}\subset\mathbb R^{N\times D_1}\) after block \(l\) and \(\mathcal A_{\phi,m}\subset\mathbb R^{N\times D_2}\) after block \(m\). Two learnable \(1\times1\) convolutional stitching layers are introduced:

$$
\mathcal S_1:\mathcal A_{\theta,l}\to\mathcal A_{\phi,m},
\qquad
\mathcal S_2:\mathcal A_{\phi,m}\to\mathcal A_{\theta,l'}.
$$

This yields Fast→Slow, Slow→Fast, Fast→Slow→Fast, and Slow→Fast→Slow routes. Representative configurations include

$$
F(\mathbf x)=f_{\phi,m\to N}\circ \mathcal S_1\circ f_{\theta,0\to l}(\mathbf x)
$$

for Fast→Slow and

$$
F(\mathbf x)=f_{\theta,l'\to N}\circ \mathcal S_2\circ f_{\phi,0\to m}(\mathbf x)
$$

for Slow→Fast. Allowing \(l,m,l',m'\) to vary produces a stitching space described as approximately \(10\times\) that of SN-Netv1.

Training samples subnetworks in a FLOPs-aware manner. Total FLOPs are discretized into buckets \(\{\tau_1,\dots,\tau_K\}\), using a step size of \(1\) G for ImageNet and \(10\) G for dense tasks. At each iteration, a budget bucket is sampled uniformly and then a stitch is sampled uniformly among those with that FLOPs value. The summary characterizes this as a resource-constrained sampling strategy, or ROS, that ensures each FLOPs level is seen equally often and avoids under-training anchors. The expected FLOPs per minibatch is exactly \(\frac1K\sum_k\tau_k\).

A further ingredient is low-rank adaptation of the stitching layers. Initial stitching matrices are obtained by least-squares matching,

$$
\mathbf M_1^{(0)}=\mathbf X_1^\dagger \mathbf X_2,\qquad
\mathbf M_2^{(0)}=\mathbf X_2^\dagger \mathbf X_1,
$$

and training freezes \(\mathbf M^{(0)}\) while learning only a low-rank update

$$
\Delta \mathbf M=\mathbf B\,\mathbf A,
\qquad
\mathbf B\in\mathbb R^{D_1\times r},\;
\mathbf A\in\mathbb R^{r\times D_2},\;
r\ll \min(D_1,D_2).
$$

According to the description, this steadies learning when both anchors must adapt to a new data domain and empirically produces a smoother, higher Pareto frontier.

At inference time, the model functions as a flexible backbone: a desired FLOPs budget is selected, one pre-evaluated stitch with approximately that budget is chosen, and the corresponding forward pass is executed with no retraining or weight loading. The switching overhead is described as negligible, since the extra modules are standard \(1\times1\) convolutions and memory is only about \(10\%\) above the largest single ViT.

The reported results quantify this flexibility across several tasks. On ImageNet-1K, anchors DeiT3-S and DeiT3-L are given as \(4.6\) G FLOPs / \(79.2\%\) and \(17.6\) G / \(85.1\%\), while SN-Netv2 yields about \(200\) stitches covering \(4\!-\!18\) G; example accuracies include roughly \(76\%\) at \(5\) G and \(82\%\) at \(10\) G, improving on SN-Netv1. On ADE20K, selected SN-Netv2 operating points are 45.6 mIoU at \(34\) G, 46.8 at \(88\) G, 49.0 at \(198\) G, and 51.4 at \(363\) G, versus 40.4 for SN-Netv1 at \(34\) G. On COCO-Stuff-10K, the reported points are 40.1 at \(34\) G, 42.7 at \(108\) G, and 48.2 at \(363\) G. On NYUv2, RMSE decreases from approximately 0.48 at \(4.6\) G to 0.42 at \(17.6\) G. Training efficiency is also reported: on ADE20K, separate training of DeiT3-S, -B, and -L costs \(75+90+174=339\) A100-GPU hours, versus \(120\) hours for one SN-Netv2 \(S/L\) run; on COCO-Stuff, the comparison is \(178\) hours versus \(60\) hours. In this line of work, FlexViT denotes a unified backbone that spans a discrete but rich set of stitched subnetworks.

## 4. Elastic ViTs through retraining-free pruning

SnapViT addresses flexibility after pretraining rather than during backbone construction. The input is a pretrained ViT and an unlabeled dataset \(D\), and the output is a continuum of elastic submodels that can be adjusted to any computational budget without retraining. The method combines self-supervised gradient information, blockwise grouping, an evolutionary approximation of global Hessian structure, and a single global importance ranking [2510.17700].

The first stage computes self-supervised DINO gradients on a small batch of approximately \(N_D\approx1000\) samples and forms a diagonal Hessian proxy per parameter:

$$
s_i=\frac1{N_D}\sum_{x\in D}\bigl\|\nabla_{\theta_i}\mathcal L^{\mathrm{SSL}}(x)\bigr\|^2.
$$

Parameters are then grouped into \(B\) functional blocks, such as FFN row–column groups or whole attention heads, with a binary membership matrix \(\mathbf M\in\{0,1\}^{N\times B}\). To approximate global off-diagonal Hessian structure, the method runs xNES over block-scaling vectors \(\mathbf c\sim\mathcal N(\boldsymbol\mu,\boldsymbol\Sigma)\). For each sampled \(\mathbf c^{(k)}\), it constructs a trial prunability vector

$$
\mathbf u^{(k)}=(\mathbf M\,\mathbf c^{(k)})\odot \mathbf s,
$$

prunes the smallest entries at several target sparsities such as \(\{10,30,50,60\}\%\), and evaluates representation drift by cosine similarity between PCA-compressed embeddings of the original and pruned networks. After \(T\) iterations, \(\Sigma^{-1}\) is interpreted as a proxy for the global Hessian component \(\mathbf H^{(g)}\approx \alpha\,\Sigma^{-1}\).

Final importance scores are

$$
P_i=s_i\,[\mathbf M\,\mathbf c_{\mathrm{final}}]_i,
$$

and these are globally ranked once. Any desired sparsity \(S\in[0,1]\) is then realized by zeroing the lowest-ranked scores, so that the subnetwork at sparsity \(S\) is determined directly by thresholding the fixed ranking. The stated objective is structured, single-shot pruning under a compute or parameter-count constraint while minimizing the induced second-order loss increase

$$
\delta\mathcal L\approx \tfrac12\,\delta\theta^\top \mathbf H\,\delta\theta,
$$

with \(\|\delta\theta\|^3\) neglected.

This elastic inference mechanism is explicitly budget-driven. At inference, a user specifies sparsity \(S\), computes a threshold \(\tau\) such that \(\#\{i:P_i<\tau\}=(1-S)N\), applies the binary mask \(\mathrm{mask}_{S,i}=1_{P_i\ge\tau}\), and runs \(f_{\theta\odot \mathrm{mask}_S}(x)\). Because the ranking is computed once, there is no per-budget retraining or optimization.

The experimental scope covers seven image-classification datasets, including ImageNet-1k, FGVC Aircraft, Oxford-Pets, DTD, EuroSAT, and CIFAR-10/100, as well as Pascal VOC 2012 for linear semantic segmentation. Backbones include self-supervised models such as DINO, DINOv3, and SigLIPv2, and supervised models such as DeIT, DeIT-III, and AugReg. Reported results include less than 5% k-NN and linear-probe drop for DINO ViT-B/16 at 40% sparsity; +15.1% versus SNIP-Mag and +53.2% versus LAMP for the linear metric; +7.0% versus SNIP and +12.3% versus FPTP for AugReg/DeIT ViT-B/16 at 50% sparsity; and, for DeIT-III ViT-H/14 at 50% sparsity, an average drop of 3.7% in k-NN and 3.4% in linear accuracy over seven datasets, with recovery within \(\pm1\%\) after one weight-correction step. The reported runtime is under five minutes on one A100 for models up to ViT-L/16, with longer xNES schedules needed for large self-supervised models such as DINOv3 and SigLIPv2. This suggests a notion of FlexViT in which elasticity is induced post hoc from a single pretrained model rather than encoded during initial architecture design.

## 5. FlexViT as an FPGA-based edge accelerator

In the most literal use of the name, FlexViT is a reconfigurable FPGA accelerator for efficient ViT inference on resource-constrained edge devices. It is built on SECDA-TFLite, which uses the TensorFlow-Lite delegate mechanism to intercept subgraphs in a TFLite model and redirect them to custom hardware kernels. FlexViT extends this framework with a custom delegate that recognizes both fully connected and convolutional operators at runtime, a lightweight host-side im2col transformation, and a compact control-packet protocol over AXI streams for layer configuration, including dimensions, mode flags, and quantization parameters [2606.31938].

The hardware–software co-design partitions responsibilities between an ARM host CPU and FPGA fabric. The ARM host loads and quantizes the TFLite model, performs im2col on convolution inputs, selects a dataflow mode by heuristic or DMA-traffic estimator, and streams activations, weights, and bias or scale parameters into the FPGA. The FPGA contains a scheduler, three GEMM cores with on-chip buffers and local accumulators, and a post-processing unit that applies INT8 requantization and packs results back to the host. Control is decoupled from compute, and AXI-stream handshakes are used to overlap DMA, computation, and post-processing.

The compute substrate is a unified INT8 GEMM engine. Each of the \(C=3\) GEMM cores has tile size \(T_N=64\) rows by \(T_M=64\) columns, reduction depth \(T_K=1024\), and SIMD width \(K_f=16\). The arithmetic is hybrid: for each 8-bit multiplication, the lower 8 bits are implemented in LUT logic and the upper 8 bits use DSP slices. The dataflow is output-stationary, so partial sums remain in 32-bit accumulator registers until the full \(K\) span has been processed.

At \(f_{\mathrm{clk}}=200\) MHz, the theoretical MAC throughput is

$$
P_{\mathrm{MAC}}
=
C\times K_f\times f_{\mathrm{clk}}
=
3\times16\times 200\times10^6
=
9.6\times10^9\ \mathrm{MAC/s},
$$

equivalent to

$$
P_{\mathrm{OPS}}
=
\frac{2P_{\mathrm{MAC}}}{10^{12}}
=
0.0192\ \mathrm{TOPS}.
$$

A single \(T_N\times T_M\times T_K\) tile requires

$$
N_{\mathrm{cycles}}
=
T_N\times T_M\times \frac{T_K}{K_f}
=
64\times64\times64
=
262{,}144\ \text{cycles},
$$

corresponding to a nominal compute latency of

$$
t_{\mathrm{tile}}
=
\frac{262{,}144}{200\times10^6}
=
1.31\ \mathrm{ms}.
$$

Both FC and convolutional layers are mapped to GEMM. For convolution, host-side im2col rearranges \(X\in\mathbb R^{N\times C\times H\times W}\) into

$$
X_{\mathrm{col}}\in\mathbb R^{(N\,H_{out}\,W_{out})\times(C\,K_h\,K_w)},
$$

with kernels reshaped into

$$
W_{\mathrm{gemm}}\in\mathbb R^{(C\,K_h\,K_w)\times M},
$$

so that

$$
Y=X_{\mathrm{col}}\times W_{\mathrm{gemm}}
\in
\mathbb R^{(N\,H_{out}\,W_{out})\times M}.
$$

FlexViT then selects between two runtime dataflows. Input-Broadcast broadcasts one input tile of size \(T_N\times T_K\) to all three cores and partitions the weight space across cores; Weight-Broadcast broadcasts one weight tile of size \(T_K\times T_M\) and partitions the input space. A simple criterion for FC layers is

$$
\text{mode}=
\begin{cases}
\text{Input-Broadcast}, & \tilde M\ge C\,T_M,\\
\text{Weight-Broadcast}, & \text{otherwise}.
\end{cases}
$$

More generally, the mode is chosen by comparing estimated bytes transferred:

$$
b_{IB}
=
\Bigl\lceil\frac{\tilde N}{T_N}\Bigr\rceil T_N\tilde K
+
\Bigl\lceil\frac{\tilde M}{C\,T_M}\Bigr\rceil C\,T_M\,\tilde K,
$$

$$
b_{WB}
=
\Bigl\lceil\frac{\tilde M}{T_M}\Bigr\rceil T_M\tilde K
+
\Bigl\lceil\frac{\tilde N}{C\,T_N}\Bigr\rceil C\,T_N\,\tilde K.
$$

A further key mechanism is depth-first tiling. On-chip buffers are dimensioned to hold a full \(K\)-depth tile,

$$
T_N\times T_K \quad \text{and} \quad T_K\times T_M,
$$

so all MACs for one \(T_N\times T_M\) output tile can be accumulated locally in a single pass. The description contrasts this with a naive tiled GEMM, where partial sums would incur approximately

$$
D_{\mathrm{psum}}\approx 2\times N\,M\times \frac{K}{T_K}
$$

bytes of 32-bit traffic; under the proposed strategy, only the final 8-bit outputs are streamed out, approximately

$$
D_{\mathrm{out}}\approx N\,M\times1.
$$

Implementation is reported on a PYNQ-Z2 FPGA at 200 MHz. Resource utilization is 71.8% of 220 DSP slices, 83.9% of 140 BRAM18 blocks, 69.3% of 53,200 LUTs, and approximately 65% of flip-flops. The depth buffer \(T_K=1024\) covers all but one layer across five ViT variants; the final \(1\times1\) classification layer in EfficientViT-b1 has \(K=1536\) and would require depth tiling if offloaded.

The evaluation includes ViT-T, DeiT-T, Swin-T, MobileViT-S, and EfficientViT-b1, all quantized to INT8 TFLite, with an ARM Cortex-A9 CPU baseline measured over 100 runs. The reported peak layer speedup is 2.74× and the peak end-to-end speedup is 1.40×. Representative results include ViT-T total latency dropping from 2.04 s on CPU to 1.46 s on CPU+accelerator, DeiT-T from 2.04 s to 1.46 s, and Swin-T from 7.67 s to 5.68 s. Off-chip traffic is reported to fall by up to 50% relative to a naive tiled GEMM with partial-sum exchanges. Under INT8 quantization with per-tensor scales for FC and per-channel scales for CONV, all five models achieve greater than 99% cosine similarity to the floating-point baseline. In this hardware sense, FlexViT is not a model family but a deployment substrate for heterogeneous ViT operators.

## 6. Comparative perspective

The surveyed works define flexibility at different abstraction levels. TubeViT changes tokenization while keeping the encoder unchanged; SN-Netv2 combines pretrained anchors into a large stitching space; SnapViT derives a continuum of sparse subnetworks from one pretrained model; and the FPGA FlexViT adapts execution strategy to operator heterogeneity and edge resource limits [2212.03229][2307.00154][2510.17700][2606.31938]. This layered view is the most faithful way to reconcile the shared label.

Several distinctions follow. First, flexibility need not imply dynamic token sparsity at inference time: TubeViT uses fixed sparse tubes and still computes full self-attention over the resulting token set. Second, flexibility need not imply a single dense backbone: SN-Netv2 explicitly exposes many stitched routes, whereas SnapViT exposes many masked subnetworks through one global score ranking. Third, hardware FlexViT does not compress or retrain the model; it re-expresses FC and CONV as INT8 GEMM and switches dataflow modes at runtime.

A further misconception is that all forms of FlexViT address the same bottleneck. The bottleneck in TubeViT is quadratic attention over dense video tokens; in SN-Netv2 it is the cost of separately training and deploying multiple ViT scales; in SnapViT it is the rigidity of predetermined pretrained sizes and the need for label-free, retraining-free adaptation; in FPGA FlexViT it is the variation in tensor shapes and operator types across modern hybrid ViTs. The shared vocabulary therefore marks a common systems objective—adaptability—rather than a single invariant mechanism.

Taken together, these papers suggest that FlexViT has become a useful umbrella for research on configurable ViTs. The term spans modality-unified tokenization, FLOPs-conditioned backbone stitching, sparsity-conditioned elastic inference, and reconfigurable edge acceleration. For researchers, the principal value of the term lies not in naming one architecture but in identifying a design space in which ViTs are made adjustable without abandoning the core transformer formulation.

Source: https://www.emergentmind.com/topics/flexvit