DIDB-ViT: High-Fidelity Binary Vision Transformer

• DIDB-ViT is a binary vision transformer that maintains high representational fidelity while enabling efficient edge deployment for tasks like image classification and segmentation.
• It introduces informative attention mechanisms with local context recovery and frequency decomposition via Haar wavelets to mitigate information loss due to binarization.
• Enhanced RPReLU activation with per-token adjustments boosts discriminative power, achieving strong benchmark results on CIFAR-100 and ImageNet-1K.

DIDB-ViT (“High-Fidelity Differential-information Driven Binary Vision Transformer”) is a vision transformer architecture that achieves high representational fidelity under strict binarization constraints, enabling practical deployment of vision transformers (ViTs) on resource-limited edge devices without the performance drop commonly associated with existing binary ViT approaches (2507.02222). DIDB-ViT preserves both weights and activations in a binary format throughout the network, while introducing a set of structural innovations—namely, an informative attention module, frequency decomposition via Haar wavelets, and an enhanced activation function—to mitigate information loss and maintain discriminative power on image classification and segmentation tasks.

1. Motivation for Binary Vision Transformers

Deploying ViTs on edge devices demands extreme quantization to reduce memory and computation, but naïve binarization of both weights and activations in ViTs often leads to severe accuracy loss or a reliance on full-precision modules, negating efficiency gains. DIDB-ViT is developed to address these limitations by focusing on information preservation throughout the binarized pipeline. Its design goal is to maintain as much of the fine-grained representation and attention diversity of full-precision models as possible, while remaining compatible with efficient XNOR and popcount-based binary operations.

2. Informative Attention via Differential Information

In standard (full-precision) ViTs, attention updates can be formulated as an aggregation of differential information between a token and others, with each token update given by:

$$v_i^t = v_i^{t-1} + \sum_{j\neq i} w_j (v_j^{t-1} - v_i^{t-1}),$$

where $w_j$ are attention-derived weights. Binarization—without further measures—flattens these weights, destroying nuance in difference contributions and resulting in information loss.

DIDB-ViT addresses this by enhancing the binary attention update formula:

$$v_i^t = \beta \cdot v_i^{t-1} + \alpha \sum_{j\in\Upsilon} B(v_j^{t-1}) - \gamma \sum_{\ell\in\Psi} B(v_\ell^{t-1}),$$

where:

• $B(\cdot)$ denotes the binarization function,
• $\Upsilon$ is the set of positively attended tokens,
• $\Psi$ represents a local 8-neighborhood around $v_i$ (reflecting spatial context in the token arrangement),
• $\beta$, $\alpha$, $\gamma$ are learnable scaling factors, with $\beta$ maintained in full precision to act as a shortcut preserving key context.

This informative attention design reconstructs some of the “differential” information erased during binarization, notably by including local neighborhood context to the update, which is vital for vision tasks demanding spatial acuity.

3. Frequency-Decomposed Similarity with Haar Wavelet

Computation of similarities between binary Query ($Q$) and Key ($K$) representations in attention is susceptible to further information loss after binarization, especially due to the removal of mid- and high-frequency features that are important for fine-grained visual discrimination.

DIDB-ViT applies a non-subsampled discrete Haar wavelet transform to decompose inputs into low and high frequency bands:

• $X^L$ (low-frequency components)
• $X^H$ (high-frequency components)

These are processed through separate binary linear layers (BL) to obtain frequency-specific queries and keys:

$Q_e = \text{cat}[BL_Q^L(X^L), BL_Q^H(X^H)] + X$

$K_e = \text{cat}[BL_K^L(X^L), BL_K^H(X^H)] + X$

The binary similarity matrix is then calculated as:

$S_{HF} = B(Q_e) \otimes B(K_e)^T$

using XNOR-popcount binary operations, with $\otimes$ denoting binary matrix multiplication. By integrating high- and low-frequency cues, this approach preserves critical discriminative information for both global structure and fine details.

4. Improved RPReLU Activation for Binary Networks

Traditional RPReLU activations adjust the distribution of activations per channel via learnable offsets and slopes, but apply the same shift across all spatial positions (tokens) of a channel. DIDB-ViT introduces a per-token parameter $t_j$, expanding the flexibility of the nonlinearity:

$$F_{i,j} = \begin{cases} (X_{i,j} - m_i) + n_i + t_j & \text{if } X_{i,j} > m_i \ k_i \cdot (X_{i,j} - m_i) + n_i + t_j & \text{otherwise} \end{cases}$$

where $m_i$, $n_i$, $k_i$ are per-channel learnable parameters, and $t_j$ is learned per spatial token. This increases representational capacity, enabling the model to better fit batch-wise activation distributions after binarization and supports more robust learning without significant parameter overhead (only $N + 3C$ additional parameters per layer).

5. Empirical Performance and Benchmark Results

DIDB-ViT is benchmarked across several datasets and architectures:

• On CIFAR-100, a two-stage training strategy yields 76.3% Top-1 accuracy, outperforming prior state-of-the-art binary ViT models and exceeding even some full-precision baselines.
• On ImageNet-1K, DIDB-ViT surpasses related binary methods by 5–9% Top-1 accuracy across multiple ViT variants (including DeiT-Tiny and DeiT-Small).
• In semantic segmentation tasks (ADE20K, road segmentation), DIDB-ViT achieves higher pixel accuracy (pixAcc) and mean Intersection-over-Union (mIoU) than previous binary methods.

These results demonstrate that DIDB-ViT not only retains but advances the capabilities of fully binarized ViT models for both classification and dense prediction tasks.

6. Applications, Edge Deployment, and Broader Implications

DIDB-ViT is particularly well suited for scenarios where memory and computation are at a premium:

• Edge AI deployments (mobile devices, IoT, embedded systems) benefit directly from the reduction in model size (due to binary parameterization) and efficiency of binary arithmetic (XNOR and popcount operations).
• In domains where inference latency and power consumption are critical, DIDB-ViT offers a highly competitive option without compromising prediction quality.
• The design principles—recovering differential and frequency-specific information in the binary attention mechanism and expanding activation expressivity—suggest transferable insights for other domains, including multi-modal transformers, object detection, and efficient natural language processing.

The methodological advances of DIDB-ViT provide a foundation for further exploration into fully binarized, information-preserving transformer architectures suitable for widespread real-world deployment, particularly where resource budgets have previously precluded the use of sophisticated transformer-based models.