--- title: 'ACC-ViT: Atrous Multi-Branch Vision Transformer' url: https://www.emergentmind.com/topics/atrous-multi-branch-formulation-acc-vit type: topic --- # ACC-ViT: Atrous Multi-Branch Vision Transformer The Atrous Multi-Branch Formulation (ACC-ViT) is a contemporary architectural paradigm in vision transformer design in which multi-scale local and global context is encapsulated using a spatial pyramid of dilated (“atrous”) attention windows and convolutions. This approach directly fuses regional (windowed) and sparse (grid) attention schemes by deploying learnable, parallelized attention branches at multiple dilation levels and adaptively fusing their outputs via a gating mechanism. The design yields a hybrid architecture, consistently surpassing both pure CNN and transformer-only models across a range of vision benchmarks, including classification, detection, segmentation, and zero-shot tasks [2406.08859], [2403.04200]. ## 1. Principle of Atrous Multi-Branch Formulation At the core of ACC-ViT is the intuition that vision models must simultaneously capture fine-grained details and long-range dependencies. Standard regional (windowed) attention restricts interactions to local non-overlapping regions of the image, effectively preserving hierarchical spatial structure but at the cost of losing global contextual information. In contrast, sparse (grid-based) attention enables broader context aggregation by interacting across distant image regions, but can dilute essential locality and introduce irrelevant dependencies. ACC-ViT resolves this duality by constructing $K+1$ parallel sets of feature windows on the input tensor $X \in \mathbb{R}^{C \times H \times W}$: an undilated partition ($d_0=1$) and $K$ dilated (stride-$d_k$) partitions ($d_k\in\{2,4,8\}$ in early stages). Each partition undergoes standard windowed multi-head self-attention (W-MHSA), and the per-branch outputs are adaptively gated and fused before further processing by a shared MLP [2406.08859], [2403.04200]. ## 2. Formal Definition and Mathematical Structure ### Regional (Windowed) Attention Divide $X$ into $N = \frac{HW}{w^2}$ non-overlapping windows of size $w \times w$. For each window $i$: \[ X^{(1)}_i \in \mathbb{R}^{w^2 \times C} \] \[ Q^{(1)}_i = X^{(1)}_i W_Q,\quad K^{(1)}_i = X^{(1)}_i W_K,\quad V^{(1)}_i = X^{(1)}_i W_V \] \[ A^{(1)}_i = \mathrm{softmax}\left(\frac{Q^{(1)}_i (K^{(1)}_i)^T}{\sqrt{d_k}} + B^{(1)}\right) \] \[ Y^{(1)}_i = A^{(1)}_i V^{(1)}_i \] ### Sparse (Grid) Attention Sample $X$ on a coarse grid, stride $s>1$: \[ X^{(\mathrm{sparse})} = P_{\mathrm{sparse}}(X) \in \mathbb{R}^{M \times C} \] with $M \ll HW$. ### Atrous Attention (Fusion) For each $d\in\{1,2,4,8\}$, the atrous windowing operator $P_d$ extracts stride-$d$ sampled windows: \[ X^{(d)}_i = P_d(X)_i \in \mathbb{R}^{w^2 \times C} \] Compute windowed attention in parallel: \[ Q^{(d)}_i = X^{(d)}_i W_Q,\quad K^{(d)}_i = X^{(d)}_i W_K,\quad V^{(d)}_i = X^{(d)}_i W_V \] \[ A^{(d)}_i = \mathrm{softmax}\left(\frac{Q^{(d)}_i (K^{(d)}_i)^T}{\sqrt{d_k}} + B^{(d)}\right) \] \[ Y^{(d)}_i = A^{(d)}_i V^{(d)}_i \] The outputs $\{Y^{(d)}\}$ are then fused using an adaptive gating network: \[ g = \mathrm{softmax}(W_g\,\mathrm{GAP}(X)) \in \mathbb{R}^{k+1},\quad \sum_{d} g_d = 1 \] \[ Y_{\mathrm{fused}} = \sum_{d\in D \cup \{1\}} g_d\, \odot\, Y^{(d)} \] This output is propagated through a shared two-layer MLP, yielding the final block output [2406.08859], [2403.04200]. ## 3. Architectural and Implementation Choices ACC-ViT stacks blocks alternating between Atrous Multi-Branch Attention and Atrous Multi-Branch Inverted Residual Convolution (MBConv) layers. In the MBConv branch, the 3×3 depthwise convolution is replaced by parallel atrous convolutions with dilations $d=1,2,3$, each gated and then fused before channel reduction and residual addition. Both attention and convolution branches use adaptive gating, and all block outputs are fused via a single shared MLP. Key architectural parameters include: - Embedding dimension per stage: $\{64, 96, 192, 384, 768\}$ (depending on variant). - Head count: typically $\{2,4,8,16\}$ for Tiny, up to $\{3,6,12,24\}$ for Base. - Window size: $w=7$ for all stages. - Dilation sets ($D$): largest ($\{2,4,8\}$) at highest spatial resolution, reduced incrementally in subsequent stages [2406.08859], [2403.04200]. ## 4. Training Regimes and Optimization ACC-ViT models are trained on ImageNet-1K for 400 epochs using AdamW with a base learning rate of $1\times 10^{-3}$ (reduced for larger models), weight decay of $0.05$, and a cosine annealing schedule with 32-epoch linear warmup. Augmentation utilizes RandAugment, Mixup ($\alpha=0.8$), CutMix ($\alpha=0.8$), and label smoothing ($0.1$). Stochastic depth rates extend up to $0.3$ for larger variants, and an EMA of model weights is maintained every 32 steps. All projections employ Xavier initialization, except transformer projections, which use TorchVision defaults [2406.08859]. ## 5. Benchmarking, Ablation, and Empirical Insights ### Performance Comparison On ImageNet-1K (224×224), ACC-ViT-Tiny ($28.4$M params, $5.7$G FLOPs) reaches $84.0\%$ top-1 accuracy, exceeding MaxViT-Tiny by $+0.35\%$ while using $9.1\%$ fewer parameters. ACC-ViT-Small ($62.9$M, $11.6$G) achieves $85.2\%$ top-1 ($+0.15\%$ over MaxViT-Small with $9.7\%$ fewer params). All ACC-ViT variants dominate the full pareto front among ViTs and multi-axis transformer designs [2406.08859], [2403.04200]. ### Ablation Studies Analysis (nano variant) reveals the impact of the individual components: | Backbone Modification | Acc (%) | |-------------------------------------------------------|---------| | Single-branch MBConv + windowed attention | 79.50 | | + Atrous convolutions + multi-branch attention | 81.52 | | + Adaptive gating (conv and attn branches) | 82.41 | | Replace shared MLP by ConvNeXt block | 81.85 | Adaptive gating of branches improves performance by $+0.89\%$, and the shared MLP between branches is superior to ConvNeXt-style alternatives. The addition of parallel atrous convolutions and multi-branch attention jointly yield a significant performance increase ($+2.02\%$ top-1 for the ablated configuration). The costs are limited: a $<10\%$ increase in parameters and FLOPs yields a $+2.9\%$ increase in top-1 accuracy [2406.08859], [2403.04200]. ## 6. Broader Applicability and Limitations ACC-ViT demonstrates substantial advantages in downstream tasks beyond classification. On medical image analysis benchmarks (HAM10000, EyePACS, BUSI), ACC-ViT-T consistently outperforms Swin-T, ConvNeXt-T, and MaxViT-T. In frozen-backbone detection/segmentation (using Mask R-CNN + FPN on COCO), ACC-ViT-T beats MaxViT-T in both $\mathrm{AP_{bbox}}$ and $\mathrm{AP_{mask}}$ at multiple resolutions. In zero-shot vision-language retrieval (ELEVATER benchmark), ACC-ViT-T surpasses MaxViT-T and ConvNeXt-T on 13/20 tasks [2403.04200]. Limitations include reliance on fixed dilation rates, lack of scaling experiments to very large backbones or extremely high input resolutions, and the added inference overhead of the gating network. Further research into learnable or conditional dilation rates and additional efficiency optimizations are proposed as future directions [2403.04200]. ## 7. Theoretical and Empirical Significance ACC-ViT’s Atrous Multi-Branch formulation fundamentally bridges the gap between local-hierarchical and global-context modeling in vision transformers. By constructing aligned sets of dilated attention windows and fusing them via input-dependent gating, the model adaptively emphasizes informative context at every spatial position. This allows ACC-ViT not only to outperform pure windowed or pure grid/sparse models but also to surpass hybrid baselines such as MaxViT and ConvNeXt in parameter efficiency and accuracy. The design paradigm is extensible to other architectures and modalities wherever multiscale feature integration and adaptive spatial context are necessary [2406.08859], [2403.04200]. Source: https://www.emergentmind.com/topics/atrous-multi-branch-formulation-acc-vit