---
title: Double-Condensing Attention Condenser
url: https://www.emergentmind.com/topics/double-condensing-attention-condenser
type: topic
---

# Double-Condensing Attention Condenser

A Double-Condensing Attention Condenser (DC-AC) is an efficient attention mechanism specifically designed to enable fine-grained, highly selective self-attention in deep neural networks while maintaining an extremely low computational and memory footprint. Originating in the context of resource-constrained machine learning (TinyML), DC-AC modules have been extensively adopted for both edge-device inference and large-scale classification tasks due to their condensed representations and selective focus mechanisms. The architecture is characterized by multi-stage feature condensation and parallel attention computation, achieving superior accuracy-to-efficiency trade-offs in tasks such as medical image analysis, particularly in skin lesion classification [2311.11656][2208.06980].

## 1. Fundamental Model Architecture

A DC-AC block receives an activation tensor $X \in \mathbb{R}^{H \times W \times C}$, with $N=H \cdot W$ tokens. The module contains two parallel "attention condenser" branches, each consisting of:

- **Condensation:** Spatial and channel reduction using a $1 \times 1$ convolution, depthwise convolution, and another $1 \times 1$ convolution.
- **Bottleneck Embedding:** Transformation to a low-dimensional subspace, yielding $E_i \in \mathbb{R}^{N_i \times d}$, where $N_i \ll N$, $d \ll C$.
- **Expansion:** Projection of this embedding back to the value space $V_i \in \mathbb{R}^{N_i \times C}$.

In parallel, the module computes a global query projection $Q \in \mathbb{R}^{N \times d}$ from $X$. Each branch produces keys $K_i \in \mathbb{R}^{N_i \times d}$. For each branch $i$, attention weights are computed as $A_i = \mathrm{softmax}(Q K_i^\top / \sqrt{d})$, with outputs $O_i = A_i V_i$. The two outputs are fused (summed or concatenated plus pointwise convolution) and undergo a residual addition with the input $X$, optionally followed by normalization and activation. The output retains the input shape $Y \in \mathbb{R}^{H \times W \times C}$ [2311.11656].

Typical dimensioning (reference implementation) uses $d=16$, $N_1=14^2$, $N_2=7^2$, with $C=64$ on $56 \times 56$ inputs, and all projections structured for minimal parameter count.

## 2. Mathematical Formulation

For input $X \in \mathbb{R}^{N \times C}$ and each branch $i=1,2$:

1. **Condensation:** $C_i = f_\textrm{condense}_i(X) \in \mathbb{R}^{N_i \times C_i}$
2. **Embedding:** $E_i = f_\textrm{embed}_i(C_i) = C_i W_{E,i} \in \mathbb{R}^{N_i \times d}$
3. **Expansion:** $V_i = f_\textrm{expand}_i(E_i) = E_i W_{V,i} \in \mathbb{R}^{N_i \times C}$

Queries: $Q = X W_Q \in \mathbb{R}^{N \times d}$

Keys: $K_i = E_i W_{K,i} \in \mathbb{R}^{N_i \times d}$

Attention and output per branch:
\[
A_i = \mathrm{softmax}\left(\frac{Q K_i^\top}{\sqrt{d}} \right) \in \mathbb{R}^{N \times N_i}
\]
\[
O_i = A_i V_i \in \mathbb{R}^{N \times C}
\]

Branch fusion and output:
\[
O = O_1 + O_2
\]
\[
Y = \mathrm{LayerNorm}(X + O)
\]
[2311.11656]

## 3. Implementation Workflow

The following pseudocode summarizes the DC-AC block computation [2311.11656]:

```python
def DCACBlock(X):
    # X: [H, W, C]
    N = H*W
    Q = reshape(X, [N, C]) @ W_Q  # [N, d]
    branch_outputs = zero([N, C])
    for i in {1,2}:
        C_i = Conv1x1_reduce_i(X)
        C_i = DepthwiseConv(C_i, stride=s_i)
        C_i = Conv1x1_embed_i(C_i)
        E_i = reshape(C_i, [N_i, d])
        K_i = E_i @ W_K_i
        V_i = E_i @ W_V_i
        A_i = softmax((Q @ K_i.T) / sqrt(d))
        O_i = A_i @ V_i
        branch_outputs += O_i
    O = reshape(branch_outputs, [H, W, C])
    Y = X + O
    Y = Activation(LayerNorm(Y))
    return Y
```

This dual-branch strategy enables a high compression ratio for attention computation, with O($N \cdot N_i$) cost per branch, in contrast to the O($N^2$) cost of conventional full-rank attention.

## 4. Integration in Network Topologies and Constraints

DC-AC modules are embedded in deep network topologies, including multi-column architectures where each column uses DC-AC blocks at multiple feature resolutions. For instance, in skin lesion classification:

- Input: $224 \times 224 \times 3$
- Stem: $3 \times 3$ convolution, stride 2
- Four parallel columns with interleaved conv and DC-AC blocks
- Unified feature map via concatenation and $1 \times 1$ conv
- Global average pooling $\rightarrow$ dense $\rightarrow$ sigmoid head

This configuration achieves approximately 1.6M parameters and 0.325G FLOPs per $224 \times 224$ input [2311.11656].

AttendNeXt [2208.06980] employs a related DC-AC block with two stages of channel-wise condensation ($C \rightarrow d_1 \rightarrow d_2$), projecting the final condensed embedding to query, key, and value via linear subspaces, and restores the representation via expansion with batch normalization and ReLU at each step. Machine-driven generative synthesis techniques are used to fix the degrees of condensation (e.g., $d_1 \approx C/4$, $d_2 \approx C/8-C/16$), anti-aliased downsampling is imposed, and the design is constrained to avoid strided pointwise convolutions.

## 5. Quantitative Evaluation and Efficiency

When applied to the SIIM-ISIC skin cancer dataset, a DC-AC-powered network reached an AUROC of 0.8865 (private test set) with a model size under 7MB (FP32) or under 2MB (8-bit quantized). Single-image inference requires only 325M FLOPs, translating to 10ms latency on an ARM Cortex-M55 or 3ms on Raspberry Pi 4 [2311.11656]. Comparison models such as MobileViT-S (5.6M parameters, 2.03G FLOPs) achieved only 0.8566 AUROC; Cancer-Net SCa variants reached up to 0.7430 [2311.11656].

AttendNeXt with DC-AC blocks achieved 75.8% ImageNet top-1 with 3.6MB and $>$10$\times$ throughput over FB-Net C (ARM Cortex-A72 benchmark). DC-AC blocks are routinely 1.2–1.5$\times$ smaller and 6–10$\times$ faster than high-accuracy MobileNet and MobileViT baselines [2208.06980].

Error tolerance to reduced precision and pruning is demonstrated: 8-bit post-training quantization results in $<1\%$ AUROC loss, and 20% structured channel pruning yields a 25% size reduction with $<0.5\%$ AUROC loss [2311.11656].

## 6. TinyML Considerations

DC-AC is explicitly engineered for TinyML constraints:

- **Memory:** Weights occupy 1.6M$\times$4B = 6.4MB (FP32) or 1.6MB (INT8), and peak activation memory is $\leq$3MB, fitting well within a 16MB SRAM budget.
- **Compute:** Depthwise-separable and condenser operations minimize multiply-accumulate counts. The double-condense structure restricts self-attention to reduced sets (O($N \cdot N_i$)), avoiding prohibitive scaling.
- **Compression:** Quantization and structured pruning can be performed aggressively without substantial loss in accuracy.

These properties facilitate deployment on mobile and embedded systems for applications such as on-device dermoscopy and tele-dermatology [2311.11656].

## 7. Relation to Prior and Parallel Work

DC-AC is conceptually and architecturally distinct from the double-attention modules in $A^2$-Nets [1810.11579]. The latter introduce "gather and distribute" second-order attention pooling, wherein features are aggregated globally and reallocated via a two-stage attention mechanism, achieving efficiency via factorization to O($d^2 N$) cost per block. However, DC-AC relies on a multi-stage condensation and lightweight self-attention applied to compressed representations within each branch, optimizing for different computational bottlenecks typical in embedded hardware.

The impact of DC-AC is most pronounced in use-cases where both fine-grained attention and tight memory or latency budgets are required. The approach exemplifies a transition from monolithic attention computation to modular, highly controlled, hardware-aware neural network design paradigms.

## Table: Key Quantitative Comparisons

| Model            | Params (MB) | Throughput (A72, rel.) | Top-1/ImageNet | AUROC (SIIM-ISIC) |
|------------------|-------------|------------------------|----------------|-------------------|
| AttendNeXt (DC-AC) | 3.6       | 10.5x                  | 75.8%          | (not reported)    |
| MobileViT-XS     | 2.8         | 4.0x                   | 74.7%          | 0.8566            |
| MobileNetV3-L    | 4.1         | 1.7x                   | 75.6%          | –                 |
| FB-Net C         | 4.5         | 1.0x                   | 74.7%          | –                 |
| DC-AC (skin cancer) | 1.6      | –                      | –              | 0.8865            |

All results from [2311.11656][2208.06980]. These results affirm the heightened throughput and competitive performance of DC-AC-equipped architectures in both generic and skin image-specific benchmarks.

---

DC-AC and its variants establish a paradigm for selective attention that is tractable for deployment under stringent TinyML requirements. By fusing efficient condensation, low-rank attention, parallelism, and hardware-aware design, DC-AC enables advanced visual recognition at dynamic, embedded, and clinical endpoints with minimal losses in discriminative power [2311.11656][2208.06980].

Source: https://www.emergentmind.com/topics/double-condensing-attention-condenser