AttentionCap: Vision and Capacitance Extraction

• AttentionCap is a term for two distinct attention-based neural architectures: one for fine-grained visual classification (CAP) and another for full-chip capacitance matrix extraction in EDA.
• In visual classification, CAP employs contextual attentional pooling with region extraction, LSTM encoding, and NetVLAD aggregation to achieve high accuracy on multiple benchmarks.
• In EDA, the Transformer-based model uses attention, Gram factorization, and a symmetric output layer to efficiently predict capacitance matrices while satisfying strict physical constraints.

Searching arXiv for the specified AttentionCap-related papers and closely related work. AttentionCap is a name used in arXiv literature for two distinct attention-based neural architectures. In computer vision, it denotes Context-aware Attentional Pooling (CAP, a.k.a. AttentionCap), a module for fine-grained visual classification that sits on top of a standard CNN backbone and combines pixel-level self-attention, integral-region extraction, bilinear re-sampling, region-to-region attention, LSTM-based spatial encoding, and differentiable NetVLAD-style aggregation (Behera et al., 2021). In electronic design automation (EDA), it denotes a customized Transformer for full-chip capacitance-matrix learning that uses a Gram representation framework, a physics-aligned symmetric-attention output layer, a normalized Laplacian loss, and a process-node embedding to support multi-layer and multi-node extraction (Huang et al., 6 Jun 2026). The shared label therefore refers not to a single model family, but to two architectures that apply attention to structured local interactions in different technical domains.

1. Terminological scope and domain separation

The two uses of AttentionCap differ in objective, input representation, and output structure. CAP for fine-grained visual classification consumes an input image resized and cropped to $224\times224$, processes it through a CNN backbone, and predicts a category label through softmax classification. Transformer-based AttentionCap for capacitance extraction consumes a sequence of rectangular Manhattan conductors $\mathcal S\in\mathbb R^{n\times4}$ together with a process-node identifier, and predicts an entire capacitance matrix $\hat C\in\mathbb R^{n\times n}$ in one pass (Behera et al., 2021, Huang et al., 6 Jun 2026).

Usage of “AttentionCap”	Domain	Core output
Context-aware Attentional Pooling (CAP)	Fine-grained visual classification	Class prediction $\hat y$
AttentionCap	Full-chip capacitance extraction	Capacitance matrix $\hat C$

This separation is important because the two systems share neither training objective nor physical assumptions. CAP is optimized by standard categorical cross-entropy for subcategory recognition, whereas the EDA model is constrained by properties of capacitance matrices, including symmetry, positive semi-definiteness, non-positive off-diagonals, and zero row sums. A plausible implication is that the identical name reflects a common reliance on attention as an aggregation mechanism, not architectural equivalence.

2. Context-aware attentional pooling for fine-grained visual classification

CAP is designed for fine-grained recognition, where significant variance within the same subcategory and subtle variance among different subcategories make discriminative localization difficult. The architecture sits on top of off-the-shelf CNN backbones including ResNet-50, Inception-V3, Xception, DenseNet-121, NASNet-Mobile, and MobileNetV2, and consists of two novel modules: a context-aware attentional pooling module and a learnable feature-encoding plus classification module (Behera et al., 2021).

At a pipeline level, the model takes an input image, produces a backbone feature map $W\times H\times C$, applies pixel-level self-attention on every location, generates a set of integral regions at multiple scales, re-samples each region to a fixed size by bilinear pooling with sub-pixel-gradient propagation, and applies learned region-to-region attention to fuse local and surrounding context. The resulting context-aware descriptors $\{c_r\}$ are then passed through an LSTM, aggregated by a differentiable NetVLAD-style encoding, and classified by a final softmax layer.

A central component is its use of bilinear interpolation for region extraction. For an integral region $r(i,j,X,Y)$ extracted from $o\in\mathbb R^{W\times H\times C}$ and re-sampled to a fixed spatial size $w\times h$, CAP uses

$\mathcal S\in\mathbb R^{n\times4}$0

Because the bilinear kernel aggregates only the four nearest neighbors in the input, gradient propagation is local. The method exploits this property to force the network to capture very fine-grained, local variations inside each region.

The model then defines a dictionary of integral regions. Let $\mathcal S\in\mathbb R^{n\times4}$1 be the minimum patch size in the $\mathcal S\in\mathbb R^{n\times4}$2 feature map; anchors $\mathcal S\in\mathbb R^{n\times4}$3 are chosen on a grid, and for each anchor all patches $\mathcal S\in\mathbb R^{n\times4}$4 are collected until reaching the map boundary. Across all anchors this yields a set $\mathcal S\in\mathbb R^{n\times4}$5 of varying sizes and aspect ratios. Context-aware region-to-region attention is then computed through linear embeddings $\mathcal S\in\mathbb R^{n\times4}$6 and $\mathcal S\in\mathbb R^{n\times4}$7, pairwise compatibility

$\mathcal S\in\mathbb R^{n\times4}$8

normalized attention weights

$\mathcal S\in\mathbb R^{n\times4}$9

and the context-aware descriptor

$\hat C\in\mathbb R^{n\times n}$0

Each $\hat C\in\mathbb R^{n\times n}$1 is focused on a region while conditioned on all other regions, capturing both self and neighborhood context.

The stated rationale for CAP is fivefold: sub-pixel attention via bilinear pooling emphasizes very local changes; integral regions of varying scales cover small parts and larger context; cross-region attention learns which parts are mutually informative; the LSTM encodes spatial arrangement; and NetVLAD aggregation emphasizes the most discriminative hidden states. This suggests that CAP combines local sensitivity with broader contextual consistency rather than relying on explicit bounding-box or distinguishable part annotations.

3. Encoding, optimization, and empirical profile of CAP

After region-to-region attention, CAP applies global average pooling to each context-aware descriptor $\hat C\in\mathbb R^{n\times n}$2 to produce $\hat C\in\mathbb R^{n\times n}$3, then feeds the sequence $\hat C\in\mathbb R^{n\times n}$4 into an LSTM: $\hat C\in\mathbb R^{n\times n}$5 The hidden states $\hat C\in\mathbb R^{n\times n}$6 are aggregated by a differentiable NetVLAD-style encoding with $\hat C\in\mathbb R^{n\times n}$7 learnable cluster weight vectors and biases: $\hat C\in\mathbb R^{n\times n}$8 Flattening $\hat C\in\mathbb R^{n\times n}$9 yields the final classifier input: $\hat y$0 All parameters $\hat y$1 are learned end-to-end by minimizing the categorical cross-entropy

$\hat y$2

The implementation details are specific. Input images are resized to $\hat y$3, randomly rotated by $\hat y$4, scaled by $\hat y$5, and cropped to $\hat y$6. The backbone final convolutional feature map is up-sampled to $\hat y$7. The integral-region grid uses $\hat y$8, spatial stride $\hat y$9, and $\hat C$0 regions of sizes $\hat C$1 up to $\hat C$2. Bilinear re-sampling outputs patches with $\hat C$3. The LSTM hidden size is $\hat C$4, the NetVLAD cluster count is $\hat C$5, and optimization uses SGD with momentum $\hat C$6, initial learning rate $\hat C$7, decay by $\hat C$8 every 50 epochs, total 150 epochs, and batch size $\hat C$9. Training on a single Titan V (12 GB) takes approximately $W\times H\times C$0–$W\times H\times C$1 hours depending on dataset size, and inference cost is approximately $W\times H\times C$2–$W\times H\times C$3 ms per image including backbone, CAP, and classifier (Behera et al., 2021).

The reported evaluation covers eight benchmark datasets. CAP achieves $W\times H\times C$4 on Aircraft, $W\times H\times C$5 on Food-101, $W\times H\times C$6 on Stanford Cars, $W\times H\times C$7 on Stanford Dogs, $W\times H\times C$8 on CUB-200 Birds, $W\times H\times C$9 on Oxford Flowers, $\{c_r\}$0 on Oxford Pets, and $\{c_r\}$1 on NABirds. The method significantly outperforms the state of the art on six datasets and is very competitive on the remaining two; on Dogs and Flowers, the results $\{c_r\}$2 narrowly trail the best methods that use joint transfer with additional data.

Ablation studies isolate the contribution of the two modules. On Aircraft, Cars, and Pets, adding CAP alone yields $\{c_r\}$3–$\{c_r\}$4 points, adding encoding alone yields $\{c_r\}$5–$\{c_r\}$6 points, and combining both recovers the full performance. Tests with $\{c_r\}$7 indicate the best trade-off at $\{c_r\}$8 patches; larger $\{c_r\}$9 slows inference without clear gain. Top-2 accuracy is approximately $r(i,j,X,Y)$0 and Top-5 approximately $r(i,j,X,Y)$1 across multiple backbones, indicating highly confident fine-grained predictions.

4. Transformer-based AttentionCap for capacitance matrix learning

In EDA, AttentionCap addresses capacitance extraction for on-chip interconnects. The governing relation is

$r(i,j,X,Y)$2

where $r(i,j,X,Y)$3 is the capacitance matrix, $r(i,j,X,Y)$4 is the total capacitance of conductor $r(i,j,X,Y)$5, and $r(i,j,X,Y)$6 for $r(i,j,X,Y)$7 is its coupling to conductor $r(i,j,X,Y)$8. The matrix is physically symmetric, positive semi-definite, has non-positive off-diagonals, and zero row sums (Huang et al., 6 Jun 2026).

The motivation for the model comes from the limitations of previous methods. MLP-based models that take conductor coordinate tuples $r(i,j,X,Y)$9 either require a fixed number of conductors or predict only a single pair’s coupling at a time. CNN-based density-grid methods such as CNN-Cap and ResCap rasterize a small cross-sectional window into multi-channel 2D grids and regress one capacitance scalar; recovering the full $o\in\mathbb R^{W\times H\times C}$0 matrix requires re-rasterization and re-inference $o\in\mathbb R^{W\times H\times C}$1 times, and published models are tied to a fixed triple of metal layers. AttentionCap is therefore designed to natively learn the entire $o\in\mathbb R^{W\times H\times C}$2 capacitance matrix, support variable $o\in\mathbb R^{W\times H\times C}$3, any layer combination, and multiple technology nodes with one unified model.

The input representation is a sequence of $o\in\mathbb R^{W\times H\times C}$4 rectangular Manhattan conductors,

$o\in\mathbb R^{W\times H\times C}$5

which is projected to an initial embedding

$o\in\mathbb R^{W\times H\times C}$6

Here $o\in\mathbb R^{W\times H\times C}$7 is a learned linear projection, $o\in\mathbb R^{W\times H\times C}$8 denotes the current process node, and $o\in\mathbb R^{W\times H\times C}$9 is a learnable process-node embedding added uniformly to each conductor. Because pattern matching is permutation-equivariant, no positional encoding is added.

The model’s central formal device is the Gram representation framework. By the cited lemma, there exists $w\times h$0, $w\times h$1, such that

$w\times h$2

The extraction problem is therefore reformulated as learning

$w\times h$3

The scaling by $w\times h$4 parallels scaled dot-product attention and stabilizes training.

The mapping $w\times h$5 is implemented by an $w\times h$6-layer Transformer encoder with multi-head self-attention, SwiGLU feed-forward blocks, and RMSNorm before each sublayer. No causal masking or positional biases are used, ensuring full $w\times h$7 interactions and permutation equivariance. After the final encoder layer, the model applies RMSNorm and a linear projection $w\times h$8 to form

$w\times h$9

This automatically guarantees symmetry. To enforce the Laplacian zero-row-sum property, the diagonal is updated as

$\mathcal S\in\mathbb R^{n\times4}$00

The loss function is a normalized Laplacian loss. Let $\mathcal S\in\mathbb R^{n\times4}$01, $\mathcal S\in\mathbb R^{n\times4}$02, and $\mathcal S\in\mathbb R^{n\times4}$03. Training minimizes

$\mathcal S\in\mathbb R^{n\times4}$04

The stated reason is that capacitances span multiple orders of magnitude, so raw MSE would bias the fit toward large values; the normalized objective equalizes relative errors across scales and aligns with the graph-Laplacian structure of $\mathcal S\in\mathbb R^{n\times4}$05.

5. Data generation, optimization, and performance of the EDA model

The training data are synthetic cross-sections of $\mathcal S\in\mathbb R^{n\times4}$06–$\mathcal S\in\mathbb R^{n\times4}$07 randomly chosen metal layers. For each sample, conductor count is drawn as $\mathcal S\in\mathbb R^{n\times4}$08 with Poisson around $\mathcal S\in\mathbb R^{n\times4}$09; each conductor center has $\mathcal S\in\mathbb R^{n\times4}$10 and $\mathcal S\in\mathbb R^{n\times4}$11 uniform in the layer; width $\mathcal S\in\mathbb R^{n\times4}$12 is sampled $\mathcal S\in\mathbb R^{n\times4}$13 uniformly and $\mathcal S\in\mathbb R^{n\times4}$14 from an exponential-like distribution between design rules $\mathcal S\in\mathbb R^{n\times4}$15; and rejection sampling ensures design-rule compliance. The window width $\mathcal S\in\mathbb R^{n\times4}$16 is chosen per node so that long-range coupling beyond $\mathcal S\in\mathbb R^{n\times4}$17 is negligible $\mathcal S\in\mathbb R^{n\times4}$18 of total$\mathcal S\in\mathbb R^{n\times4}$19: for $\mathcal S\in\mathbb R^{n\times4}$20 nm, $\mathcal S\in\mathbb R^{n\times4}$21, and for $\mathcal S\in\mathbb R^{n\times4}$22 nm, $\mathcal S\in\mathbb R^{n\times4}$23. The dataset contains $\mathcal S\in\mathbb R^{n\times4}$24K samples per node for ASAP7 $\mathcal S\in\mathbb R^{n\times4}$25 nm, FreePDK15 $\mathcal S\in\mathbb R^{n\times4}$26 nm, Real28, and Real65, with a $\mathcal S\in\mathbb R^{n\times4}$27 train/validation split. For testing, approximately $\mathcal S\in\mathbb R^{n\times4}$28K cross-sections are extracted from three unseen real designs per node, retaining the 10 conductors nearest the window center. Field-solver ground truth takes approximately $\mathcal S\in\mathbb R^{n\times4}$29 hours on 24 CPU threads for the 50K samples (Huang et al., 6 Jun 2026).

The base model uses $\mathcal S\in\mathbb R^{n\times4}$30, head dimension $\mathcal S\in\mathbb R^{n\times4}$31, $\mathcal S\in\mathbb R^{n\times4}$32 heads, $\mathcal S\in\mathbb R^{n\times4}$33, and $\mathcal S\in\mathbb R^{n\times4}$34 layers for $\mathcal S\in\mathbb R^{n\times4}$35M parameters. AttentionCap-L uses $\mathcal S\in\mathbb R^{n\times4}$36, $\mathcal S\in\mathbb R^{n\times4}$37, $\mathcal S\in\mathbb R^{n\times4}$38, $\mathcal S\in\mathbb R^{n\times4}$39, and $\mathcal S\in\mathbb R^{n\times4}$40 for $\mathcal S\in\mathbb R^{n\times4}$41M parameters. A pure MLP ablation disables attention and SwiGLU and has approximately $\mathcal S\in\mathbb R^{n\times4}$42M parameters. Training uses AdamW with $\mathcal S\in\mathbb R^{n\times4}$43, $\mathcal S\in\mathbb R^{n\times4}$44, weight decay $\mathcal S\in\mathbb R^{n\times4}$45, linear warm-up from $\mathcal S\in\mathbb R^{n\times4}$46 to $\mathcal S\in\mathbb R^{n\times4}$47 over $\mathcal S\in\mathbb R^{n\times4}$48K steps, then linear decay to zero, for a total of $\mathcal S\in\mathbb R^{n\times4}$49K steps. Horizontal-flip augmentation $\mathcal S\in\mathbb R^{n\times4}$50 is applied. For the 50K dataset, batch size is $\mathcal S\in\mathbb R^{n\times4}$51; for the 200K mixed dataset, batch size is $\mathcal S\in\mathbb R^{n\times4}$52; inference uses batch size $\mathcal S\in\mathbb R^{n\times4}$53 uniformly.

On unseen real-design test sets in the multi-layer, single-node setting, the reported Real65 results are: CNN-Cap $\mathcal S\in\mathbb R^{n\times4}$54 and $\mathcal S\in\mathbb R^{n\times4}$55; pure MLP $\mathcal S\in\mathbb R^{n\times4}$56 and $\mathcal S\in\mathbb R^{n\times4}$57; and AttentionCap $\mathcal S\in\mathbb R^{n\times4}$58 and $\mathcal S\in\mathbb R^{n\times4}$59. On ASAP7 $\mathcal S\in\mathbb R^{n\times4}$60 nm, CNN-Cap records $\mathcal S\in\mathbb R^{n\times4}$61 and $\mathcal S\in\mathbb R^{n\times4}$62, pure MLP records $\mathcal S\in\mathbb R^{n\times4}$63 and $\mathcal S\in\mathbb R^{n\times4}$64, and AttentionCap records $\mathcal S\in\mathbb R^{n\times4}$65 and $\mathcal S\in\mathbb R^{n\times4}$66.

In the multi-node joint-learning setting, training on the union of $\mathcal S\in\mathbb R^{n\times4}$67 nm synthetic data, the base $\mathcal S\in\mathbb R^{n\times4}$68M-parameter model achieves on unseen $\mathcal S\in\mathbb R^{n\times4}$69 nm $\mathcal S\in\mathbb R^{n\times4}$70 and $\mathcal S\in\mathbb R^{n\times4}$71, and on $\mathcal S\in\mathbb R^{n\times4}$72 nm $\mathcal S\in\mathbb R^{n\times4}$73 and $\mathcal S\in\mathbb R^{n\times4}$74. AttentionCap-L further reduces these to $\mathcal S\in\mathbb R^{n\times4}$75 and $\mathcal S\in\mathbb R^{n\times4}$76, reported as a $\mathcal S\in\mathbb R^{n\times4}$77 improvement over CNN-Cap. On $\mathcal S\in\mathbb R^{n\times4}$78K $\mathcal S\in\mathbb R^{n\times4}$79 $\mathcal S\in\mathbb R^{n\times4}$80 $\mathcal S\in\mathbb R^{n\times4}$81 $\mathcal S\in\mathbb R^{n\times4}$82K capacitance elements, CNN-Cap test time is approximately $\mathcal S\in\mathbb R^{n\times4}$83 s, with $\mathcal S\in\mathbb R^{n\times4}$84 overhead to build density grids and total $\mathcal S\in\mathbb R^{n\times4}$85B FLOPs, while AttentionCap test time is approximately $\mathcal S\in\mathbb R^{n\times4}$86 s, or $\mathcal S\in\mathbb R^{n\times4}$87 s for the large variant, with $\mathcal S\in\mathbb R^{n\times4}$88–$\mathcal S\in\mathbb R^{n\times4}$89B FLOPs. This is approximately $\mathcal S\in\mathbb R^{n\times4}$90 faster than CNN-Cap and approximately $\mathcal S\in\mathbb R^{n\times4}$91 fewer FLOPs.

The transfer setting uses pretraining on $\mathcal S\in\mathbb R^{n\times4}$92 nm and fine-tuning for $\mathcal S\in\mathbb R^{n\times4}$93K steps on only $\mathcal S\in\mathbb R^{n\times4}$94K new $\mathcal S\in\mathbb R^{n\times4}$95 nm samples, corresponding to $\mathcal S\in\mathbb R^{n\times4}$96 of the full train data. The fine-tuned base AttentionCap immediately outperforms the best model trained from scratch even with $\mathcal S\in\mathbb R^{n\times4}$97 of the $\mathcal S\in\mathbb R^{n\times4}$98 nm data. A plausible implication is that the process-node embedding isolates stack-specific variation effectively enough for few-shot adaptation.

6. Comparison, recurrent themes, and common points of confusion

The principal source of confusion around AttentionCap is nominal rather than technical. In the 2021 literature, the name refers to a context-aware attentional pooling framework for fine-grained visual recognition; in the 2026 literature, it refers to a Transformer for capacitance-matrix extraction. The former operates on CNN feature maps and integral regions, whereas the latter operates on conductor sequences and learns an entire matrix through a Gram factorization and symmetric-attention output layer (Behera et al., 2021, Huang et al., 6 Jun 2026).

The shared motif is structured attention over sets of local entities. CAP builds pixel-level self-attention, then region-to-region attention over integral regions of varying scales, and finally an LSTM-plus-NetVLAD encoding that captures both informativeness and spatial structure. The EDA model applies full self-attention over conductors, then converts the resulting embeddings into a symmetric matrix through $\mathcal S\in\mathbb R^{n\times4}$99. This suggests a common methodological pattern: attention is used not merely as a generic feature mixer, but as a device for learning relations among discrete elements whose configuration carries the target signal.

Several misconceptions are explicitly contradicted by the cited works. CAP does not require the bounding-box and/or distinguishable part annotations. It is also not limited to a single backbone, having been evaluated with six state-of-the-art backbone networks. The EDA AttentionCap is not restricted to a fixed metal-layer triple or a single process node; it is designed for variable $\hat C\in\mathbb R^{n\times n}$00, any layer combination, and multi-node learning with a process-node embedding. Conversely, the two models should not be conflated as direct variants of one another. Their losses, invariances, and output constraints are domain-specific: cross-entropy classification in one case, and normalized Laplacian regression with physical constraints in the other.

From a broader methodological perspective, the two systems illustrate different uses of attention under strong inductive bias. CAP uses sub-pixel local gradients, integral-region coverage, and spatial-order encoding to resolve subtle inter-class variation in visual categories. Transformer-based AttentionCap uses permutation equivariance, Gram-factorized symmetry, and Laplacian structure to address efficiency and generality in capacitance extraction. The commonality lies in relation modeling; the divergence lies in the geometry and physics of the target space.