Hemlet: Chiplet Architecture for Vision Transformers

• Hemlet is a heterogeneous compute-in-memory chiplet architecture designed specifically for Vision Transformer acceleration, integrating analog, digital, and intermediate-data chiplets.
• It employs a novel group-level parallelism (GLP) and NoP-aware dataflow to boost ADC utilization and mitigate inter-chiplet communication bottlenecks.
• Evaluations on varied ViT models demonstrate high throughput (8.68 TOPS) and significant speedups, albeit with trade-offs in energy efficiency and NoP overhead.

Hemlet is a heterogeneous compute-in-memory chiplet architecture for Vision Transformers (ViTs) that combines analog CIM, digital CIM, and intermediate-data-processing chiplets within a 2.5D package. It was proposed to address a specific scaling problem in ViT acceleration: monolithic CIM designs can be energy-efficient, but they are constrained by single-die size, technology-node compromises, and communication bottlenecks once model size and on-chip weight capacity increase. Hemlet addresses these issues through inter-chiplet heterogeneity, a mapping strategy called group-level parallelism (GLP), and a NoP-aware execution model for static and dynamic matrix operations in transformer inference (Wang et al., 19 Nov 2025).

1. Architectural position and design rationale

Hemlet targets the hardware characteristics of ViTs rather than generic dense linear algebra. In a standard transformer block, most computation is concentrated in static linear layers such as the query, key, value, and output projections, as well as the two feed-forward layers, while the attention-score and attention-value products are dynamic because their effective weights are generated at runtime from activations. Hemlet assigns these two classes of operations to different CIM substrates.

The architecture is explicitly heterogeneous at the chiplet level. Analog CIM is used for static vector-matrix multiplications because pretrained weights can remain resident in high-density RRAM arrays. Digital CIM is used for dynamic products such as $\mathbf{Q}\mathbf{K}^\top$ and $\mathbf{P}\mathbf{V}$, where repeated rewriting of analog arrays would be impractical. Intermediate-data-processing chiplets provide SRAM-based buffering and SIMD support for operations such as LayerNorm, GELU, softmax, and residual addition. This partitioning is intended to preserve the efficiency advantages of CIM while avoiding the write-cost and endurance limitations of analog arrays for dynamic attention workloads (Wang et al., 19 Nov 2025).

A further design objective is node decoupling. Hemlet does not force analog and digital CIM macros into a single process technology. The reported implementation uses RRAM-based ACIM modeled at 22 nm and SRAM-based DCIM modeled at 7 nm. This is a central distinction from monolithic heterogeneous CIM accelerators, where mixed macro types can impose a single-node compromise.

2. Chiplet organization and functional decomposition

Hemlet consists of three chiplet classes with distinct roles.

Chiplet type	Primary role	Representative operations
ACIM chiplet	Static weight storage and static VMM	$\mathbf{X}\mathbf{W}_Q$, $\mathbf{X}\mathbf{W}_K$, $\mathbf{X}\mathbf{W}_V$, $\mathbf{S}\mathbf{W}_O$, $\mathbf{Z}\mathbf{W}_1$, $\mathbf{H}_1'\mathbf{W}_2$
DCIM chiplet	Dynamic VMM with runtime-generated weights	$\mathbf{Q}\mathbf{K}^\top$, $\mathbf{P}\mathbf{V}$
IDP chiplet	Global buffering and SIMD processing	LayerNorm, GELU, softmax, residual addition, intermediate aggregation

An ACIM chiplet uses a hierarchical organization of subarrays, processing engines, and tiles. In the default configuration, each ACIM chiplet contains 16 tiles, each tile contains 16 PEs, and each PE contains 6 analog CIM subarrays. A crucial microarchitectural feature is that each ADC is shared across several columns through a multiplexer; the set of columns sharing one ADC is defined as a “group.” Hemlet uses a group size of 8 columns.

A DCIM chiplet is flatter in structure. In the default configuration it contains 16 PEs, each PE with 4 DCIM subarrays. Its chiplet buffer is intentionally larger than that of ACIM so that $\mathbf{P}\mathbf{V}$0, $\mathbf{P}\mathbf{V}$1, and $\mathbf{P}\mathbf{V}$2 for assigned heads can be stored locally during attention execution.

IDP chiplets contain SRAM banks, a SIMD unit, control logic, and NoP interfaces. They act both as shared intermediate storage and as the execution substrate for non-CIM-friendly operations. All chiplets connect through a mesh network-on-package. Each chiplet has eight unidirectional links, and the default NoP bandwidth is 256 GB/s per chiplet, with evaluation also reported down to 64 GB/s (Wang et al., 19 Nov 2025).

3. Group-level parallelism

The central architectural novelty of Hemlet is GLP, which is a weight-mapping method designed around the ADC-sharing structure of ACIM arrays. Under conventional layer-wise mapping, all columns within a group belong to the same layer. Because a shared ADC can only digitize one column from that group at a time, much of the available column-level analog parallelism is not translated into effective throughput.

GLP changes the mapping granularity. Instead of assigning each group to a single layer, it interleaves columns from multiple compatible layers across the columns that share the ADC. Hemlet formalizes this with the “GLP_LayerSet,” a set of layers that have identical weight dimensions and are not concurrently active. For a GLP_LayerSet, the corresponding weight matrices are interleaved column-wise into an augmented matrix. The effect is that, for any active layer in the set, all groups can participate simultaneously, improving ADC utilization.

This required a ViT-specific adaptation because feed-forward layers are larger than attention projections. Hemlet therefore partitions the FFN weights into $\mathbf{P}\mathbf{V}$3 sub-layers: $\mathbf{P}\mathbf{V}$4 so that the static linear operators in attention and FFN can be grouped into a common shape class.

The GLP construction procedure is staged. It first packs FFN sub-layers across transformer blocks into GLP_LayerSets, then fills residual slots with MHA projection layers, then processes remaining MHA layers, and finally assigns any incomplete remainder to a baseline, non-GLP mapping. This is not merely a placement heuristic; it is an attempt to align ViT layer regularity with ACIM group structure.

A direct implication is that GLP increases ACIM-side parallelism but can also spread a single layer across more chiplets. This improves compute utilization while potentially increasing NoP traffic. Hemlet therefore couples GLP with communication-aware dataflow rather than treating mapping and scheduling as separate problems (Wang et al., 19 Nov 2025).

4. ViT execution model and system dataflow

Hemlet’s execution model separates static VMMs, dynamic attention products, and elementwise or reduction-style operations.

In the baseline flow, the input hidden states $\mathbf{P}\mathbf{V}$5 are sent from IDP to ACIM chiplets for the $\mathbf{P}\mathbf{V}$6 projections. The resulting tensors return to IDP, are forwarded to DCIM for $\mathbf{P}\mathbf{V}$7, returned again for softmax, then sent back to DCIM for $\mathbf{P}\mathbf{V}$8, and finally routed to ACIM for $\mathbf{P}\mathbf{V}$9 and the FFN layers. This flow is functionally straightforward but NoP-intensive.

Hemlet replaces this with a more tightly pipelined schedule. For static VMMs on ACIM, the sequence dimension is partitioned into blocks of size $\mathbf{X}\mathbf{W}_Q$0, and ACIM uses a virtual double buffer built from its chiplet buffer and local buffer so that communication for block $\mathbf{X}\mathbf{W}_Q$1 overlaps computation for block $\mathbf{X}\mathbf{W}_Q$2. This hides part of the NoP latency.

Attention execution is optimized more aggressively. After ACIM computes $\mathbf{X}\mathbf{W}_Q$3, $\mathbf{X}\mathbf{W}_Q$4, and $\mathbf{X}\mathbf{W}_Q$5, these tensors are sent directly to DCIM, bypassing IDP. DCIM then performs tiled attention, using a FlashAttention-style local softmax strategy. Instead of materializing the full

$\mathbf{X}\mathbf{W}_Q$6

matrix, Hemlet computes local softmax tiles,

$\mathbf{X}\mathbf{W}_Q$7

uses them immediately to form partial outputs

$\mathbf{X}\mathbf{W}_Q$8

and only later performs the global normalization needed to recover the final $\mathbf{X}\mathbf{W}_Q$9. This reduces both intermediate storage and NoP traffic, since the full $\mathbf{X}\mathbf{W}_K$0 attention score matrix never has to be preserved or routed through IDP (Wang et al., 19 Nov 2025).

5. Implementation characteristics and reported performance

Hemlet was evaluated on ViT-S/16, ViT-B/16, and ViT-L/16, all with $\mathbf{X}\mathbf{W}_K$1 patches and sequence length 197. The corresponding hidden dimensions are 384, 768, and 1024, with 6, 12, and 16 heads, and 12, 12, and 24 transformer blocks, respectively.

The reported implementation uses INT8 quantization via I-ViT. ACIM modeling assumes 2-bit RRAM cells with $\mathbf{X}\mathbf{W}_K$2 and 9-bit ADCs. Under this setup, Hemlet reports 80% top-1 accuracy on ImageNet-1K for ViT-B/16, with no degradation relative to the software baseline.

At the peak-system level, Hemlet reports 8.68 TOPS and 3.86 TOPS/W. In ablation studies against its own baseline mapping and dataflow, GLP plus dataflow optimization yields system-level speedups ranging from $\mathbf{X}\mathbf{W}_K$3 to $\mathbf{X}\mathbf{W}_K$4, depending on chiplet count and NoP bandwidth. GLP alone can provide substantial ACIM throughput gains, but in the smallest-chiplet, lowest-bandwidth configuration it can also expose NoP bottlenecks; the full Hemlet dataflow is what restores consistent gains across configurations.

The paper also reports a comparison against a DCIM-only design and a 3D heterogeneous ACIM/DCIM design. In that comparison, the other systems reported 0.828 TOPS and 1.61 TOPS throughput, while the 3D design reported 7.1 TOPS/W energy efficiency. Hemlet’s position in that comparison is therefore higher throughput but lower energy efficiency than the 3D-integrated alternative, which reflects the communication cost of a 2.5D chiplet NoP relative to monolithic or vertically integrated interconnect (Wang et al., 19 Nov 2025).

6. Trade-offs, scope, and nomenclature

Hemlet’s performance derives from a specific balance of computation placement, mapping, and communication scheduling. That balance introduces several explicit trade-offs.

GLP versus communication cost: GLP raises ACIM utilization by activating more groups in parallel, but it also tends to distribute a layer across more chiplets. This can increase NoP traffic. The architecture therefore depends on sufficiently capable NoP bandwidth and on the optimized ACIM–DCIM–IDP pipeline.

ADC precision versus energy: the reported 9-bit ADC configuration supports INT8 ViT accuracy and greater row parallelism, but it also contributes substantially to analog energy cost.

Chiplet scalability versus monolithic efficiency: the chiplet organization solves reticle and node-coupling problems, but it does not eliminate interconnect penalties. Hemlet’s reported 3.86 TOPS/W should be interpreted in that context.

Model specificity: GLP is strongly aligned with the regular block structure of ViTs. A plausible implication is that adaptation to less regular transformer families or to architectures with different linear-layer shapes would require a different layer-set construction procedure.

The term also has a nomenclature issue. “Hemlet” denotes the chiplet-based ViT accelerator described here, whereas “HEMlets” refers to “Part-Centric Heatmap Triplets” for 3D human pose and shape estimation, a distinct representation-learning method unrelated to compute-in-memory hardware (Zhou et al., 2020). This distinction is useful because the near-homography of the names can obscure that the two terms belong to different research areas.

Hemlet is therefore best understood as a specialized accelerator architecture rather than a generic CIM framework: a 2.5D heterogeneous chiplet system whose novelty lies in matching transformer workload structure to ADC-sharing groups, then compensating for chiplet-scale communication costs through explicit NoP-aware scheduling (Wang et al., 19 Nov 2025).