AngelSlim: Accelerating Large-Model Inference

• AngelSlim is an open-source framework that enhances large-model inference by integrating speculative decoding and ultra–low-bit compression techniques.
• Its speculative decoding track leverages methods like DFlash, DFlare, and SpecExit to reduce latency and optimize reasoning through parallel block prediction and early exit strategies.
• The compression pathway uses Sherry’s 1.25-bit ternary quantization to achieve hardware-efficient on-device deployment, balancing speedup with reduced memory footprint.

Searching arXiv for AngelSlim-related papers to ground the article in current published work. AngelSlim is Tencent’s open-source toolkit and codebase for accelerating large-model inference through two distinct but related technical tracks: speculative decoding for LLMs and large reasoning models, and ultra–low-bit model compression for on-device deployment. Across the literature, the name denotes a practical repository that hosts methods including DFlash and DFlare for block diffusion speculative decoding, SpecExit for reasoning-aware early exit within speculative decoding, and Sherry-style 1.25-bit ternary quantization used in multilingual translation systems such as Hy-MT2 (Zhang et al., 1 Jun 2026, Yang et al., 29 Sep 2025, Huang et al., 12 Jan 2026, Zheng et al., 21 May 2026). The unifying theme is end-to-end inference acceleration under real deployment constraints, with emphasis on wall-clock speedup, hardware efficiency, and integration into serving stacks rather than isolated algorithmic gains.

1. Scope and positioning

AngelSlim is described in the literature as an open, practical toolkit from Tencent and Peking University for accelerating LLM inference via model compression and speculative decoding (Zhang et al., 1 Jun 2026). In the speculative-decoding line, AngelSlim includes DFlash and DFlare as block-diffusion speculative decoding components that replace autoregressive draft models with parallel block denoising drafters. In the reasoning-model line, AngelSlim hosts SpecExit, an inference-time framework for accelerating Large Reasoning Models by combining speculative decoding with a learned early-exit mechanism derived from hidden states (Yang et al., 29 Sep 2025). In the compression line, AngelSlim provides hardware-efficient ternary quantization and, in Hy-MT2, a 1.25-bit extreme quantization pathway for on-device multilingual translation (Huang et al., 12 Jan 2026, Zheng et al., 21 May 2026).

This distribution of methods indicates that AngelSlim is not a single algorithm but a systems-oriented framework spanning multiple inference bottlenecks. One bottleneck is autoregressive latency in general-purpose LLM serving; another is overthinking in reasoning-centric models; a third is memory bandwidth and storage on mobile or edge hardware. A plausible implication is that AngelSlim should be understood as a deployment toolkit organized around inference efficiency primitives rather than as a model family.

2. Speculative decoding in AngelSlim: from DFlash to DFlare

Within AngelSlim, DFlash and DFlare implement block diffusion speculative decoding. In this setting, the draft model predicts all tokens in a block of size $B$ at once, specifically positions $t+1 \ldots t+B-1$, using a discrete diffusion process and bidirectional attention within the block; the target model then verifies the entire block in a single parallel forward pass and accepts the longest correct prefix (Zhang et al., 1 Jun 2026). This preserves the target distribution while reducing wall-clock time by amortizing verification over multiple positions per step.

The AngelSlim literature positions block diffusion against two other speculative-decoding families. Autoregressive draft-then-verify methods such as EAGLE and EAGLE-3 generate draft tokens sequentially and thus scale latency with depth, even when feature conditioning is added. Medusa-style multi-head methods predict several next tokens in parallel but do not exploit block diffusion’s ability to predict entire blocks in one pass. Diffusion drafters including SpecDiff-2, DiffuSpec, Fast-dLLM, DFlash, and DFlare instead use discrete diffusion to predict all tokens in a block simultaneously, enabling parallel verification by the target model (Zhang et al., 1 Jun 2026).

DFlash is the prior AngelSlim baseline. It introduced KV injection of fused target hidden states and achieved state-of-the-art acceleration, but all draft layers share a single fused representation derived from only a few target layers (Zhang et al., 1 Jun 2026). Concretely, DFlash defines, at each position $t$,

$$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$

with $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$, $b_{fc} \in \mathbb{R}^{d}$, and $c_t \in \mathbb{R}^{d}$; the same $c_t$ is injected into all draft layers (Zhang et al., 1 Jun 2026). The stated limitation is that such shared conditioning constrains per-layer specialization and saturates depth scaling.

DFlare extends DFlash by “flaring out” this conditioning bottleneck through adaptive layer-wise fusion and heterogeneous KV projections (Zhang et al., 1 Jun 2026). It introduces a learnable fusion matrix $W^{fuse} \in \mathbb{R}^{D \times T}$ for a draft with $D$ layers. For draft layer $t+1 \ldots t+B-1$0,

$t+1 \ldots t+B-1$1

$t+1 \ldots t+B-1$2

The fused features across $t+1 \ldots t+B-1$3 positions form $t+1 \ldots t+B-1$4 (Zhang et al., 1 Jun 2026). After training, $t+1 \ldots t+B-1$5 can be cached, so inference-time fusion reduces to a scalar-weighted sum per layer with negligible overhead.

DFlare also decouples KV projections for injected target context and draft hidden states. For draft layer $t+1 \ldots t+B-1$6,

$t+1 \ldots t+B-1$7

$t+1 \ldots t+B-1$8

$t+1 \ldots t+B-1$9

where $t$0 are projections for target context and $t$1 are projections for draft block positions (Zhang et al., 1 Jun 2026). The paper’s interpretation is that this avoids representational interference and grants independent subspaces for draft and target information.

A further training modification is the progressive position-weighted loss,

$t$2

$t$3

which upweights early positions in the block because early-position errors invalidate the rest of the block (Zhang et al., 1 Jun 2026). In practice, $t$4 and $t$5 is incremented by $t$6 each epoch, except for GPT-OSS-20B where $t$7 and $t$8 is fixed at $t$9 (Zhang et al., 1 Jun 2026).

3. DFlare pipeline, scaling behavior, and empirical performance

The DFlare inference loop has a drafting phase and a verification phase. Inputs are the target model’s current decode token at position $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$0, denoted $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$1, the hidden states $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$2 for selected target layers across $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$3 context tokens, and the block size $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$4 (Zhang et al., 1 Jun 2026). In drafting, each draft layer computes layer-wise fused features $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$5, builds $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$6, $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$7, and $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$8 with heterogeneous projections, applies bidirectional attention within the block and attention to the full target context, and propagates only block positions to the next draft layer. After the final draft layer, the LM head outputs logits for masked positions $$c_t = W_{fc}[h_t^{(1)}; \ldots; h_t^{(T)}] + b_{fc},$$9, and $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$0 candidate tokens are sampled in parallel (Zhang et al., 1 Jun 2026). In verification, the target model performs one forward pass on the $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$1-token block, accepts the longest prefix matching its own predictions, denotes the accepted length as $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$2, generates one extra token after the accepted prefix, updates the KV cache, and repeats (Zhang et al., 1 Jun 2026).

Architecturally, DFlare uses $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$3 Transformer draft layers for Qwen3 targets with block size $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$4; for GPT-OSS-20B, the block size is $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$5 (Zhang et al., 1 Jun 2026). A broad set of target layers means $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$6 uniformly selected layers between the 2nd and the 3rd-to-last layer of the target model, while for GPT-OSS, $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$7 is adapted as $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$8, $W_{fc} \in \mathbb{R}^{d \times (T \cdot d)}$9, or $b_{fc} \in \mathbb{R}^{d}$0 depending on model configuration (Zhang et al., 1 Jun 2026). The draft reuses the target embedding and LM head, which remain frozen; only the draft Transformer layers are trained (Zhang et al., 1 Jun 2026).

The reported overhead is minor. Fusion introduces only $b_{fc} \in \mathbb{R}^{d}$1 scalar parameters; for $b_{fc} \in \mathbb{R}^{d}$2 and $b_{fc} \in \mathbb{R}^{d}$3, this is $b_{fc} \in \mathbb{R}^{d}$4 parameters. Heterogeneous KV adds two extra projection matrices per draft layer, while attention complexity remains the same order as in DFlash because the set of KV sources is unchanged (Zhang et al., 1 Jun 2026).

Training was scaled from $b_{fc} \in \mathbb{R}^{d}$5 to $b_{fc} \in \mathbb{R}^{d}$6 samples, with a mixture including NVIDIA Nemotron Post-Training Dataset V2, CodeAlpaca, and Step-3.5-Flash-SFT, covering mathematical reasoning, code generation, and conversational data (Zhang et al., 1 Jun 2026). Preprocessing keeps original prompts and regenerates responses using the corresponding target model with temperature $b_{fc} \in \mathbb{R}^{d}$7; the draft is trained to predict only these regenerated responses to match the target’s output distribution. To prevent test leakage, training samples with at least $b_{fc} \in \mathbb{R}^{d}$8-gram overlap with test sets are removed (Zhang et al., 1 Jun 2026). For each sequence, $b_{fc} \in \mathbb{R}^{d}$9 anchors are randomly sampled; blocks are concatenated and trained jointly using a sparse mask that allows bidirectional attention within each block and attention to its context, but disallows cross-block attention (Zhang et al., 1 Jun 2026).

The main quantitative results are summarized below.

Target	DFlash average	DFlare average
Qwen3-4B	4.99× speedup, $c_t \in \mathbb{R}^{d}$0	5.52× speedup, $c_t \in \mathbb{R}^{d}$1
Qwen3-8B	5.05× speedup, $c_t \in \mathbb{R}^{d}$2	5.46× speedup, $c_t \in \mathbb{R}^{d}$3
GPT-OSS-20B	3.71× speedup, $c_t \in \mathbb{R}^{d}$4	3.91× speedup, $c_t \in \mathbb{R}^{d}$5

For Qwen3-4B, DFlare improves over DFlash by $c_t \in \mathbb{R}^{d}$6 in speedup and $c_t \in \mathbb{R}^{d}$7 in $c_t \in \mathbb{R}^{d}$8; for Qwen3-8B, by $c_t \in \mathbb{R}^{d}$9 and $c_t$0; for GPT-OSS-20B, by $c_t$1 and $c_t$2 (Zhang et al., 1 Jun 2026). On Qwen3-8B at temperature $c_t$3, per-task results include GSM8K at $c_t$4 with $c_t$5, MATH500 at $c_t$6 with $c_t$7, AIME25 at $c_t$8 with $c_t$9, HumanEval at $W^{fuse} \in \mathbb{R}^{D \times T}$0 with $W^{fuse} \in \mathbb{R}^{D \times T}$1, MBPP at $W^{fuse} \in \mathbb{R}^{D \times T}$2 with $W^{fuse} \in \mathbb{R}^{D \times T}$3, and MT-Bench at $W^{fuse} \in \mathbb{R}^{D \times T}$4 with $W^{fuse} \in \mathbb{R}^{D \times T}$5 (Zhang et al., 1 Jun 2026). Stochastic decoding at temperature $W^{fuse} \in \mathbb{R}^{D \times T}$6 also improves average $W^{fuse} \in \mathbb{R}^{D \times T}$7 over DFlash by $W^{fuse} \in \mathbb{R}^{D \times T}$8 on Qwen3-4B, $W^{fuse} \in \mathbb{R}^{D \times T}$9 on Qwen3-8B, and $D$0 on GPT-OSS-20B (Zhang et al., 1 Jun 2026).

Throughput measurements on SGLang with an H20 GPU and Qwen3-8B show the systems orientation of AngelSlim’s speculative-decoding work. On GSM8K at concurrency $D$1, baseline throughput is $D$2 tok/s, DFlash reaches $D$3, and DFlare reaches $D$4. On HumanEval at concurrency $D$5, the corresponding values are $D$6, $D$7, and $D$8 tok/s (Zhang et al., 1 Jun 2026). The paper states that DFlare’s throughput advantage grows with concurrency, suggesting favorable scaling under load.

Ablations support the design choices. DFlare scales well from $D$9 layers, whereas DFlash exhibits diminishing returns beyond $t+1 \ldots t+B-1$00–$t+1 \ldots t+B-1$01 layers (Zhang et al., 1 Jun 2026). Leveraging $t+1 \ldots t+B-1$02 target layers yields monotonically increasing $t+1 \ldots t+B-1$03 and speedup in DFlare, while DFlash gains little at $t+1 \ldots t+B-1$04 (Zhang et al., 1 Jun 2026). Removing softmax normalization on fusion weights degrades performance the most, and removing heterogeneous KV projections or progressive loss also reduces speedup and $t+1 \ldots t+B-1$05 (Zhang et al., 1 Jun 2026). Data scaling from $t+1 \ldots t+B-1$06 samples increases average speedup from $t+1 \ldots t+B-1$07 to $t+1 \ldots t+B-1$08 to $t+1 \ldots t+B-1$09, and $t+1 \ldots t+B-1$10 from $t+1 \ldots t+B-1$11 to $t+1 \ldots t+B-1$12 to $t+1 \ldots t+B-1$13 on Qwen3-8B (Zhang et al., 1 Jun 2026).

4. SpecExit and reasoning-aware early exit

AngelSlim’s second speculative-decoding line targets Large Reasoning Models rather than general LLM decoding. SpecExit addresses overthinking, defined as the tendency of LRMs to generate unnecessarily long, verbose chains of thought before producing the final answer, increasing token usage and end-to-end latency without accuracy gains (Yang et al., 29 Sep 2025). Earlier early-exit methods relied on probing mechanisms such as inserted prompts and re-forwarding, which introduced detection overhead and reduced wall-clock gains. SpecExit instead predicts both future tokens and an early-exit signal directly from hidden states without probing overhead (Yang et al., 29 Sep 2025).

The core architecture extends the Multi-Token Prediction layer so that a hidden state $t+1 \ldots t+B-1$14 yields token logits and three auxiliary signals—confidence, reasoning progress, and remaining reasoning length—while keeping the backbone model unchanged:

$t+1 \ldots t+B-1$15

Here $t+1 \ldots t+B-1$16 are token logits and the remaining terms are auxiliary heads (Yang et al., 29 Sep 2025).

Training uses a multi-task objective,

$t+1 \ldots t+B-1$17

with regression losses

$t+1 \ldots t+B-1$18

$t+1 \ldots t+B-1$19

$t+1 \ldots t+B-1$20

To stabilize joint optimization, SpecExit uses gradient-based dynamic weighting,

$t+1 \ldots t+B-1$21

and

$t+1 \ldots t+B-1$22

where $t+1 \ldots t+B-1$23 (Yang et al., 29 Sep 2025).

The supervision target is the minimal “sufficient” CoT. The method extracts the reasoning between > and </think>, tries inserting </think> after each paragraph, and retains the minimal CoT for which the final answer remains unchanged (Yang et al., 29 Sep 2025). Confidence is annotated as the geometric mean of token probabilities, progress as a normalized $t+1 \ldots t+B-1$24 trajectory of CoT completion, and remaining reasoning length as the number of tokens from <think> to the earliest valid </think> insertion point that still yields the original answer (Yang et al., 29 Sep 2025).

At inference, SpecExit runs inside a standard speculative-decoding loop with a lightweight draft model $t+1 \ldots t+B-1$25 and a target LRM $t+1 \ldots t+B-1$26 (Yang et al., 29 Sep 2025). The paper describes two equivalent, no-overhead options for computing signals: extending $t+1 \ldots t+B-1$27’s MTP head to output both token candidates and signals, or reusing $t+1 \ldots t+B-1$28’s last-accepted-token hidden state and passing it through a small linear head during verification (Yang et al., 29 Sep 2025). Early exit is constrained to semantically natural boundaries via step-split tokens such as paragraph delimiters ".\n\n" or "\n\n", which are used by default because of their generality (Yang et al., 29 Sep 2025).

Because raw signals are volatile, SpecExit applies Exponentially Weighted Moving Average smoothing with factor $t+1 \ldots t+B-1$29, defaulting to $t+1 \ldots t+B-1$30:

$t+1 \ldots t+B-1$31

The default multi-signal thresholds are confidence $t+1 \ldots t+B-1$32, progress $t+1 \ldots t+B-1$33, and remaining length $t+1 \ldots t+B-1$34 (Yang et al., 29 Sep 2025). If the model is still in the thinking phase, a step-split token has been accepted, and the smoothed signals exceed thresholds, SpecExit truncates accepted content at that boundary, substitutes the recover token with ``, and ends the reasoning phase (Yang et al., 29 Sep 2025).

The method is evaluated on Qwen3-4B-Thinking-2507 and DeepSeek-R1-Distill-Llama-8B, across GSM8K, MATH500, AIME, HumanEval+, GPQA-D, and ARC-Challenge, with baselines including Think, NoThink, DEER, and EAGLE3 (Yang et al., 29 Sep 2025). The headline results are an average generation-length reduction of up to $t+1 \ldots t+B-1$35 and an end-to-end latency speedup of up to $t+1 \ldots t+B-1$36 versus the speculative decoding baseline EAGLE3, with comparable accuracy (Yang et al., 29 Sep 2025).

Representative numbers illustrate the trade-off. On Qwen3-4B-Thinking-2507, GSM8K changes from $t+1 \ldots t+B-1$37 to $t+1 \ldots t+B-1$38 tokens, latency from $t+1 \ldots t+B-1$39 to $t+1 \ldots t+B-1$40, and accuracy from $t+1 \ldots t+B-1$41 to $t+1 \ldots t+B-1$42 under SpecExit versus EAGLE3; ARC-Challenge changes from $t+1 \ldots t+B-1$43 to $t+1 \ldots t+B-1$44 tokens, latency from $t+1 \ldots t+B-1$45 to $t+1 \ldots t+B-1$46, and accuracy from $t+1 \ldots t+B-1$47 to $t+1 \ldots t+B-1$48; GPQA-D changes from $t+1 \ldots t+B-1$49 to $t+1 \ldots t+B-1$50 tokens, latency from $t+1 \ldots t+B-1$51 to $t+1 \ldots t+B-1$52, and accuracy from $t+1 \ldots t+B-1$53 to $t+1 \ldots t+B-1$54 (Yang et al., 29 Sep 2025). On DeepSeek-R1-Distill-Llama-8B, GSM8K changes from $t+1 \ldots t+B-1$55 to $t+1 \ldots t+B-1$56 tokens, latency from $t+1 \ldots t+B-1$57 to $t+1 \ldots t+B-1$58, and accuracy from $t+1 \ldots t+B-1$59 to $t+1 \ldots t+B-1$60; HumanEval+ changes from $t+1 \ldots t+B-1$61 to $t+1 \ldots t+B-1$62 tokens, latency from $t+1 \ldots t+B-1$63 to $t+1 \ldots t+B-1$64, and accuracy from $t+1 \ldots t+B-1$65 to $t+1 \ldots t+B-1$66 (Yang et al., 29 Sep 2025).

Ablations show that confidence-only signals produce the largest token reduction but overestimate certainty, progress-only signals rise early but fluctuate during reflection, and remaining-only signals often trigger premature exits on hard problems. Combined signals offer the best balance (Yang et al., 29 Sep 2025). Among smoothing methods, EWMA gives the best overall trade-off and is therefore adopted by default (Yang et al., 29 Sep 2025). This suggests that AngelSlim’s reasoning-acceleration work treats hidden-state readout as a control signal for dynamic termination rather than merely as an auxiliary confidence estimator.

5. Quantization and on-device deployment: Sherry and Hy-MT2

A separate AngelSlim line concerns model compression for edge deployment. In "Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification," AngelSlim is presented as Tencent’s open-source project for hardware-efficient ternary LLMs (Huang et al., 12 Jan 2026). Its core contribution is Sherry, which introduces a 3:4 fine-grained sparsity pattern so that every contiguous block of $t+1 \ldots t+B-1$67 weights contains exactly one zero and three non-zeros in $t+1 \ldots t+B-1$68 (Huang et al., 12 Jan 2026).

The bit arithmetic is central. An unconstrained 4-weight ternary block has $t+1 \ldots t+B-1$69 states, but 3:4 sparsity reduces this to choosing one zero position in $t+1 \ldots t+B-1$70 ways and one sign assignment for the remaining three positions in $t+1 \ldots t+B-1$71 ways, for a total of $t+1 \ldots t+B-1$72 states (Huang et al., 12 Jan 2026). These $t+1 \ldots t+B-1$73 states fit exactly in $t+1 \ldots t+B-1$74 bits, so one 4-weight block is encoded in $t+1 \ldots t+B-1$75 bits, or $t+1 \ldots t+B-1$76 bits/weight (Huang et al., 12 Jan 2026). Relative to 2-bit aligned packing, this is $t+1 \ldots t+B-1$77 fewer bits; relative to 1.67-bit irregular packing, it is about $t+1 \ldots t+B-1$78 fewer bits (Huang et al., 12 Jan 2026).

Sherry’s packing uses a 4-bit canonical index and a 1-bit global sign. The 4-bit index encodes zero position and canonical sign pattern; the sign bit flips all three nonzero signs if set (Huang et al., 12 Jan 2026). This is explicitly designed to be SIMD- and LUT-friendly, avoiding the 3-way shuffle penalty of prior 1.67-bit ternary schemes (Huang et al., 12 Jan 2026).

The training-time quantization form is

$t+1 \ldots t+B-1$79

where $t+1 \ldots t+B-1$80 and $t+1 \ldots t+B-1$81 is per-channel or per-group (Huang et al., 12 Jan 2026). Under the 3:4 block constraint, the smallest-magnitude weight in each block is pruned and the remaining three are mapped to their signs (Huang et al., 12 Jan 2026). Sherry identifies a “weight trapping” issue in sparse ternary training, which produces representational collapse and low effective rank. To mitigate this, it introduces Arenas, an annealing residual synapse:

$t+1 \ldots t+B-1$82

with

$t+1 \ldots t+B-1$83

The gate $t+1 \ldots t+B-1$84 starts near $t+1 \ldots t+B-1$85 and decays to $t+1 \ldots t+B-1$86 by the end of quantization-aware training; schedules include linear, cosine, exponential, and warmup variants (Huang et al., 12 Jan 2026). The residual path is removed at inference.

Empirically, on LLaMA-3.2-1B and 3B across PIQA, ARC-Easy, ARC-Challenge, HellaSwag, and WinoGrande, Sherry matches the strongest 1.67-bit ternary baseline on average at 1B while using fewer bits, and achieves the second-best average at 3B (Huang et al., 12 Jan 2026). For the 1B model, BF16 has average accuracy $t+1 \ldots t+B-1$87, Tequila at $t+1 \ldots t+B-1$88b has $t+1 \ldots t+B-1$89, and Sherry at $t+1 \ldots t+B-1$90b also has $t+1 \ldots t+B-1$91 (Huang et al., 12 Jan 2026). On an Intel i7-14700HX CPU with BitNet.cpp-style kernels, throughput for the 0.7B model is $t+1 \ldots t+B-1$92 t/s for Sherry versus $t+1 \ldots t+B-1$93 t/s for the 1.67b TL2 baseline, and for the 3B model $t+1 \ldots t+B-1$94 t/s versus $t+1 \ldots t+B-1$95 t/s, with correspondingly lower model sizes (Huang et al., 12 Jan 2026).

Hy-MT2 applies AngelSlim’s compression techniques to multilingual translation (Zheng et al., 21 May 2026). In that work, AngelSlim is Hunyuan’s in-house compression toolkit, with two ultra-low-precision pathways: a 2-bit quantization-aware training path using Stretched Elastic Quantization and an extreme 1.25-bit path implemented by Sherry (Zheng et al., 21 May 2026). The headline deployment result is that the 1.25-bit AngelSlim variant of Hy-MT2-1.8B requires about $t+1 \ldots t+B-1$96 MB of storage and yields about $t+1 \ldots t+B-1$97 speedup over a 4-bit Hy-MT1.5 baseline on Apple A15 (Zheng et al., 21 May 2026).

The 1.25-bit scheme in Hy-MT2 uses the ternary alphabet $t+1 \ldots t+B-1$98 with a 3:4 fine-grained structure: every block of 4 consecutive weights contains exactly one zero and three sign-only weights (Zheng et al., 21 May 2026). For each 4-weight block, it stores 2 bits for the zero position and 3 bits for the three signs, again totaling 5 bits per 4 weights or 1.25 bits/weight (Zheng et al., 21 May 2026). The training objective for both 1.25-bit and 2-bit models is a distillation-based QAT objective,

$t+1 \ldots t+B-1$99

with $t$00 and $t$01 adjusted per token using teacher confidence (Zheng et al., 21 May 2026).

The paper does not tabulate benchmark quality numbers for the 1.25-bit Hy-MT2 variant, but does provide 2-bit results to indicate the compression-quality trade-off. For Hy-MT2-1.8B, 2-bit quantization lowers FLORES-200 XX↔XX from $t$02 to $t$03 and IFMTBench Total from $t$04 to $t$05, with smaller drops on DMTB and WMTB (Zheng et al., 21 May 2026). The paper therefore treats QAT with bidirectional, confidence-weighted KL distillation as the mechanism that preserves translation quality as much as possible under extreme quantization (Zheng et al., 21 May 2026).

6. Training regimes, implementation surfaces, and reproducibility

AngelSlim’s methods are presented as deployable systems rather than only conceptual architectures, and the papers report concrete implementation environments. For DFlare, the repository is https://github.com/Tencent/AngelSlim, with dependencies including torch v2.9.1, transformers v4.57.1, datasets v4.8.4, sglang v0.5.6, and numpy v2.4.3 (Zhang et al., 1 Jun 2026). Training uses 32 GPUs with distributed data parallelism, and approximate wall-clock per target is reported as about 160 GPU-hours per device for Qwen3-8B, about 100 hours for Qwen3-4B, and about 90 hours for GPT-OSS-20B (Zhang et al., 1 Jun 2026). Inference evaluation runs on a single GPU per target, and a full benchmark pass is about 4 hours for Qwen3-8B (Zhang et al., 1 Jun 2026).

DFlare’s training scripts follow a specific configuration: $t$06 target layers selected uniformly between the 2nd and 3rd-to-last target layer, $t$07 for Qwen or $t$08 for GPT-OSS, $t$09 draft layers, AdamW with learning rate $t$10, cosine schedule with warmup ratio $t$11, maximum sequence length $t$12, $t$13 anchors per sequence, and the stated $t$14 schedule (Zhang et al., 1 Jun 2026). The draft model reuses the target embedding and LM head, which are frozen; only draft layers are trained (Zhang et al., 1 Jun 2026).

SpecExit is implemented in both PyTorch and vLLM, with the vLLM integration using tree attention and KV-cache updates for speculative verification and early truncation at step-split tokens (Yang et al., 29 Sep 2025). The reported runtime environment is an 8×H20 GPU cluster (Yang et al., 29 Sep 2025). The code targets compatibility with Qwen3-4B-Thinking-2507, DeepSeek-R1-Distill-Llama-8B, and standard speculative-decoding baselines such as EAGLE3-like MTP heads (Yang et al., 29 Sep 2025).

For Sherry, the paper assumes quantization-aware training on multi-GPU hardware, stating that 32 NVIDIA GPUs were used with approximately 10B UltraFineWeb tokens (Huang et al., 12 Jan 2026). The implementation includes weight-only ternary quantization with per-tensor, per-channel, or per-group scales, group size 128 by default, and a LUT-based multiplication-free inference path integrated with BitNet.cpp-style kernels (Huang et al., 12 Jan 2026). Activations and KV-cache remain BF16 in this release (Huang et al., 12 Jan 2026).

Hy-MT2’s 1.25-bit deployment emphasizes mobile or edge CPU execution, specifically Apple A15 (Zheng et al., 21 May 2026). The runtime uses Sherry’s 5-bit packs per 4 weights and custom kernels for linear layers (Zheng et al., 21 May 2026). The paper does not specify activation or KV-cache quantization for the 1.25-bit deployment, nor explicit kernel details, but attributes speedup to reduced memory bandwidth and SIMD-friendly regularity (Zheng et al., 21 May 2026).

7. Limitations, edge cases, and research directions

The AngelSlim papers describe several limitations and underperformance regimes. For DFlare, deeper draft models and larger corpora increase training compute substantially; the authors state that although these costs are amortized by inference savings, the up-front cost is high (Zhang et al., 1 Jun 2026). Acceptance length continues to improve with more data, but exploration beyond 2.4M samples was not feasible due to resource constraints (Zhang et al., 1 Jun 2026). Larger targets such as GPT-OSS-20B show a larger representational gap and lower absolute acceptance lengths, and open-ended conversation tasks such as MT-Bench have lower $t$15 across all methods because of diversity (Zhang et al., 1 Jun 2026). Gains may also diminish if too few or poorly chosen target layers are fused (Zhang et al., 1 Jun 2026).

For SpecExit, premature exit remains a risk if one relies on a single signal; combined signals and EWMA smoothing mitigate but do not eliminate the issue on the most complex problems (Yang et al., 29 Sep 2025). Discourse-marker-based step splits can be model- and data-dependent, so paragraph delimiters are used by default because they are more general (Yang et al., 29 Sep 2025). Threshold selection directly controls the speed-accuracy trade-off and varies across task and model family (Yang et al., 29 Sep 2025).

For Sherry, the evaluation is limited to models up to 3B parameters and is tuned for edge CPUs; its behavior on much larger server-class models remains to be validated (Huang et al., 12 Jan 2026). The method is edge-centric and does not exploit vendor-specific sparse tensor cores on GPUs (Huang et al., 12 Jan 2026). Because activations and KV-cache remain BF16, it is not an end-to-end low-precision inference stack (Huang et al., 12 Jan 2026). The rigid one-zero-per-4 block can conflict with layers whose optimal quantization needs different sparsity or magnitude patterns, making QAT with Arenas essential (Huang et al., 12 Jan 2026, Zheng et al., 21 May 2026).

For Hy-MT2, the paper does not publish per-benchmark quality numbers for the 1.25-bit variant, even though it reports storage and speed (Zheng et al., 21 May 2026). It explicitly notes that instruction-following and multi-constraint tasks are more sensitive under extreme quantization, as indicated by 2-bit results (Zheng et al., 21 May 2026). This suggests that deployment decisions in AngelSlim’s compression pipeline remain application-specific, especially when task fidelity is more important than raw footprint.

Future directions named in the speculative-decoding work include scaling training data further, exploring even deeper drafts under DFlare’s fusion, adaptive selection of target layers per draft layer, dynamic block sizing, cross-architecture generalization, and finer latency-memory trade-offs (Zhang et al., 1 Jun 2026). Future directions in SpecExit include broader use of hidden-state signals for efficient reasoning, such as richer self-verification, planning, and adaptive policies (Yang et al., 29 Sep 2025). For Sherry, proposed extensions include mixed precision, variable $t$16 sparsity, adaptive per-layer sparsity, and co-design with ASIC or FPGA LUT capacities (Huang et al., 12 Jan 2026).

Taken together, the literature presents AngelSlim as a heterogeneous acceleration framework with a common design philosophy: exploit structure already present in model internals—target hidden states, verification passes, sparsity topologies, or low-entropy weight alphabets—to reduce end-to-end inference cost without altering the target model’s intended predictive behavior more than necessary (Zhang et al., 1 Jun 2026, Yang et al., 29 Sep 2025, Huang et al., 12 Jan 2026, Zheng et al., 21 May 2026).