EdgeInfinite-Instruct Framework

• EdgeInfinite-Instruct is a long-context language model framework for edge devices that integrates supervised instruction tuning, compressed-memory execution, and quantization.
• It employs Segmented Supervised Fine-Tuning to align task-specific instruction-following with blockwise memory processing, addressing self-attention and cache growth challenges.
• The system uses fixed-shape computation graphs and fine-grained post-training quantization to achieve efficient mobile NPU deployment with reduced memory and latency.

Searching arXiv for the cited paper and closely related work. EdgeInfinite-Instruct is a long-context language-model deployment framework for resource-constrained edge devices that combines supervised instruction tuning with compressed-memory Transformer execution and NPU-oriented runtime optimization. It is presented as an extension of EdgeInfinite, retaining a standard Transformer backbone while adding a training and deployment recipe designed for long-sequence summarization and question answering on smartphones and similar platforms. The central problem is the mismatch between the quadratic cost of self-attention, the linear growth of the Key-Value cache during autoregressive decoding, and the static-shape, quantization-friendly execution model preferred by mobile NPUs. EdgeInfinite-Instruct addresses that mismatch through Segmented Supervised Fine-Tuning, fine-grained post-training quantization, and fixed-shape computation graphs, with the stated goal of improving instruction-following quality without sacrificing on-device efficiency (Chen et al., 1 Aug 2025).

1. Problem formulation and design objective

EdgeInfinite-Instruct is situated in the setting of long-sequence language understanding and generation on resource-constrained edge devices, especially smartphones with NPUs. The stated bottlenecks are fourfold: Transformer self-attention is quadratic in sequence length; KV cache grows linearly with context length, increasing memory pressure and bandwidth cost during autoregressive generation; edge deployment requires static or fixed shapes, careful cache sizing, and hardware-friendly quantization; and instruction-following quality is necessary for real applications such as summarization and question answering, whereas purely continual-pretrained long-context models often lag in task adaptation (Chen et al., 1 Aug 2025).

The method is explicitly positioned against two existing directions. KV-cache pruning methods such as SnapKV, PyramidKV, and StreamingLLM reduce memory by dropping tokens, but may reduce fidelity and do not reliably improve time to first token, or TTFT. Alternative sequence-model architectures such as SSMs, Mamba, and RWKV improve asymptotic sequence efficiency, but require architectural replacement and full retraining, which reduces compatibility with standard Transformer ecosystems and existing deployment stacks. EdgeInfinite is described as an intermediate approach: it preserves Transformer compatibility and introduces a trainable memory-gating module with compressed memory, but remains limited in instruction following and lacks mobile-specific optimization.

Within that framing, EdgeInfinite-Instruct combines three elements that are often treated separately: supervised fine-tuning for task quality, compressed-memory long-context modeling for efficiency, and PTQ plus fixed-shape graph execution for edge NPU deployment. A plausible implication is that the work should be understood less as a new language-model family than as a systems-level integration of training-time and deployment-time constraints.

2. Architectural basis in EdgeInfinite

The architectural substrate is EdgeInfinite, a memory-efficient Transformer variant that keeps the standard Transformer backbone but augments it with a trainable memory-gating mechanism and compressed memory so that arbitrarily long sequences can be processed in blocks (Chen et al., 1 Aug 2025).

The input sequence is split into fixed-length segments of length $L_{\text{seg}}$. Each segment uses standard Transformer attention with RoPE for positional encoding. Past KV pairs from previous segments are compressed into a memory block, and for the $i$-th segment the memory and normalization are updated as

$M_i = M_{i-1} + \sigma(K_r)^T V$

and

$$z_i = z_{i-1} + \sum_{j=1}^{L_{\text{seg}}} \sigma(k_{r,j}),$$

where $\sigma$ is a nonlinearity; ReLU is used for efficiency.

For training and quantization stability, a normalization variant is added:

$$M_i = M_{i-1} + \mathrm{norm}\left(\sigma(K_r)^T V\right), \quad M_i = \mathrm{norm}(M_i)$$

and

$$z_i = z_{i-1} + \mathrm{norm}\left(\sum_{j=1}^{L_{\text{seg}}} \sigma(k_{r,j})\right), \quad z_i = \mathrm{norm}(z_i).$$

The stated motivation is that memory-compressed activations can exceed FP16 representable range, which can destabilize training and degrade quantization.

During inference, memory is decompressed to form attention with current queries. The memory attention is transformed by an MLP,

$$\tilde{A}_{\mathrm{mem}} = W_2 \cdot \mathrm{ReLU}(W_1 A_{\mathrm{mem}} + b_1) + b_2,$$

and fused with local dot-product attention through a sigmoid gate,

$$A_{\mathrm{com}} = \mathrm{sigmoid}(g) \odot \tilde{A}_{\mathrm{mem}} + \left(1 - \mathrm{sigmoid}(g)\right) \odot A_{\mathrm{dot}}.$$

This allows dynamic balancing between short-range segment information and long-range compressed memory.

EdgeInfinite also retains two explicit token classes in KV form: sink tokens, defined as initial tokens that preserve global anchors, and window tokens, defined as the most recent tokens for local recency. This selective retention is similar in spirit to streaming methods, but operates within the compressed-memory formulation rather than pure token pruning.

3. Segmented Supervised Fine-Tuning

The principal training contribution is Segmented Supervised Fine-Tuning, or S-SFT, which aligns supervised instruction tuning with the segmented memory mechanism used by EdgeInfinite (Chen et al., 1 Aug 2025). The design objective is to train on long instruction examples without breaking the model’s blockwise memory structure.

The training pipeline has two stages. In continual pre-training, only the memory-gating module is adapted to general long-context corpora. The trainable parameters are about 0.15% of total weights; the data comprise about 183k samples from long documents, books, Wikipedia, and related sources; and training is performed for 1 epoch with learning rate 5e-3. In the second stage, SFT with S-SFT specializes the same module for domain tasks such as summarization and question answering, using about 30k task-specific datasets, again for 1 epoch, now with learning rate 8e-3.

Segmentation proceeds by packing the input sequence $X=\{x_1,\ldots,x_L\}$ into blocks of length

$i$0

followed by

$i$1

For continual pre-training, token-level cross-entropy is computed per segment,

$i$2

and aggregated as

$i$3

For supervised fine-tuning, both input $i$4 and label $i$5 are padded to the same block length $i$6, allowing every training sample to decompose into fixed-size segments. The S-SFT objective computes loss only on target tokens,

$i$7

where $i$8 indexes the segments containing response tokens. This excludes prompt tokens from the supervised loss and directly targets instruction-following behavior.

A distinctive mechanism is the differentiated forward pass:

$i$9

Here, $M_i = M_{i-1} + \sigma(K_r)^T V$0 performs a full forward pass through all layers and is used when a segment contains target tokens, whereas $M_i = M_{i-1} + \sigma(K_r)^T V$1 performs a memory-only forward pass that stops after the final attention layer and updates only the memory state $M_i = M_{i-1} + \sigma(K_r)^T V$2 and normalization term $M_i = M_{i-1} + \sigma(K_r)^T V$3. Padding segments are skipped. The algorithmic flow consists of shuffling the dataset, splitting it into batches, initializing $M_i = M_{i-1} + \sigma(K_r)^T V$4 and $M_i = M_{i-1} + \sigma(K_r)^T V$5, splitting each example into segments, using memory-only forward on non-target segments and full forward on target segments, backpropagating the total loss, and updating parameters.

The practical significance assigned to S-SFT is explicit: continual pre-training alone improves long-context handling but leaves the model weak at instruction following. The original EdgeInfinite without SFT is reported to show degraded on-device performance and more hallucination in real tasks. S-SFT therefore functions as the mechanism that connects long-context efficiency to downstream task utility.

4. Quantization and fixed-shape NPU deployment

EdgeInfinite-Instruct couples its training recipe to a deployment path tailored to mobile NPUs (Chen et al., 1 Aug 2025). The deployed configuration uses mixed precision with 4-bit weights and 16-bit activations under the MediaTek Neuropilot SDK. Static quantization is used for better NPU performance, calibration datasets are collected on relevant prompts, multiple experiments are run to select the best quantization algorithm and precision configuration, and Hessian-guided optimization is used to improve quantization quality. Related quantization families mentioned in the discussion include RTN, GPTQ, and rotation-based methods such as QuaRot and SpinQuant, but the deployed implementation uses Neuropilot’s PTQ pipeline with Hessian guidance.

The accuracy-efficiency tradeoff is handled by retaining activations at 16-bit to reduce activation degradation, using 4-bit weights to obtain most of the memory and bandwidth savings, calibrating on task-relevant prompts, and separately processing the memory update module because the memory compression path can produce activations that exceed FP16 range during training. The normalization added to the memory update equations is explicitly motivated by this deployment constraint.

A second deployment layer is the fixed-shape computation graph. The work assumes that NPUs generally prefer fixed tensor shapes, static graphs, predictable memory allocation, and minimized runtime branching. Since a vanilla autoregressive LLM has variable prompt length and variable cache growth, the model is compiled into two fixed-shape graphs: a prefill graph and a generation graph. The prefill graph processes a fixed chunk of prompt tokens in one pass, with typical input length 128 or 256 tokens depending on use case and hardware utilization. The generation graph is autoregressive with input token length 1.

Cache sizes are customized to scenario-specific bounds on maximum prompt length and maximum generation length. Under the reported quantized setup, FullKV cache grows with prompt length, reaching 4512 / 8512 / 16512 for prompt lengths 4k / 8k / 16k in prefill and generation. EdgeInfinite-Instruct instead uses fixed cache sizes: prefill cache 2548 and generation cache 3060, independent of prompt length. The deployed model uses $M_i = M_{i-1} + \sigma(K_r)^T V$6, $M_i = M_{i-1} + \sigma(K_r)^T V$7, and $M_i = M_{i-1} + \sigma(K_r)^T V$8. The average retained KV cache is about 1524 tokens, composed of 300 sink tokens, 200 window tokens, and approximately 1024 residual segment length on average.

The runtime differs from a standard LLM runtime in three stated respects. First, the memory update module requires separate handling when the cache is full. Second, the runtime must compress current KV cache contents into memory state $M_i = M_{i-1} + \sigma(K_r)^T V$9 and normalization term $$z_i = z_{i-1} + \sum_{j=1}^{L_{\text{seg}}} \sigma(k_{r,j}),$$0, clear the cache, and reset positions, while keeping sink tokens and handling window tokens specially so that they are not merged into the current memory update. Third, cache clearing may require recomputing window tokens and switching modes; in practical deployment this is avoided by expanding the generation cache to the maximum expected generation length $$z_i = z_{i-1} + \sum_{j=1}^{L_{\text{seg}}} \sigma(k_{r,j}),$$1 for scenarios with long prompts and short generations, thereby eliminating memory updates during generation. This suggests that deployment simplicity is treated as a first-class systems objective rather than a secondary engineering detail.

5. Evaluation protocol and reported results

Evaluation is reported on LongBench tasks and smartphone business scenarios, together with a 32K-context variant using Qwen3-4B-32K (Chen et al., 1 Aug 2025). The LongBench task families are single-document QA, multi-document QA, and summarization. The listed datasets are NrtvQA, Qasper, MF-en, and MF-zh for single-document QA; HotpotQA, 2WikiMQA, MuSiQue, and DuReader for multi-document QA; and GovReport, QMSum, MultiNews, and VCSUM for summarization. Smartphone business scenarios include key-point extraction and summarization. The compared baselines are FullKV, SnapKV, PyramidKV, StreamingLLM, EdgeInfinite, and, for the long-context variant, Qwen3-4B-32K FullKV.

On LongBench in full precision, the overall average scores are 24.24 for FullKV, 22.74 for SnapKV, 22.92 for PyramidKV, 20.61 for StreamingLLM, 22.67 for EdgeInfinite, and 24.42 for EdgeInfinite-Instruct. The task-level averages for EdgeInfinite-Instruct are 26.71 for single-document QA, 25.09 for multi-document QA, and 21.20 for summarization. The reported interpretation is that EdgeInfinite-Instruct improves strongly over EdgeInfinite on most datasets, especially summarization and multi-document QA.

Under quantization, the overall LongBench averages are 21.63 for FullKV, 13.75 for EdgeInfinite, and 21.11 for EdgeInfinite-Instruct. The quantized model remains strong on multi-document QA and summarization, though FullKV still leads on some single-document QA settings. On the Qwen3-4B-32K long-context variant, the overall average is 37.59 for FullKV and 31.13 for EdgeInfinite-Instruct. The stated explanation is that long-context models benefit from RoPE extrapolation, whereas chunked compression limits the effective RoPE window to $$z_i = z_{i-1} + \sum_{j=1}^{L_{\text{seg}}} \sigma(k_{r,j}),$$2, so the method cannot fully exploit 32K positional extrapolation.

In smartphone business evaluation, with Qwen2.5-Max used as evaluator and scores reported out of 10, FullKV obtains 6.312 on key-point extraction and 6.519 on summary, while EdgeInfinite-Instruct obtains 5.952 and 6.233, corresponding to 94.3% and 95.6% of FullKV respectively. The reported conclusion is near-parity with the full-KV baseline in realistic product scenarios.

The on-device measurements are collected on a MediaTek Dimensity 9400 NPU, with throughput in tokens per second and peak memory in GB. At prompt lengths 4k / 8k / 16k, FullKV prefill throughput is 643.85 / 335.02 / 190.95, while EdgeInfinite-Instruct reaches 1293.35 / 1240.86 / 1165.98. FullKV generation throughput is 26.62 / 23.21 / 13.35, while EdgeInfinite-Instruct records 24.85 / 24.90 / 24.38. Peak memory for FullKV is 1.86 / 2.13 / 2.72 GB, whereas EdgeInfinite-Instruct remains at 1.76 / 1.77 / 1.80 GB.

Metric	FullKV	EdgeInfinite-Instruct
LongBench overall, full precision	24.24	24.42
LongBench overall, quantized	21.63	21.11
Prefill throughput at 16k	190.95	1165.98
Generation throughput at 16k	13.35	24.38
Peak memory at 16k	2.72 GB	1.80 GB

The source also emphasizes TTFT improvements qualitatively, but no separate numeric TTFT table is provided in the excerpt. No explicit energy numbers are reported.

6. Limitations, interpretation, and place within edge LLM research

The main reported limitation is that single-document QA remains relatively weaker than some baselines, and the stated reason is information loss from chunk-wise memory compression, which is less ideal for precise localization (Chen et al., 1 Aug 2025). A second limitation is structural rather than merely empirical: because the method compresses context in chunks, its effective RoPE window is bounded by $$z_i = z_{i-1} + \sum_{j=1}^{L_{\text{seg}}} \sigma(k_{r,j}),$$3, so it does not fully exploit long-context positional extrapolation in 32K models. These constraints distinguish it from full-KV execution, which preserves exact token-level locality and long positional spans at substantially higher runtime cost.

The work also clarifies a common misunderstanding about what is being changed. EdgeInfinite-Instruct does not introduce a new backbone in the sense of replacing the Transformer with an alternative sequence architecture. Its novelty lies in a combined training-and-deployment recipe built on top of EdgeInfinite: compressed-memory execution for long-context efficiency, Segmented Supervised Fine-Tuning for instruction-following behavior, fine-grained PTQ for 4-bit-weight and 16-bit-activation inference, and fixed-shape graphs with scenario-specific cache sizing for mobile NPU compatibility.

Its ablation-style interpretation is distributed across comparisons rather than presented as a dedicated table. The comparison between EdgeInfinite and EdgeInfinite-Instruct indicates that S-SFT is essential for quality; the quantized results indicate that mixed-precision deployment preserves most task performance; the throughput and memory results indicate that fixed-shape graph design and cache control have direct systems impact; and the added normalization indicates that numerical-range control is important for memory-compressed models under FP16 and PTQ constraints.

Taken together, the method is best characterized as a practical bridge between supervised instruction optimization and NPU-level efficiency for long-context edge deployment. This suggests a research direction in which edge LLM design is driven not by a single algorithmic innovation, but by co-design across model adaptation, cache management, quantization, and hardware execution constraints.