HUOZIIME: On-Device LLM-Powered Chinese IME
- HUOZIIME is a fully on-device LLM-driven Chinese input method that integrates deep personalization with context-aware, keystroke-level candidate generation.
- It employs a hierarchical memory system and GRPO-based control to support cross-session adaptation and real-time, personalized short-phrase prediction under mobile constraints.
- The system achieves low latency through optimizations like KV-Splice and thread-affinity scheduling while ensuring privacy by confining all processing to the device.
Searching arXiv for the specified paper and closely related sources to ground the article in current literature. HUOZIIME is a fully on-device, LLM-driven Chinese input method editor (IME) designed for deep personalization, integrating a lightweight LLM directly into the keystroke-level candidate generation loop rather than treating generation as a separate assistant feature. It is presented as a response to three limitations of existing IMEs: manual, shallow interaction; weak personalization based on short in-session context or static persona presets; and privacy risks induced by cloud-based inference. The system combines stylized post-training, a three-level hierarchical memory, GRPO-based memory control, and mobile-oriented runtime optimizations to support personalized short-phrase prediction, cross-session adaptation, and privacy-preserving local execution under mobile resource constraints (Shan et al., 23 Mar 2026).
1. Problem Setting and Design Objective
Mobile IMEs are the primary interface for text entry on phones, and in Chinese they additionally mediate pinyin-to-character conversion. In the formulation underlying HUOZIIME, conventional IMEs remain largely transliteration engines: they map keystrokes to characters effectively for literal text entry, but they do not model user intent, writing style, or long-range discourse context in a persistent way. Even when LLMs are incorporated into commercial systems, they are typically used for whole-sentence rewriting or long-form suggestions and remain loosely coupled from the core keystroke loop.
The central concept introduced for HUOZIIME is deep personalization. In this usage, personalization extends beyond a short context window or a static style preset and includes a user’s stylistic identity, longer-term preferences and facts, and cross-session, cross-app context. The intended outcome is real-time personalized short-phrase prediction and completion, with a user-specific memory that can be maintained and updated over time while remaining fully on-device.
This objective is framed as a quality-efficiency-privacy trilemma. LLMs are compute- and memory-intensive, phones have limited CPU, RAM, and power budgets, and IME interaction requires sub-100 ms per step for acceptable typing experience. Personalization by continual adaptation can further destabilize a model if it depends on frequent fine-tuning or heavyweight memory management. HUOZIIME is therefore designed around the constraint that personalization must be strong, latency must remain compatible with typing, and private text must not leave the device (Shan et al., 23 Mar 2026).
2. System Architecture and Interaction Loop
HUOZIIME consists conceptually of five coordinated components: a base LLM with stylized post-training, a hierarchical memory engine, a GRPO-based memory trigger, an optimized on-device inference runtime, and IME/UI integration through GhostText and a candidate panel. The backbone is from the Qwen3-0.6B family; the memory system is structured into L1, L2, and L3; the control policy decides whether to complete text directly, retrieve memory, extract new memories, or refuse extraction; and the runtime builds on llama.cpp with RadixTree-based KV reuse, KV-Splice, and thread-affinity scheduling.
The user-facing behavior is intentionally close to a conventional IME. The top predicted continuation appears inline at the cursor as GhostText, while a secondary candidate panel shows alternative completions. Acceptance occurs through normal tapping behavior, so the system augments rather than replaces familiar IME workflows. This tight coupling distinguishes HUOZIIME from systems in which the LLM functions as an auxiliary writing agent outside the main typing loop.
The interaction loop is incremental. As text is entered, the system collects the current context, which may include conversation history from the host app, prior inputs, and internal memory state. The LLM decodes candidates using cached style and recent-context information, and it may trigger memory retrieval when needed. The top candidate is rendered inline, alternatives are shown in the suggestion panel, and subsequent edits reuse prior prefixes so that only the changed suffix is recomputed. After a message is sent, the interaction trace is queued for asynchronous memory decisions, including structured fact extraction, persistent logging, or refusal to store noisy or sensitive content. This creates a self-updating personalization loop that remains local to the device (Shan et al., 23 Mar 2026).
3. Base Model, Stylized Post-Training, and Hierarchical Memory
The base model is a decoder-only Transformer from the Qwen3-0.6B series with RoPE positional encoding. On top of that backbone, HUOZIIME applies a stylized post-training pipeline intended to give the model initial human-like prediction ability and persona-conditioned style control. The pipeline uses large-scale Chinese conversational corpora from the Internet, augments them with high-quality LLM-synthesized dialogues, filters low-quality or off-style samples, and assigns each retained sample to one of three predefined persona styles. Training instances include a persona system prompt and dialogue history, and the model is fine-tuned to generate the next response conditioned on persona. The resulting model is intended to produce style-consistent, interactive short completions rather than only generic neutral text.
The core personalization mechanism is a three-level hierarchical memory inspired by CPU memory hierarchies and MemGPT-like agent architectures. The three layers serve different latency and persistence regimes.
| Memory level | Stored content | Primary role |
|---|---|---|
| L1 | Style-specific KV blobs; per-memory KV blobs | High-speed personalization without re-prefill |
| L2 | User-specific plaintext records with HNSW vector index | Auditable, interpretable semantic retrieval |
| L3 | Interaction traces and model decisions | Future on-device or offline fine-tuning |
L1 is a high-speed cache layer that stores style-specific KV blobs and precomputed attention states for frequently used facts or patterns. Its purpose is to inject KV states directly into decoding and avoid repeated prefill. L2 stores plaintext facts such as workplace, nicknames, location, or preferred terminology, and it maintains a lightweight HNSW vector index for semantic recall. This layer is explicitly auditable and interpretable. L3 stores interaction traces, user choices, and model decisions such as whether retrieval was triggered or refused, providing data for slower structural learning through later fine-tuning.
A common misconception would be to equate this personalization scheme with continual weight updates during ordinary typing. The prototype does not emphasize on-device gradient updates such as LoRA in the online path; most adaptation is memory-based, with the base model kept stable and user specificity externalized into memory. This suggests that the system’s notion of continual personalization is deliberately designed to reduce catastrophic forgetting and uncontrolled drift while preserving persistent adaptation (Shan et al., 23 Mar 2026).
4. Memory Control, Retrieval, and Continual Adaptation
HUOZIIME does not retrieve or store memory at every step. Instead, it uses a GRPO-based policy, described as PPO-like RL, to decide among four task classes: A, direct completion; B, memory retrieval; C1, memory extraction; and C2, extraction refusal. Control tokens are emitted by the model to indicate these operations, including <MEM_RETRIEVAL> for retrieval and <NO_MEM> for refusal.
For retrieval, the current context is converted into a query, the L2 HNSW index is searched, and the top-K memories are returned. Their plaintext content and any precomputed KV segments can then be injected into the active context. For extraction, portions of the interaction are converted into structured facts for persistent storage in L2, typically after an interaction completes and in an asynchronous manner. Refusal is used when the context is noisy, irrelevant, or sensitive, and is explicitly intended both to prevent memory pollution and to protect privacy.
The GRPO training objective combines a thinking-format reward and a task-content reward. Reward shaping depends on the task class. Normal continuation rewards high-quality direct completion and penalizes unnecessary retrieval tags. Memory retrieval rewards proper formatting and a meaningful query while penalizing leakage of user text outside tags. Memory extraction rewards valid JSON with rich fields and penalizes invalid JSON or inappropriate refusal. Refusal rewards correctly emitting pure NO-MEM when refusal is required. The paper characterizes this RL stage as sharpening the boundary between retrieval, extraction, and refusal, thereby making the memory policy more reliable and autonomous.
Memory is integrated at two levels. First, retrieved plaintext facts are appended textually to the prompt, grounding generation in explicit records. Second, precomputed KV segments from L1 can be merged directly into the active KV cache through KV-Splice. The result is memory-grounded generation without requiring frequent full re-prefills or continuous model fine-tuning. Privacy is preserved by keeping all three memory layers on-device and by using the refusal mechanism to avoid storing sensitive or irrelevant content (Shan et al., 23 Mar 2026).
5. On-Device Runtime and Systems Optimization
A substantial part of HUOZIIME is systems work required to make an LLM-based IME responsive on mobile hardware. The original model size is reported as approximately 1.34 GB, while the deployed model uses Q4_0 quantization and is reduced to approximately 485 MB. The L1 KV cache is bounded to 24 MB. On a device with 12 GB RAM and a MediaTek Dimensity 9000 SoC, the dynamic peak additional RAM is approximately 1.12 GB, leaving room for other applications.
Interactive typing produces frequent small edits, so prefix recomputation is a central cost. HUOZIIME models prefixes as a compressed RadixTree whose nodes store KV caches. When the user edits text, the runtime finds the longest common prefix with previous states and resumes decoding from the matched node, recomputing only the changed suffix. The paper describes this as transforming interactive editing into incremental tree traversal.
Memory insertion into an existing prompt ordinarily requires re-running the model over the augmented sequence. HUOZIIME addresses this with KV-Splice. Given a base sequence and a memory-augmented prompt of the form , the runtime identifies the common prefix and suffix, determines the deleted and inserted spans, adjusts RoPE-based KV states through position-offset mapping and phase-shift correction, recomputes only the new memory tokens and the last tail token, and overlays the updated KVs into the main sequence. If KV states are inconsistent, a full baseline prefill is used as a safety fallback.
The runtime also uses thread-affinity scheduling on big.LITTLE hardware, pinning attention and GEMM kernels to big cores to reduce tail latency and improve thermal stability under bursty workloads. Measured performance is reported as prefill throughput greater than 260 tokens/s, decode throughput stable at 24–25 tokens/s across contexts up to 512 tokens, and cold-start Time To First Candidate (TTFC) of approximately 800–1700 ms depending on context length. Once interaction begins, however, prefix reuse and KV-Splice produce what is described as near-zero perceived TTFC for incremental typing. This suggests that the system’s practical responsiveness depends less on cold-start latency alone than on how effectively it amortizes computation across successive keystrokes (Shan et al., 23 Mar 2026).
6. Evaluation, Comparative Position, and Limitations
Evaluation focuses on two axes: memory-pipeline effectiveness and on-device inference performance. The memory pipeline is assessed at four stages: triggering, processing, retrieval, and grounded generation. Reported results are 342 / 343 = 99.7% for memory trigger success, 163 / 169 = 96.4% for normal extraction processing, 87 / 122 = 71.3% for refusal processing, 179 / 200 = 89.5% for Retrieval@4, and 156 / 179 = 87.2% for grounded generation by manual inspection. The paper notes that Retrieval@4 aligns with the four-candidate UI, so high top-4 recall maps naturally onto the interaction design.
The user study is preliminary: N = 4 participants with NLP backgrounds engaged in free multi-turn conversations. Qualitative findings include a strong sense of cross-session recall, improved context-aware styling, and reduced typing friction. These observations are stated to be consistent with improvements in keystroke savings rate, although exact KSR numbers are not reported. The evidence therefore establishes feasibility and promising behavior rather than a large-scale behavioral benchmark.
Relative to traditional IMEs based on n-gram models or small neural models for pinyin-to-character conversion, HUOZIIME reframes the IME as LLM-based next-phrase generation with persistent memory and persona conditioning. Relative to cloud-based LLM keyboards such as SwiftKey + GPT-4, BaiduIME + ERNIE, or SogouIME + Hunyuan, the distinction is that inference and memory are strictly local, with no automatic upload of keystrokes or context. Relative to other on-device personalization approaches based on federated learning, lightweight adapters, or lexicon adaptation, the claimed novelty lies in tight keystroke-level integration, hierarchical memory with GRPO control, advanced KV management under RoPE, and use of a unified post-trained LLM for both generation and memory decisions. To the authors’ knowledge, it is the first generative Chinese IME with fully on-device, memory-augmented personalization in this setting (Shan et al., 23 Mar 2026).
The authors also state several limitations. A 0.6B-parameter model constrains deep reasoning, complex tool use, and multi-step planning, and may introduce agentic instabilities such as over-retrieval or drift in retrieval tokens during autoregressive sampling. Cross-app context acquisition on Android remains constrained by sandboxing, with the Model Context Protocol helping only when supported. Filtered training data may still contain bias, and the system still depends on OS-level protections if the device itself is compromised. Future directions include customized pre-training for stronger agentic capabilities and better formatting, more advanced edge-side optimization, stronger multilingual support, broader domain coverage, and better user-facing tools for inspecting, editing, and deleting L2 memories. The code and package are released openly, and the implementation builds on llama.cpp, YuyanIME, and Qwen3-0.6B.
