Kimi K3: Open Frontier Intelligence
Abstract: We introduce Kimi K3, a 2.8T parameter Mixture-of-Experts model with 104 billion activated parameters, native vision capabilities, and a 1-million-token context window. Kimi K3 is built on Kimi Delta Attention and Attention Residuals, which improve information flow across sequence length and model depth. Together with Stable LatentMoE, which effectively activates 16 of 896 routed experts per token, and refined training and data recipes, these advances yield an approximately 2.5x improvement in overall scaling efficiency over Kimi K2. Post-training highlights reinforcement learning across general, agentic, and coding domains and multiple reasoning-effort levels, enabling compositional generalization and robust long-horizon execution. At 2.8T scale, Kimi K3 is supported by infrastructure advances in multiple areas: algorithm-system co-design for KDA, perfectly balanced expert-parallel training with efficient memory management, million-token agentic RL with persistent rollout and sandbox states, and deployment innovations. Extensive evaluations show that Kimi K3 achieves frontier-level performance across long-horizon coding, agentic, knowledge, reasoning, and vision tasks. While its overall performance still trails the most powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol, Kimi K3 consistently outperforms other open and proprietary models evaluated in our suite. We release the full Kimi K3 model weights to facilitate future research and accelerate the broader deployment and adoption of frontier intelligence.
First 10 authors:
Summary
- The paper introduces Kimi K3, a 3T-parameter model with novel Kimi Delta Attention and Stable LatentMoE for efficient long-context processing.
- It demonstrates a 2.5× improvement in scaling efficiency, integrating native multimodal MoE design with advanced RL and post-training pipelines.
- The study details critical infrastructure innovations—such as expert parallelism and optimized memory management—that enable low-cost, ultra-long sequence inference.
Kimi K3: An Expert Analysis of Open Frontier Intelligence
Model Overview and Key Innovations
Kimi K3 represents a substantial advance in open-weight LLMs, being the first released model in the 3T-parameter class with 2.8T total parameters and 104B activated per-token parameters. The architecture is a native multimodal Mixture-of-Experts (MoE) design, supporting both million-token context and direct vision input without post-hoc modality adaptation. The core architectural contributions include the introduction of Kimi Delta Attention (KDA) for efficient and stable long-sequence processing, Attention Residuals (AttnRes) for cross-layer representation aggregation, and Stable LatentMoE for scalable, highly sparse expert routing with robust optimization and load balancing.
The pre-training strategy scales bidirectionally: both in model size and in reinforcement learning (RL) phase, covering test-time scaling (agentic reasoning, tool use, coding, multimodal tasks) at long context. Infrastructure advances address core bottlenecks arising from this regime: context and expert parallelism, memory management for the 3T-class MoE, and scalable RL with persistent, resumable sandboxes for agentic rollouts. The model weights are released openly.
Architecture Analysis
Information Flow and Sequence Scaling
Kimi K3's architecture explicitly targets scaling along sequence, depth, and width axes:
Token Mixing. A hybrid attention structure interleaves three KDA layers and one Gated MLA per block, leveraging KDA's efficient, recurrent long-sequence mixing and MLA's global attention for periodic content-based aggregation.
Depth Mixing. AttnRes allows each layer to attend over all previous representations, replacing uniform residuals with learned, selective depth attention, implemented efficiently in both full and block-partitioned forms.
Channel Mixing. Stable LatentMoE routes each token through 16 of 896 experts per layer, with routed experts operating in a reduced latent space and shared experts guaranteeing fallback capacity and stabilization.
Figure 1: Kimi K#1{3} main results.
These advances deliver a reported 2.5× improvement in scaling efficiency over prior Kimi iterations.
Key Architectural Modules
- KDA with Lower-bounded Decay: KDA introduces log-decay gating per channel with a lower-bound enforced by a scaled sigmoid, enabling numerically stable, high-capacity recurrent attention over million-token contexts without expensive position-pair computations. All causal tiles use dense matrix multiplications, maximizing GPU utilization.
- Attention Residuals: Full/top-K attention over depth, with RMS normalization, enables selective information routing—empirically recovering most of the benefit with blockwise aggregation.
- Stable LatentMoE: Combines RMSNorm before up-projection, SiTU-GLU activation for smoothing large activations, and quantile-based (rather than loss-augmented) expert load balancing. Quantile balancing is implemented in a communication-efficient manner using histogram-based accumulation.
- MoonViT-V2: Vision encoder trained from scratch with next-token prediction yields high stability and matches contrastively initialized baselines, demonstrating that contrastive pretraining is not essential at scale.
Figure 2: From-scratch MoonViT-V2 exhibits stable gradient norms, outperforming SigLIP-initialized configurations.
Training and Scaling
Data Composition
Kimi K3 is trained on a data mixture including high-quality web, code, mathematical, and curated knowledge texts, as well as a large corpus of captioned and programmatic multimodal data. Rephrased corpora and deduplicated, quality-filtered vision data are upsampled for long-context adaptation, with synthetic permutation ensuring the model must leverage full context for certain tasks.
Scaling Laws
A comprehensive scaling law search (token-per-parameter ratio, batch size, learning rate schedule) demonstrates that the new architecture achieves roughly 2.5× scaling efficiency relative to Kimi K2, and that cosine decay schedules outperform prior WSD configurations after independent hyperparameter tuning.
Figure 3: Scaling-law curves, demonstrating enhanced data and computational efficiency of Kimi K#1{3}.
Long-context Strategy
No explicit positional encoding is used: KDA's recurrence and gating encode positional signals, enabling zero-modification extrapolation to 1M tokens. Training follows a progressive context-length curriculum, with upsampling and synthetic permutations guaranteeing global context utilization.
Post-Training and RL
Multi-stage Post-training
The post-training pipeline is SFT → domain/effort-specialized RL → Multi-Teacher On-Policy Distillation (MOPD):
- SFT: Human and model-annotated large-scale data, expanding agentic task coverage with a unified XTML-based chat format.
- RL: Nine domain–effort experts (spanning general, agentic, coding × low/high/max effort) are trained using partial rollouts and per-problem budget control.
- MOPD: The nine experts distill their behaviors into a single unified model, with per-token, per-domain OPD reward signals.
Figure 4: RL test-time computation scaling: as RL FLOPs scale, both capability and average assistant trajectory length increase.
Environments and Task Synthesis
Long-horizon RL is implemented in white-box sandboxed environments, with tasks synthesized from hierarchically organized, agent-constructed knowledge graphs. Specialized evaluators enable verifiable supervision in agentic, coding, web, kernel optimization, and personal-assistant tasks.
Figure 5: Knowledge-graph-guided task synthesis systematically explores fine-grained concepts and retrieves domain-specific source material.
Agentic RL leverages persistent, resumable sandbox states to support ultra-long trajectories—a distinguishing feature for robust evaluation and training.
Figure 6: Completion curves on Camera Repair Management System, highlighting black-box system replication by agentic RL.
Infrastructure and Systems
Infrastructure is co-designed for three unique stressors: hybrid-KDA attention, ultra-sparse MoE at scale, and million-token contexts:
- Expert parallelism is perfectly balanced with MoonEP, ensuring all-to-all token routing is always optimally scheduled with redundant experts, with a provably tight bound and minimal communication overhead.
- Memory management combines unified activation storage and quantized offloading, with pipeline ZeRO-2 and P2P Muon optimizer communication, supporting large activations and optimizer states.
- KDA Context Parallelism overcomes the recursive, state-carrying nature of KDA, with efficient intra- and inter-device planning and transfer, supporting linear scaling for extremely long sequences.
- Resumable sandbox infrastructure (AgentENV) enables highly parallel, isolated, and revivable agent environments, critical for persistent long-horizon RL.
Figure 7: OVL and scheduling of computation, communication, offloading across pipeline-parallel stages in 3T-class training.
Performance, Cost, and Case Analysis
External and Internal Benchmarks
Kimi K3 approaches, but does not match, the performance of leading proprietary models (Claude Fable 5, GPT-5.6 Sol) on aggregate, but outperforms all other open and many proprietary models on long-horizon coding, agentic, and multimodal vision benchmarks. Key areas of relative strength include agentic orchestration, research, and process-based code generation.
Cost Efficiency
Kimi K3 achieves top scores at substantially lower inference cost than closed models: for example, on BrowseComp it achieves higher accuracy than leading proprietary models at approximately half the per-task cost, due in large part to aggressive MoE sparsity and architectural efficiency.
Figure 8: Score vs. per-task inference cost across coding and agentic evaluations; Kimi K#1{3} sits near the Pareto frontier.
Technical Case Studies
- GPU Kernel Optimization: Kimi K3 demonstrates substantial autonomous improvement in GPU kernel code, halving execution times on critical kernels and matching or exceeding performance of Claude Fable 5, well ahead of open baselines.
- Compiler Development: The model developed MiniTriton, an efficient and accurate Triton-like kernel library and compiler, with end-to-end validation across benchmarks and strong convergence properties.
Figure 9: MiniTriton case: Rooflines, loss curves, and DP scaling for compiler and kernel development produced autonomously by Kimi K#1{3}.
- Hardware RTL Design, Scientific Computing, Knowledge Synthesis, Video Editing: The model builds usable, performant RTL, implements complex research pipelines with high data and code literacy, and schedules fine-grained multimodal tasks involving both code and vision.
Implications, Open Problems, and Future Directions
Kimi K3 demonstrates the feasibility and substantial utility of open 3T-class models for complex agentic and multimodal workloads, with strong evidence that both algorithmic innovation and scale remain important even in highly optimized regimes. The integration of scalable-long-context modeling, native multimodality, and stable MoE routing sets a new bar for open research, but the performance gap for highest-difficulty creative/research tasks persists.
Future developments may include:
- Further algorithmic advances in scalable context modeling (potentially combining recurrence and attention with more explicit memory models),
- Hierarchically structured RL and continual knowledge graph expansion,
- Further reductions in MoE/activation overhead through more efficient sparse expert initialization and task-bound routing.
The infrastructure advances in fast RL trajectory management and context-parallel optimization also suggest continued convergence of LLM and RLHF frameworks for high-throughput, multi-domain learning and deployment.
Conclusion
Kimi K3 is a substantial technical achievement in the open frontier of large-scale, agentic, multimodal modeling. It delivers strong performance and efficiency in both coding and agentic tasks at scale, is backed by robust infrastructure innovation, and stands as a practical testbed for further research in large-system learning, RL, and deployment at the 3T-parameter regime.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Kimi K3: What This Paper Is About
1) Simple overview
This paper introduces Kimi K3, a very large open-source AI model that can read, write, and understand images and videos. It’s built to handle extremely long inputs (up to about a million tokens — think hundreds of thousands of words) and to “think” for longer when a problem is hard. The team created new ways to make the model fast, stable, and efficient, and they trained it to plan and solve multi-step tasks like writing software over many steps. Most importantly, they’re releasing the full model so others can use and study it.
2) What questions the paper tries to answer
The paper focuses on a few big, easy-to-understand questions:
- How can we make open AI models much bigger and still run them efficiently?
- How can a model pay attention to really long documents or conversations without slowing down too much?
- How can we use a “team of experts” inside the model so only the right specialists are called when needed?
- How can we train the model to think longer and break big problems into steps (not just answer quickly)?
- What kind of computer systems are needed to train and run a model this huge?
3) How they built it (methods) in everyday language
Think of Kimi K3 as a giant school with many specialist teachers. For each word or image it processes, it doesn’t talk to every teacher — it picks a small group of the best-suited ones. The paper’s main ideas make that process smarter and faster.
- Mixture-of-Experts (MoE): a city of specialists
- The model has a total of 2.8 trillion parameters (think “knobs” the model can tune), but only about 104 billion are active at any moment.
- It has 896 “experts” (specialist mini-networks), and for each token (word/chunk), it consults only 16 of them — like calling the right few specialists instead of everyone. This keeps things fast and efficient.
- Kimi Delta Attention (KDA): focus for very long reads
- Attention is how AI “pays attention” to important parts of text.
- KDA is a new attention style that remembers what matters over long stretches without running out of memory. You can imagine it as smart note-taking: it keeps helpful summaries and updates them as it reads more, so it can handle super long documents.
- Gated MLA: regular “zoom-outs”
- Every so often, the model does a full “zoom-out” pass that lets it look globally across everything it has seen. This balances the fast, efficient reading (KDA) with moments of looking at the big picture.
- Attention Residuals: layers can look back
- Normally, each layer of the model only sees the layer right before it. Here, a layer can look back across many earlier layers, like flipping back through your notes to find the most relevant page. This helps information flow better through the whole network.
- Stable LatentMoE: keeping experts safe and balanced
- Safety valves for big numbers: the team uses special math tricks (like SiTU-GLU and RMSNorm) to prevent numbers inside the network from “blowing up,” which can crash training.
- Fair expert usage: a new “Quantile Balancing” method spreads the workload evenly across experts (like making sure a call center doesn’t overload one operator while others sit idle).
- Training to think in steps (Reinforcement Learning, or RL)
- They trained the model to plan, act, check results, and adapt — not just give one-shot answers.
- It practices in realistic settings (like coding, web tasks, professional knowledge work, and vision + tools), sometimes over hundreds or thousands of steps, and over very long contexts.
- They combine multiple specialized “teacher” policies into one model so it can do many things well.
- Heavy-duty infrastructure: the highways and warehouses
- Custom low-level code (“kernels”) and memory tricks for speed.
- Perfectly balanced expert-parallel training so all experts stay busy and efficient.
- “Save game” systems for long RL runs, keeping state across many steps.
- Smarter server scheduling so the model runs predictably in production.
4) What they found and why it matters
- Much better efficiency
- Kimi K3 uses about 2.5× better scaling efficiency than their previous model (Kimi K2). In plain terms: it gets more learning and performance per unit of compute.
- Strong performance across many tasks
- On long coding tasks, multi-step “agent” jobs, knowledge questions, reasoning, and vision tasks, Kimi K3 performs at a frontier level.
- It’s still slightly behind the most powerful closed models (like Claude Fable 5 and GPT-5.6 Sol), but it consistently beats other open and many proprietary models they tested.
- Handles very long inputs
- Up to 1 million tokens of context means the model can keep track of very long documents, conversations, or multi-step project histories without losing the thread.
- It’s open
- They are releasing the full model weights. This is rare for a model this large and capable, and it can accelerate research and practical use across the community.
Why it matters: These results show that open-source models can push into the same territory as the strongest private models by combining smarter architecture, training that teaches planning, and careful systems engineering. That’s good for transparency, research, and access.
5) What this could change (impact)
- Research boost: Scientists and engineers can study a truly massive, modern model in detail, try new ideas on top of it, and share results openly.
- Real-world apps: Long documents, complex software projects, web workflows, and vision + tools are all better served by a model that can think in steps over a huge context.
- Better AI habits: The model learns to plan, act, check, and revise — behaviors that make AI more reliable on complicated tasks.
- Blueprint for the future: Their hardware–software co-design, expert balancing, and attention tricks are likely to influence how other large models are built and served.
In short, Kimi K3 shows a practical path to making very large, open AI systems that can read massive inputs, think longer when needed, and perform well across many tough tasks — and it shares the full model so others can build on it.
Knowledge Gaps
Below is a single, consolidated list of concrete knowledge gaps, limitations, and open questions that remain unresolved by the paper. Each item is phrased to be actionable for future work.
- Quantify the trade-offs of the 3:1 KDA-to-MLA hybrid attention ratio across tasks and scales; ablate alternative ratios and spacing (e.g., 2:1, 1:1, periodicity) and measure impacts on long-context quality, latency, and memory.
- Analyze sensitivity to the KDA lower-bounded decay hyperparameter g_min (set to −5): learnable vs fixed bounds, per-head/per-channel bounds, and effects on long-range recall, stability, and gradient variance.
- Provide theoretical guarantees for KDA with a nonzero forgetting floor (α > e{g_min}): memory capacity, mixing properties, bias toward stale content, and conditions for numerical stability under BF16.
- Characterize how chunk size and 16-token tile size in KDA affect accuracy, numerical stability, streaming latency, and throughput on diverse hardware; explore adaptive chunking and tile-size autotuning.
- Evaluate the NoPE design in MLA: identify tasks where lack of explicit positional encoding in global layers harms performance (e.g., algorithmic positional tasks), and study interactions with KDA’s recency modeling.
- Compare full-rank, input-dependent output gates (in KDA and Gated MLA) against low-rank/parameter-efficient variants for accuracy, compute cost, overfitting risk, and quantization-friendliness.
- Formalize the precision–throughput trade-offs of keeping flash-attention outputs in FP32 during training; quantify inference-time precision choices on accuracy and latency, especially at 1M context.
- For Attention Residuals (AttnRes), ablate the number of blocks (N≈8 used): quantify memory/communication overhead vs. accuracy gains; when do Full vs Block AttnRes yield net benefits across depths and modalities?
- Investigate how AttnRes pseudo-queries are learned and regularized: risks of mode collapse (e.g., over-attending to shallow layers), stability across scales, and interpretability of depth-wise retrieval patterns.
- Provide head-to-head comparisons of AttnRes against deep skip/residual variants (e.g., parallel residuals, highway, dense connections) for training stability and depth efficiency.
- In Stable LatentMoE, ablate latent width ℓ, number of routed experts (896), and top-k (16) to map specialization, interference, and scaling curves; identify diminishing returns and optimal sparsity regimes.
- Assess whether RMSNorm before the up-projection in LatentMoE hampers amplitude-coded information; compare to alternative normalization points or norm-aware gating strategies.
- SiTU-GLU activation: study hyperparameter sensitivity (β1=4, β2=25), Lipschitz bounds, gradient flow, and comparisons to ReGLU/GeGLU/PowLU under equal FLOPs; provide kernels and runtime benchmarks.
- Evaluate SiTU-GLU under quantization (INT8, NF4, FP8) and mixed-precision serving; report accuracy/regression and kernel availability across vendors.
- Quantile Balancing (QB) routing: provide convergence/stability analysis under nonstationary data, variable batch sizes, and Top-k>1; quantify communication cost, staleness across DP shards, and fairness–specialization trade-offs.
- Investigate expert “deadness” and churn at 896 experts: detection criteria, rejuvenation/retirement policies, curriculum or temperature schedules that prevent expert collapse or oversaturation.
- Report expert utilization histograms over training and across domains/tools; link load balance to expert specialization and downstream generalization.
- Long-context (1M) fidelity: publish retrieval/needle-in-a-haystack curves across positions, degradation vs length, cross-chunk interactions, and catastrophic forgetting risks during ultra-long sequences.
- Provide end-to-end latency, memory footprint, and throughput for 1M-token inference across batch sizes and decoding strategies; identify practical serving limits and scheduler behaviors.
- Validate external KV-cache retention across long agentic rollouts: correctness under interruptions/resumptions, memory leaks, eviction policies, and reproducibility of stateful trajectories.
- Compare the proposed KDA+MLA+AttnRes stack against strong RoPE/YaRN baselines on long-context extrapolation beyond training length; test generalization to contexts >1M and hybrid windowing schemes.
- Reinforcement learning at million-token horizons: detail reward shaping, credit assignment, off-policy corrections, and sample efficiency; quantify stability and variance across seeds and domains.
- Multi-teacher on-policy distillation: analyze how effort-level and domain-specialized policies are weighted; measure loss of peak specialist capabilities, interference across skills, and catastrophic forgetting post-merging.
- Tool-use and autonomy: measure tool hallucination rates, verification efficacy, recovery from tool failures, and sandbox escape robustness; characterize safety under prompt injection and web content drift.
- Native vision pathway: provide data composition (images/videos), pretraining objectives, temporal resolution limits, video length handling, and synchronization between visual frames and text tokens.
- Cross-modal grounding: evaluate OCR-heavy, diagrammatic, and chart reasoning robustness; quantify visual-text grounding errors and calibration under distribution shift.
- Training data transparency: report dataset composition, deduplication strategy, licensing, multilingual coverage, and benchmark contamination audits, especially for coding and agentic benchmarks.
- Safety and alignment: supply red-teaming coverage, bias/toxicity metrics, privacy/PII leakage assessments, and jailbreak robustness—particularly for long-context and tool-mediated interactions.
- Robustness: stress-test under adversarial inputs, corrupted/noisy documents, contradictory long prompts, and dynamic web environments; document failure modes and mitigation strategies.
- Reproducibility: disclose compute budget, training schedules, optimizer and clipping settings, instability incidents, and variance across seeds; clarify which kernels/systems components are released and their maturity.
- Energy and cost: report training and inference energy use, carbon footprint, and cost-per-token vs. smaller baselines; study Pareto frontiers for quality vs. cost under MoE sparsity and test-time scaling.
- Portability and maintainability: assess whether custom kernels (KDA, MoonEP, QB) generalize across GPU vendors/architectures; provide fallbacks, compiler support, and performance regressions on non-NVIDIA hardware.
- Quantization and memory-optimized serving: characterize 4-bit/8-bit accuracy at long contexts, KV-cache compression with MLA, and interactions with AttnRes and KDA gating under memory pressure.
- Calibration and adaptive compute: expose confidence measures, abstention behaviors, and policies for dynamically allocating reasoning budget/tools; evaluate overthinking vs underthinking trade-offs.
- Post-training adaptation: develop methods (e.g., expert-specific LoRA, router-tuning, selective SFT/RL) to adapt K3 to new domains without destabilizing expert balance or eroding prior capabilities.
- Interpretability: map expert specializations, visualize routing dynamics, and analyze AttnRes layer-attention patterns; link these to failure cases and guide targeted data or routing interventions.
- Open-weights risks: assess misuse vectors, capability control mechanisms (watermarking, safety adapters), and feasibility of post-hoc editing/steering to mitigate unsafe behaviors.
- Gap analysis vs top proprietary models: provide fine-grained error typologies (reasoning steps, tool orchestration, long-horizon coding failures) and a prioritized roadmap of targeted improvements.
- Long-horizon coding evaluation: release deterministic harnesses, flakiness controls, timeouts, and verifiable scoring; quantify reproducibility and variance across toolchain versions.
- Inference-time precision and kernels: study accuracy/latency trade-offs when moving from FP32 training behavior to lower-precision serving for attention outputs and KDA updates.
- Video-at-1M use cases: benchmark end-to-end multimodal streaming (long videos + long text), report memory/latency ceilings, and analyze temporal reasoning limits in extended contexts.
Practical Applications
Immediate Applications
Below are concrete, deployable use cases that can be built now by leveraging the released Kimi K3 weights, its 1M-token context, native vision, agentic RL post-training, and the accompanying systems/serving techniques.
- Enterprise-wide long-context knowledge assistants: cross-document analysis at deal-room or archive scale
- Sectors: finance, legal, consulting, government, enterprise IT
- What: ingest and reason over millions of tokens from filings, contracts, emails, tickets, policies, and procedures; e-discovery, due diligence, audit and compliance reporting
- Tools/workflows: “room-scale” Q&A copilots; due-diligence checklists with verifiable references; secure on‑prem deployments of K3 with retrieval-augmented generation (RAG) that can stream 100k–1M context windows; context-prefix caching to amortize repeated headers/boilerplate
- Assumptions/dependencies: GPU memory and throughput to support 1M context; robust RAG/indexing; PII redaction/policy controls; careful prompt instrumentation to control cost of very long contexts
- Codebase-scale software engineering copilots
- Sectors: software, cloud, embedded/HPC
- What: refactoring and reasoning across monorepos; kernel-level tuning; long-horizon debugging with persistent state
- Tools/workflows: IDE extensions that load entire repos via sliding 1M-token windows; “agentic CI” that plans, edits, runs tests in resumable microVM sandboxes (as in the paper’s agentic RL infrastructure); kernel-optimization recipes distilled from K3’s coding RL curricula
- Assumptions/dependencies: repository access and code indexing; guardrails to avoid destructive edits; test-time budget policies to cap agent loops; secure sandboxing
- Document and form understanding with vision-in-the-loop
- Sectors: insurance, logistics, finance ops, public sector
- What: native parsing of scanned PDFs, tables, receipts, claims, waybills; multi-page/multi-document reconciliation within a single session
- Tools/workflows: multimodal intake pipelines that pair MoonViT-V2 encoders with long-context text reasoning; KV prefix caching for repeated headers/templates; human-in-the-loop validation
- Assumptions/dependencies: document quality (OCR when needed), domain-specific schemas, latency budgets for high-page-count inputs
- Persistent customer support and CRM copilots
- Sectors: retail, SaaS, telecom, fintech
- What: multi-session memory of user history, previous tickets, contracts, product versions; root-cause analysis spanning months of logs
- Tools/workflows: “million-token customer timeline” viewers; cache-aware serving that reuses long prefixes; budget-aware agent loops that escalate to human agents with evidence packs
- Assumptions/dependencies: data consent/governance; long-context cost management; clear escalation/verification policies
- Scientific literature and code lab notebooks
- Sectors: academia, biotech, materials, energy
- What: automated literature triangulation across thousands of papers; code+results notebooks with persistent context over experiments; protocol variance tracking
- Tools/workflows: “million-token notebook” environments; paper-to-code traceability; citation-grounded summaries
- Assumptions/dependencies: access to full texts/embargo constraints; provenance logging; reproducibility checks
- Web development and content production agents
- Sectors: media, marketing, SMB web agencies
- What: multi-step site/app scaffolding, asset generation, A/B variant trees with long-running context and tool calls
- Tools/workflows: web-dev sandboxes with resumable states; asset libraries streamed in as context; agentic RL heuristics to cap iterations
- Assumptions/dependencies: hosting/CI integration; IP/licensing review; content safety filters
- Model-building and serving efficiency for AI infra teams
- Sectors: AI infrastructure, cloud providers, model labs
- What: adopt Stable LatentMoE (896 experts, 16 active), Quantile Balancing (QB) for auxiliary-loss-free load balancing, KDA fused kernels, KDA Context Parallelism, and prefix caching
- Tools/workflows: expert-parallel training stacks (MoonEP-like) with static shapes and zero-copy comms; cache-/budget-aware schedulers; per-head normalization and SiTU-GLU activations for numerical stability
- Assumptions/dependencies: engineering investment to port kernels; hardware with BF16/FP32 headroom; MoE-aware orchestration; telemetry to monitor expert load
- Unified multi-specialist policy distillation
- Sectors: enterprises with many narrow copilots (legal, HR, finance, engineering)
- What: merge domain-specialized policies into one generalist model (multi-teacher on-policy distillation used in K3)
- Tools/workflows: fine-tuning pipelines that consolidate domain agents; evaluation harnesses at multiple reasoning-effort levels
- Assumptions/dependencies: access to teacher policies/datasets; license compatibility; careful catastrophic-forgetting mitigation
- Regulatory and compliance copilots on large archives
- Sectors: public sector, banking, healthcare admin
- What: cross-year rule traceability; policy impact summaries; audit trails with exact citations
- Tools/workflows: open‑weights K3 deployed on sovereign/on‑prem infrastructure; 1M-token “policy book” sessions; verifiable search with inline source spans
- Assumptions/dependencies: security accreditation; human review; retention and audit policies; cost controls for very-long contexts
Long-Term Applications
These are plausible applications enabled by the paper’s methods (long-context KDA+MLA, AttnRes, Stable LatentMoE, agentic RL at 1M tokens), but they will likely require further research, scaling, safety, or integration work.
- Autonomous software project agents with week-long memory
- Sectors: software, enterprise IT
- What: plan, implement, test, and deploy features over multi-week horizons across numerous tools and repos, maintaining consistent objectives in a 1M-token loop
- Dependencies/assumptions: robust verification, secure permissions, strong red-teaming; reliable budgeted reasoning policies; stronger grounding against hallucination; formal rollback strategies
- Longitudinal, multimodal clinical assistants
- Sectors: healthcare
- What: integrate imaging, labs, longitudinal EHR, and guidelines to produce draft care summaries and care-path suggestions with citations
- Dependencies/assumptions: regulatory approvals, medical-grade validation, bias and safety audits, specialized data and fine-tuning; hospital-grade privacy/on‑prem deployment; human clinician oversight
- Continuous compliance and risk observatories
- Sectors: finance, critical infrastructure, public sector
- What: always-on agents that process streams of logs, communications, and regulatory updates with million-token rolling windows to flag emerging risks
- Dependencies/assumptions: streaming long-context serving efficiency; drift detection; strong access controls and privacy; robust false-positive management
- Scientific discovery copilots and lab automation
- Sectors: pharma/biotech, materials, energy, climate
- What: persistent hypothesis planners that connect literature, lab protocols, and results across many runs; propose next experiments and analyze failures
- Dependencies/assumptions: integration with LIMS/ELNs and robotics; experiment verifiers; domain-specific safety constraints; increased reliability on tool use
- Personalized lifetime assistants with private long-term memory
- Sectors: consumer, productivity
- What: user-authorized “memory vaults” enabling planning and reflection across years of notes, files, and media
- Dependencies/assumptions: privacy-preserving storage and on-device/on‑prem inference; consent/granular deletion; efficient long-context at consumer cost points; societal norms and regulation
- Multimodal robotics planners with long-horizon context
- Sectors: manufacturing, logistics, service robotics
- What: fusing videos, task specs, and environment history for long-horizon planning and recovery
- Dependencies/assumptions: tight latency constraints; safety-certified control stacks; sim-to-real transfer; dependable multimodal grounding
- Unified enterprise “model-of-models” via multi-teacher distillation
- Sectors: large enterprises, platforms
- What: integrate many internal task-specific agents/policies into a single, capability-calibrated generalist
- Dependencies/assumptions: high-quality teacher policies; harmonized data licensing; continual learning infrastructure; comprehensive evals for regressions
- Energy- and cost-efficient frontier models inspired by KDA/AttnRes/LatentMoE
- Sectors: AI infra, cloud, model labs
- What: next-gen open and proprietary models that sustain accuracy at lower serving costs via sparse MoE, attention residuals across depth, and long-context linear attention hybrids
- Dependencies/assumptions: further kernel/system co-design; cross-vendor GPU support; community-standard libraries; stability at even larger scales
- Policy simulation and governance sandboxes
- Sectors: public policy, international orgs
- What: simulate policy changes using massive corpora (laws, case files, comments) with transparent reasoning chains and long-memory comparisons across jurisdictions
- Dependencies/assumptions: curated, updated corpora; rigorous bias/impact analysis; procedural safeguards and public oversight
Notes on feasibility and cross-cutting assumptions
- Cost/latency: Although only ~104B parameters are activated per token, million-token contexts are expensive; aggressive prefix caching, RAG, and budget-aware schedulers are necessary.
- Safety and reliability: Agentic RL enables long-horizon action loops; deploy with verifiers, tool permissioning, human approval gates, and audit logs.
- Data governance: Many applications hinge on secure, on‑prem or VPC deployments, PII controls, and detailed access policies.
- Hardware/software: Benefits like KDA Context Parallelism, fused kernels, and MoonEP-like expert balancing assume modern GPU hardware (BF16/FP32) and engineering to integrate or port kernels.
- Performance envelope: The paper states K3 trails the top proprietary models; high-stakes uses should include human review and targeted domain fine-tuning.
Glossary
- Adaptive throttling: Dynamic control of execution rate to manage system load and latency during long-running agentic rollouts. "our co-located system combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes"
- Agentic RL: Reinforcement learning where the model acts as an autonomous agent over long horizons with tools and environments. "For million-token agentic RL, our co-located system combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes to preserve long-lived model and environment state."
- Attention Residuals (AttnRes): A depth-wise attention mechanism that lets each layer attend to representations from all preceding layers instead of relying on uniform residual accumulation. "Attention Residuals (AttnRes) use learned pseudo-queries () to derive attention weights () over the embedding and preceding block outputs, enabling selective information flow across depth."
- Auxiliary-loss-free routing: MoE expert selection that balances load using bias updates rather than auxiliary losses. "Unlike auxiliary-loss-based routing~#1{fedus2022switch}, Kimi K#1{3} adopts auxiliary-loss-free routing~#1{deepseekaiv3}."
- BF16 dynamic range: The representable numerical range of the bfloat16 floating-point format, relevant for avoiding overflow. "The corresponding reciprocal rescaling factor is therefore smaller than and remains within the BF16 dynamic range."
- Block AttnRes (Block Attention Residuals): A memory-efficient variant of AttnRes that aggregates layers within blocks and applies attention across block summaries. "Under Block AttnRes, memory and communication overhead drop from to "
- Cache- and budget-aware fleet scheduling: Serving-time scheduling that considers cache reuse and computational budgets to ensure predictable performance. "Finally, specialized kernels, and cache- and budget-aware fleet scheduling translate these innovations into predictable production serving."
- Channel-wise one-step retention factor: Per-channel decay parameter that controls how much state is retained in KDA’s recurrent update. "Here, is the channel-wise one-step retention factor"
- Chunkwise parallel form: A computation scheme that is recurrent across chunks but fully parallel within each chunk to improve efficiency. "Chunkwise parallel form"
- Compositional generalization: Ability to solve new tasks by composing previously learned skills and behaviors. "enabling compositional generalization and robust long-horizon execution."
- Delta-rule recurrence: An online update rule that incrementally updates a state via a delta term; KDA extends it with gating. "KDA extends the delta-rule recurrence~#1{schlag-2021-deltanet,yang-2025-gdn} with a channel-wise forget gate~#1{team2025kimi}."
- Expert-parallel training: Distributing different experts of an MoE model across devices to parallelize and balance computation. "perfectly balanced expert-parallel training with efficient memory management"
- Flash attention: A memory- and bandwidth-efficient attention kernel; precision handling is important to control rounding bias. "To correct the biased rounding error that arises in flash attention, we adopt the method of~#1{qiu2025lowprecision}"
- Full Attention Residuals: The full variant of AttnRes that attends over all preceding layers without block reductions. "Full Attention Residuals"
- Full-rank gate: An input-dependent gating projection with full rank that modulates attention outputs channel-wise. "Full-rank gate"
- Gated Linear Units (GLUs): Feed-forward layers that multiply a value branch by a gated activation to control information flow. "Gated Linear Units (GLUs) modulate a linear value branch with a sigmoid-activated gate, computing ~#1{dauphin2017language}."
- Gated MLA: A global attention layer (MLA) augmented with an input-dependent output gate to modulate channels. "Kimi K#1{3} uses a layerwise hybrid of linear and global attention, combining KDA~#1{team2025kimi} with Gated MLA."
- Head-wise RMSNorm: Applying RMS normalization independently to each attention head’s output to stabilize scaling. "After applying head-wise RMSNorm~#1{zhang2019root} to the recurrent output, KDA applies data-dependent output gating"
- KDA Context Parallelism: A systems technique to parallelize KDA computations over long contexts efficiently across hardware. "we develop fused kernels, KDA Context Parallelism, and state-aware prefix caching to make KDA efficient within devices, across devices, and across requests."
- Kimi Delta Attention (KDA): A long-sequence attention mechanism that uses recurrent delta updates with channel-wise decay and efficient chunkwise computation. "Kimi Delta Attention (KDA)~#1{team2025kimi} provides efficient long-sequence mixing, with periodically interleaved Gated MLA layers preserving global interaction."
- KV-cache retention: Persisting key–value caches across steps or sessions to support very long contexts and agentic rollouts. "our co-located system combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes"
- LatentMoE: An MoE design that routes tokens through narrow latent-width experts while retaining shared full-width paths. "LatentMoE~#1{elango2026latentmoe} makes this expansion affordable by separating the full model width from the routed-expert width"
- MoonEP: The expert-parallel training infrastructure used to balance MoE expert execution and communication. "MoonEP provides perfectly balanced expert execution with static computation shapes and zero-copy communication,"
- MoonViT-V2: The visual encoder used to provide native vision capabilities for images and videos. "For native vision, MoonViT-V2 encodes images and videos, and a lightweight projector maps the resulting visual features into the shared embedding space"
- Multi-head Latent Attention (MLA): A factorized attention that caches low-dimensional content vectors and reconstructs keys/values on the fly to reduce KV-cache size. "Multi-head Latent Attention (MLA), introduced in DeepSeek-V2~#1{deepseekv2}, compresses the key--value representation of each token into a low-dimensional latent vector"
- Multi-teacher on-policy distillation: Consolidating specialized policies into one model by distilling from multiple teachers using trajectories generated by the student’s policy. "Domain- and effort-specialized policies are consolidated into a unified model through multi-teacher on-policy distillation~#1{lu2025onpolicydistillation, xiao2026mimov2flash, deepseekv4}."
- No Position Encoding (NoPE): An attention setup that omits explicit positional encodings in certain layers, relying on other mechanisms for order sensitivity. "Kimi K#1{3} follows the hybrid design of Kimi Linear~#1{team2025kimi} and applies No Position Encoding (NoPE) to all MLA layers."
- Online softmax: A streaming algorithm for computing softmax that reduces memory and enables partial-result merging at inference time. "enabling the parallel inter-block results to be better merged with the sequential intra-block partial sums via online softmax~#1{milakov2018online}, significantly reducing inference time cost."
- Per-Head Muon: A per-head optimization/normalization component included to improve stability or efficiency of attention heads. "Together with Per-Head Muon (\S\ref{sec:perhead_muon}), these components provide a unified architecture for scaling information flow across tokens, layers, and channels."
- Persistent rollout: Keeping rollouts and environment state alive across long interactions to support million-token trajectories. "million-token agentic RL with persistent rollout and sandbox states"
- Quantile Balancing (QB): A routing-bias update rule that sets expert biases from score quantiles to hit target loads without auxiliary losses. "we introduce Quantile Balancing (QB), which sets each expert bias from the router-score quantile that matches its target load~#1{su2026qb}."
- RoPE frequency base: The base parameter of Rotary Positional Embeddings that may need retuning when extending context length. "This separation also avoids modifying positional-encoding parameters when extending the context length, such as retuning a RoPE frequency base or applying YaRN~#1{peng2023yarn}."
- ShortConv: A short convolutional projection used in forming attention queries, keys, and values. "The query, key, and value projections apply followed by ~#1{yang-2025-gdn},"
- Sigmoid Tanh Unit GLU (SiTU-GLU): A bounded GLU variant that uses tanh soft-caps in both gate and up branches to prevent activation explosions while preserving local linearity. "To satisfy these requirements, we propose Sigmoid Tanh Unit GLU (SiTU-GLU)."
- Softcap: A smooth bounding function defined as beta times tanh of x over beta, used to cap activations. "SiTU-GLU applies the smooth cap to the linear factor of the Swish gate and independently to the up branch:"
- State-aware prefix caching: Reusing cached computation prefixes with awareness of request/model state to accelerate long-context processing. "we develop fused kernels, KDA Context Parallelism, and state-aware prefix caching to make KDA efficient within devices, across devices, and across requests."
- Tensor Cores: Specialized matrix-multiplication units on modern GPUs used to accelerate dense tile computations. "the bounded range in Kimi K#1{3} allows all causal tiles to use dense Tensor Core matrix multiplications."
- Test-time scaling: Increasing computation during inference (e.g., more reasoning or tool use) to boost capability without changing model parameters. "These advances have made test-time scaling a central focus of frontier research."
- Top-k routing: Selecting the k highest-scoring experts per token in an MoE router, often with bias adjustments for load balancing. "Routing replaces the Top- selection with Top- on the biased score "
- UT transform: A transformation in KDA’s chunkwise formulation that produces auxiliary matrices used for efficient intra-chunk computation. "The UT transform produces and , from which we define the pseudo-value term $\widetilde{\mathbf{V}_{[t]}:=\mathbf{U}_{[t]}-\mathbf{W}_{[t]}\mathbf{S}_{[t]}$."
- YaRN: A method to extend context length for rotary embeddings without retraining positional parameters. "such as retuning a RoPE frequency base or applying YaRN~#1{peng2023yarn}."
Collections
Sign up for free to add this paper to one or more collections.