Kimi K3 from Moonshot AI: 2.8 Trillion Parameters Unveiled

An Impressive Advancement in the Kimi Series

The latest model from Moonshot AI, the Kimi K3, stands out with its impressive size, reaching 2.8 trillion parameters. This figure is nearly three times the size of its predecessor, the Kimi K2. With 104 billion activated parameters and a context window of 1 million tokens, the Kimi K3 also features native vision. These characteristics demonstrate a 2.5 times greater scaling efficiency than the Kimi K2, underscoring the ongoing relevance of the scaling law. However, the technical report emphasizes that understanding the internal workings of these models is crucial for fully grasping their potential.

In this article, we will explore four architectural innovations that make the scale of the Kimi K3 possible:

• Core attention architectures: featuring Kimi Delta Attention (KDA) and Gated MLA with No Position Encoding (NoPE) attention residuals.
• Latent MoE with quantile-based load balancing.
• A vision tower that operates without contrastive pre-training.

We will conclude with a discussion on co-designing infrastructure, a trend initiated with DeepSeek V3.

Core Attention Architectures

The foundation of the Kimi K3's attention consists of three KDA layers and one Gated MLA layer, each followed by a stable LatentMoE layer.

Kimi Delta Attention (KDA)

Unlike traditional models that use multi-head attention, the Kimi K3 adopts Kimi Delta Attention (KDA) for three of its four attention layers. This mechanism, introduced in Kimi Linear (2025), is derived from Gated DeltaNet. DeltaNet proposes a delta rule, based on the generalized Householder transformation, to enhance the performance of linear attention models. To optimize computational efficiency, the KDA can be reformulated into a piecewise parallel format, typical of linear attention computation. Additionally, a negative-softplus mapping is used to prevent precision overflows by limiting the declining logits.

Gated MLA & No Position Encoding (NoPE)

The Multi-head Latent Attention (MLA), introduced by DeepSeek-V2, aims to reduce the size of the KV cache by compressing keys and values into a shared low-rank latent space. Unlike traditional MLA that uses RoPE for positional encoding, the Kimi K3 opts for NoPE (No Position Encoding). This approach is justified by the fact that KDA layers already provide a position-sensitive mixture through their recurrent decay, allowing MLA layers to focus on their primary functions.

Attention Residuals: A Sequential Approach to Depth

The Kimi K3 technical report proposes an innovative concept: treating attention residuals similarly to the sequential dependencies of transformers. This involves calculating the residual pairwise across layers. With a model depth of less than 100, the total computational cost remains O(L²d), which is manageable and can be optimized through block-wise computation.

Stable LatentMoE: Unprecedented Scaling

The Kimi K3 pushes the MoE architecture to new heights with 896 routed experts and 16 activated per token, compared to the 384 routed / 8 active of the Kimi K2. This 133% jump in sparsity requires architectural adjustments. The LatentMoE allows shared experts to operate across the full hidden width of the model, while routed experts function in a more constrained latent space. This reduces the cost of the router while enabling the economical activation of 16 experts per token.

A Vision Tower Without Contrastive Pre-training

One of the most surprising findings from the report concerns the vision encoder of Kimi K2.5. Unlike other multimodal LLM vision towers, this one was not initialized through contrastive pre-training. This innovative approach marks a break from traditional methods, opening new avenues for future vision architectures.