DeepSeek V4.1: 763 Billion Parameters and Reduced Memory

DeepSeek presents V4.1 Flash, a model with 763 billion parameters designed to reduce costs and latency. The architecture relies on offloaded N-gram weights and lightweight KV caches. Google and Alibaba are exploring similar avenues, signaling a broader movement towards less resource-intensive LLMs.

N-grams in vogue and the promise of infrastructure streamlining

Google is already applying a nearby technique, Per-Layer Embedding (PLE), to its Gemma models to shift some weights to local storage, primarily in smaller models. At the end of August, Alibaba unveiled Qwen 3.8-Flash-Next, a model with 180 billion parameters, of which 51 billion are N-grams, inspired by research published by DeepSeek in January. Alibaba indicates that this architecture will serve as the foundation for its next-generation Qwen 4. These advancements suggest that N-grams could become a regular feature in the design of large language models, aimed at reducing overhead and the proliferation of servers, which are costly to manufacture and maintain. However, in a production environment, the memory required remains significant due to KV caches, whose capacity varies based on context length and the number of users connected simultaneously.

N-gram mechanism: consulted tables and offloaded weights

In V4.1, 196 billion of the 763 billion parameters are N-grams, forming a conditional memory module intended to separate memory and computation. These groups of tokens serve as implicit knowledge and enhance predictions by providing information through inexpensive lookups. They function like large lookup tables and do not need to be fully loaded into the GPU for each token generation: just a few dozen queries are sufficient. This feature allows them to be stored in system memory or on fast storage. Theoretically, weights in FP8 require 763 GB of GPU memory, but offloading the N-grams reduces this need to about 567 GB.

Revised KV caches: 13% to 25% of the previous footprint

DeepSeek has rethought how to manage key-value caches, which represent a significant memory consumption for applications such as conversational agents. The use of these caches now accounts for between 13% and 25% of the amount consumed by V4 Flash, a reduction achieved through modifications in attention systems and the addition of a new causal encoder-decoder module. For the same KV footprint, the model can support four to eight times the number of users. This improvement explains why memory requirements do not increase proportionally with model size.

An oversized model in the lineage of V3 and R1

With 763 billion parameters, V4.1 Flash is over 2.5 times larger than the previous model and surpasses the V3 and R1 models, which marked the emergence of DeepSeek in early 2025. Unveiled on a Thursday, V4.1 is positioned by DeepSeek as a cost and latency reduction-focused update. Despite its version number, the company highlights significant architectural changes and presents this evolution as a step towards more efficient models with contained resources.