Decoding LLM Names: Size, MoE, Quantization, GGUF

Choosing a local LLM often involves a complex file name. These labels condense precise information: parameter size, dense structure or MoE, actual active amount during inference, quantization level and scheme, not to mention the file format. Here’s an overview of the segments that guide required memory and use cases.

The GGUF format signals the model's packaging and structure

GGUF is a model file format that tells the software how an LLM is packaged and stored. A name like Qwen3-30B-A3B-Instruct-2507-q2ks-mixed-AutoRound-gguf gathers several pieces of information: Qwen3 refers to the model, 30B indicates the number of parameters, A3B denotes the number of active parameters per token, Instruct signifies the type of fine-tuning, 2507 is an identifier or a dated version, gguf is the file format, q2ks is the quantization scheme, mixed indicates that not all layers use the same bit width, and AutoRound refers to the quantization algorithm.

Quantization: from Q3 to Q8 to reduce memory, with a quality cost

To reduce the size of a model locally, it is common to compress the weights through quantization. The main levels used are Q8 (approximately 8 bits), Q6 (approximately 6 bits), Q5 (approximately 5 bits), Q4 (approximately 4 bits), and Q3 (approximately 3 bits): the lower the value, the more compact the model becomes. The precise size varies according to the quantization methods, which incorporate metadata and do not always adhere to the announced number of bits per value. Low-bit quantization reduces the required memory, usually at the expense of quality. Labels like Q4_K_M denote a specific 4-bit quantization scheme, while variants like q2ks exist. Comparing Q4_K_M to Q6_K is akin to comparing different levels and quantization schemes. Modern methods use various groupings, scales, and precisions to balance size and quality.

Raw memory: FP16, BF16, and the magnitude of weights

FP16 and BF16 each store values on 16 bits, but according to different representations. For example, a model with 35 billion parameters in 16 bits requires about 35B × 16 bits, which is nearly 70 GB, solely for the weights. This size exceeds the capabilities of many consumer machines, hence the interest in quantization.

The number of parameters and structure affect required memory

The suffixes 7B, 14B, 35B, or 70B indicate the total number of parameters, with the letter B signifying billion. Parameters are the learned values that make up the model, and locally, the more there are, the more memory is required. Proprietary models like Gemini 3 Pro or Claude Opus 5 can reach counts in the trillions. Dense models use nearly all of their parameters during inference, while Mixture-of-Experts (MoE) models distribute a large set of weights among experts activated on demand through a routing mechanism. Thus, an MoE can display a large total of parameters without mobilizing them simultaneously for each token.

A3B: total billions, but how many are actually active per token

In a name like Qwen3.5-35B-A3B, 35B indicates the total number of parameters and A3B provides an estimate of the active parameters per token, here approximately 3B. The letter A refers to activation. This functionality allows an MoE to limit computation during inference despite a high total volume. It is important to note that A3B does not mean the model only has 3B parameters: it retains a total of 35B.

The Instruct version allows the model to follow instructions

The same family of models can exist in Base and Instruct versions, such as Qwen3.5-35B-A3B-Base and Qwen3.5-35B-A3B-Instruct. The Base version is pre-trained, not fine-tuned to follow instructions, while the Instruct version has been refined through instruction tuning to better respond to commands, questions, and conversational uses. They can share architecture, size, and quantization while behaving differently. The names may also indicate the presence of FP16 or BF16. Reading these names from left to right is akin to going through a condensed technical sheet: knowing whether the model is Base or Instruct is one of the key elements to consider. Understanding these segments facilitates the choice of a local model.