Pruning LLMs: Gains Measured by Ising Optimization

Researchers propose selecting blocks to be removed in LLMs by solving a binary optimization problem inspired by Ising glasses. The system's energy serves as a proxy for quality, allowing for efficient exploration of numerous configurations using classical or quantum-inspired solvers. Gains have been reported on Llama models and benchmarks such as MMLU, AIME25, and GPQA.

Up to 23 Points on MMLU at 50% Compression of Llama-3.3-70B

At 50% compression of Llama-3.3-70B-Instruct, the method reports nearly a 23 percentage point advantage on MMLU compared to the best competing block pruning method. The low-energy states resulting from this formulation correspond to high-performing pruned models, aligning energy minimization with the pursuit of better scores. However, energy remains an imperfect proxy, making it useful to examine not only the ground state but also low-energy excited states. Once the Hamiltonian is established, generating this spectrum of candidates is inexpensive, and having multiple options close to the minimum meets practical evaluation needs.

Rapid Exploration: Brute Force, Tabu Search, and QUBO

Calculating the energy of a configuration is inexpensive, allowing for brute-force evaluation of up to tens of billions of combinations on a single GPU. Processing several million configurations takes only a few seconds; the most demanding example mentioned involves removing 8 blocks out of 80 in Llama-3. For Llama-3-70B, which represents nearly 29 billion configurations, the process took about two days. When this threshold is exceeded, the constraint is incorporated as a penalty term to form a QUBO, which is then solved using methods like quantum annealing, QAOA, tabu search, or specialized branch-and-bound algorithms. An open-source tabu solver regularly achieves low-energy states in a few seconds, including on cases verified by brute force, enabling the tackling of models where enumeration is out of reach. The goal is not to aim for the exact ground state but to quickly generate a few good states, which is compatible with lightweight solvers and multiple executions. These solvers are used at Multiverse.

An Example Contradicts the Idea of a Single Pruned Segment at the End of the Network

On Llama-3.1-8B-Instruct with 16 removals out of 32 blocks, most of the best proposed configurations cut blocks towards the end of the model, as expected from previous work. The 17th excited state is the first to suggest removing a block near the beginning; after slight retraining, this configuration surpasses the ground state on several benchmarks. This result contradicts the common hypothesis that the best pruning consists of a consecutive block in the middle or end of the network.

Transfer to Heterogeneous Architectures Without Retraining

Block removal becomes more complex when different types of layers are interwoven, but the Ising formulation treats all couplings in the same way, regardless of block type. Applied to the NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, which combines Mamba2, attention, and MoE in a non-uniform pattern, the method was tested without any retraining. By removing 2 to 3 MoE layers or 2 attention layers, the constrained binary optimization finds configurations that outperform the block influence heuristic on AIME25 and GPQA. These results indicate that redundancy in such heterogeneous models is effectively managed by this approach.

From Mean Field to Interactions: Why Model Couplings

Many existing approaches rank blocks one by one using heuristics of magnitude, sensitivity, or influence, often imposing consecutive removal. These techniques assume independent contributions, akin to mean field, while couplings arise between decisions, such as between blocks 19, 20, and 24. As networks gain depth and heterogeneity, ignoring these interactions degrades quality, especially when many blocks are removed simultaneously. As the number of combinations grows exponentially, tools from statistical physics become relevant.

Formulating Pruning as an Ising Energy Minimization

Each block receives a binary variable, 0 for keeping and 1 for removing. A second-order Taylor expansion of the loss with respect to these variables provides an approximate Hessian, whose diagonal informs individual importance and whose off-diagonal terms encode pairwise couplings. The goal becomes to select exactly M blocks from N to minimize xᵀH⁰x, which is a constrained binary optimization equivalent to an Ising glass with fixed magnetization, where the constraint on M plays the role of fixed total spin. Low-energy states correspond to high-performing models on benchmarks. This approach combines with techniques like quantization or low-rank compression, and by reducing the number of blocks, it enables predictable inference speed gains in addition to memory savings.