Apple and Qwen 397B: Local AI on MacBook Pro M3 Max
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisis ton rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
A notable advancement has been made by Dan Woods, who successfully ran a customized version of the model Qwen3.5-397B-A17B on a MacBook Pro M3 Max equipped with 48 GB of RAM. This model, although large at 209 GB (with 120 GB quantized), was able to operate at a speed of 5.5+ tokens per second thanks to the use of innovative techniques.
The Qwen3.5-397B-A17B model is based on a Mixture-of-Experts (MoE) architecture. This means that each token interacts only with a portion of the weights of the overall model, allowing these weights to be loaded from an SSD instead of storing them all in RAM. This approach optimizes memory usage and enables larger models to run on machines with limited resources.
Dan Woods applied techniques described in Apple's 2023 paper titled LLM in a flash: Efficient Large Language Model Inference with Limited Memory. This paper suggests storing model parameters in flash memory and transferring them to DRAM on demand. The goal is to reduce the volume of data transferred and optimize the reading of data in large and contiguous amounts.
To successfully carry out his project, Dan used Claude Code and a variant of Andrej Karpathy's autoregressive model. This allowed Claude to conduct 90 experiments, producing code in MLX Objective-C and Metal for efficient model execution. The code and results of this experiment are available in the danveloper/flash-moe repository, which also contains a PDF document primarily authored by Claude Opus 4.6 detailing the entire experiment.
The final model uses quantized experts at 2 bits, but certain parts, such as the embedding table and routing matrices, retain their original precision, totaling 5.5 GB in memory during execution. While Qwen 3.5 typically uses 10 experts per token, this configuration reduced that number to 4, noting that the most significant drop in quality occurred at 3 experts per token. According to Claude, the output quality at 2 bits is comparable to that at 4 bits, although the details of the evaluations remain limited.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.