How the memory model works
Offloading MoE experts is not a new idea, and you should be skeptical of anyone
who presents it as one. llama.cpp has shipped --cpu-moe and -ot tensor
overrides for a long time, and expert caching with speculative prefetch was
published in 2023.
lean is not different because it pages experts. It is different in where the experts live and where they are computed.
What --cpu-moe actually does
Section titled “What --cpu-moe actually does”It does not stream. It places the expert tensors in system RAM and computes them on the CPU. The router picks experts per token, the CPU multiplies them, and the results go back to the GPU.
That has two consequences:
- You need as much RAM as the model is large. The weights are resident, not streamed. A 75 GB model wants roughly 75 GB of RAM.
- Expert throughput is bounded by CPU FLOPs and memory bandwidth, not by your GPU.
It works, it is free, and on a machine with enough RAM it is a perfectly reasonable way to run these models.
What lean does instead
Section titled “What lean does instead”Experts are stored as individual files inside the .lmpack directory, one file
per expert. At inference time the router picks the active experts for a token,
and lean copies just those onto the GPU, where they are computed. A per-GPU LRU
cache keeps the hot set resident, and a prefetcher reads the routing profile to
pull likely-needed experts several layers ahead of where the model currently is.
The file-per-expert layout is what makes this work rather than being a packaging detail. In a single monolithic file, fetching one expert means seeking into a 73 GB blob, and the kernel’s page cache pulls in neighbouring data you will never use. One file per expert makes each load a discrete, page-aligned read the OS caches cleanly, so the resident set converges on the experts you actually route to.
Measured
Section titled “Measured”Both engines serving Qwen3.5-122B-A10B Q4_K_M, 75 GB of weights, on the same machine: 2x RTX 3090, 62 GB RAM, weights on NVMe.
| System RAM used | Experts compute on | |
|---|---|---|
llama.cpp --cpu-moe |
58 GB, plus 5 GB of swap | CPU |
| lean | 7 GB | GPU |
On this machine llama.cpp can just barely hold the model, and pays for it in swap. On a 32 GB machine it cannot load it at all.
The gap widens with model size, because the RAM requirement tracks the weights:
| model | weights | RAM needed by --cpu-moe |
|---|---|---|
| lean-agent-middle | 75 GB | ~75 GB |
| lean-agent-cruiser | 155 GB | ~155 GB |
| lean-reason-heavy | 226 GB | ~256 GB |
| lean-think-heavy | 242 GB | ~256 GB |
lean runs the largest of those on 48 GB of VRAM and an NVMe drive.
Cache behaviour
Section titled “Cache behaviour”Because only a fraction of the model is resident, the useful number is how often a routed expert is already on the GPU:
- 35B: 93% of expert lookups hit VRAM and need no transfer.
- 122B: 79% hit rate with roughly a third of the pack resident across two GPUs.
Routing is stable enough that a profile generalises across quantizations. When the 122B source weights were replaced with a different quantization, every one of the 12,273 expert files changed, and the profile built on the old weights still predicted routing on the new ones to within 0.4%.
What we are not claiming
Section titled “What we are not claiming”lean is not faster than llama.cpp on hardware that can hold the model in RAM. On that hardware throughput is comparable, llama.cpp is free, and you should use it.
The claim is narrower: model size stops being bounded by your system RAM. If the model fits your RAM, you have options. If it does not, you have fewer.