Benchmarks

Offloading performance and model quality, measured on real hardware.

Offloading Performance

Token generation rates with full expert offloading, measured on a single RTX 3090 (24 GB) and on 2x RTX 3090 (48 GB) — 64 GB RAM, NVMe SSD.

Model	Quant	Hardware	Prefill	Decode
lean-agent-35b	Q4_K_M	1× RTX 3090	14-31 tok/s	19-31 tok/s
lean-agent-35b	Q4_K_M	2× RTX 3090	10-23 tok/s	15-27 tok/s
lean-coder-80b	Q4_K_XL	1× RTX 3090	12-23 tok/s	14-20 tok/s
lean-coder-80b	Q4_K_XL	2× RTX 3090	8-17 tok/s	10-18 tok/s
lean-agent-122b	Q4_K_M	1× RTX 3090 (24 GB)	6-11 tok/s	5-6 tok/s
lean-agent-122b	Q4_K_M	2× RTX 3090	4-8 tok/s	6-8 tok/s
lean-reason-397b	Q4_K_M	Coming soon
lean-think-398b	Q4_K_M	Coming soon

Ranges span decode at 64 and 256 generated tokens (prefill at short prompts, cold cache). Single-GPU uses the x16 card; 2-GPU uses pipeline parallelism. Profile-guided preloading and speculative router prefetch throughout. Smaller models run fastest on a single card — the engine defaults to single-GPU when the model fits — while the 122B is faster across two. The 122B loads on one 24 GB card (17 GB free after core weights, 1,512 hot experts preloaded). Measured on trunk, 2026-07-05.

Engine Features

Performance infrastructure built into the runtime.

93%

VRAM cache hit rate (35B)

LRU cache with profile-guided preloading

66-74%

Speculative prefetch hit rate

Router predicts next-layer experts ahead of computation

1-6 GB/s

Expert preload throughput

Async I/O via background thread pool

2-5s

Model load time

Core weights into VRAM, experts lazy-loaded via mmap

Bit-identical

Output vs llama.cpp

Cross-validated on 10 diverse prompts, same GGUF weights

Multi-GPU

Pipeline parallelism

Layers split across GPUs, output bit-identical to single-GPU

Hardware Reference Configurations

All benchmarks run on local hardware. No cloud GPUs.

Tier	VRAM	RAM	NVMe	Target Models
Minimal	12 GB	16 GB	1.8 TB	lean-agent-35b, lean-coder-80b
Prosumer	24 GB	32 GB	1.8 TB	lean-agent-122b
Enthusiast	48 GB	64 GB	1.8 TB	lean-reason-397b, lean-think-398b

Q8_0

423.7 GB

Model Quality Benchmarks

Standard evals to confirm no capability regression from the lmpack pipeline.

Results coming soon.

MMLU, HumanEval, BFCL, IFEval, GSM8K, MATH - run via lm-evaluation-harness against lean serve.

Methodology

Offloading benchmarks measure tok/s, VRAM cache hit rate, prefetch hit rate, and expert preload throughput. All results from lean bench.

Cross-validation compares lean-engine output against llama.cpp on the same GGUF weights with greedy decoding. 4/10 prompts match token-for-token; 6/10 diverge only in freeform thinking text due to expected FP precision differences.

Quality benchmarks will be run via lm-evaluation-harness against the lean serve OpenAI-compatible API. Models must match base model scores before release.

Hardware: All benchmarks run locally on reference configurations. No cloud GPUs. Results are reproducible.