An open-source engine runs a 125B model on a 12GB gaming card
Strata, an MIT-licensed inference engine, published measured numbers for running Qwen3.8-Flash-Next — a 125-billion-parameter mixture-of-experts model with roughly 6 billion parameters active per token — on an ordinary gaming PC. On an RTX 5070 with 12GB of VRAM it writes answers at 94 tokens per second in the Q2_0 quantization and reads a 32K-token prompt at 2,650 tokens per second; an RX 9070 XT manages 60 and 1,160.
It asks for a 12GB-or-better GPU, 32GB of RAM and about 80GB of disk, and its installer stands up an OpenAI-compatible API on localhost so coding agents can point at it. A separate Coder build with half the experts removed reaches 91% of the full model’s SWE-bench Verified score inside 32GB of RAM. The repository has drawn more than 11,000 stars since 24 September, and its Hacker News thread reached the front page on Sunday.
WHY IT MATTERS
The headline says a 4090 at 100 tokens per second; the artifact’s own table says an RTX 5070 12GB at 94, and the repo’s estimate for a 24GB 3090 is 100 to 140 — the measured number is the one to trust. What it changes is the floor: a model that would normally want a datacenter now runs on a card that costs less than a month of frontier API spend, with nothing leaving the machine. For anyone wiring a coding agent, the localhost endpoint means the swap is a base URL, which is the same practical case for open weights that the data-provenance stories keep making from the other direction. The caveat deserves one line: these are the project’s own measurements on its own settings, and the low-bit quantizations that fit a 12GB card trade quality for size.