• •A developer successfully ran the Gemma-4 26B model on a CPU without a GPU, reaching 40 tok/s single-stream and 124 tok/s batched.
• •The process utilized selective compression, identifying that compressing the output head provides greater efficiency than compressing model experts.
• •The practical ceiling for this technique on consumer-grade hardware and the specific impact on model precision remain unverified.

A developer successfully demonstrated running a 26B parameter model on a CPU at 124 tokens per second, according to reports on Hacker News. While previous optimization efforts typically focused on compressing model experts, this test revealed that compressing the output head—which accounts for 32% of per-token bytes—yields significantly better performance. Though the results show promise for high-speed inference on standard hardware, the trade-off in output quality and the hardware-specific constraints needed to achieve these speeds are not yet fully documented. Achieving such throughput on CPU hardware could lower the barrier to entry for local AI deployment, provided these efficiency gains scale across diverse processor architectures.