Nate@nate_512

フォロー

Kimi K3 on a single CPU 8 GB RAM

Ok so someone actually got a 2.78T parameter model running on a single cpu with just 8 GB of RAM. it's Kimi K3, a mixture-of-experts thing with 896 experts per layer but only 16 fire per token. the project is kimi-k3-in-c, all in portable C99, zero external dependencies, no gpu no framework. the trick is 93% of the model lives on NVMe disk and gets streamed in when needed, weights are stored and multiplied in 4-bit, and the dense trunk processes one layer at a time. the whole engine is 176 KB of C code. it's painfully slow, like 32 seconds per token at 8 GB, and you need 1.7 TB of free disk space, but it produces byte-identical output whether you have 8 GB or 224 GB of RAM; more memory just makes it faster. 100% free and open-source under Apache-2.0, runs on Linux x86-64. peak RSS measured 8.24 GB, checkpoint on disk is 1.56 TB. this is the kind of mad science that makes me want to dig through code i barely understand

1mo

Nate@nate_512

フォロー

Kimi K3 on a single CPU 8 GB RAM

Ok so someone actually got a 2.78T parameter model running on a single cpu with just 8 GB of RAM. it's Kimi K3, a mixture-of-experts thing with 896 experts per layer but only 16 fire per token. the project is kimi-k3-in-c, all in portable C99, zero external dependencies, no gpu no framework. the trick is 93% of the model lives on NVMe disk and gets streamed in when needed, weights are stored and multiplied in 4-bit, and the dense trunk processes one layer at a time. the whole engine is 176 KB of C code. it's painfully slow, like 32 seconds per token at 8 GB, and you need 1.7 TB of free disk space, but it produces byte-identical output whether you have 8 GB or 224 GB of RAM; more memory just makes it faster. 100% free and open-source under Apache-2.0, runs on Linux x86-64. peak RSS measured 8.24 GB, checkpoint on disk is 1.56 TB. this is the kind of mad science that makes me want to dig through code i barely understand

1mo