Running Kimi K3 on macOS: Challenges and Breakthroughs
In the realm of large language models, Kimi K3 has garnered attention for its impressive size—launching at an enormous 1.56 terabytes. Recently, I embarked on an experiment to run Kimi K3 locally on my MacBook, a device not typically designed for such heavy workloads. While the experience underscored some significant hurdles, it also yielded valuable insights and notable performance improvements.
Initial Setup and Challenges
Kimi K3’s sheer size posed immediate practical issues. My M1 MacBook, equipped with 64GB of RAM, struggled with storage constraints. The full model requires a substantial disk footprint, and actively downloading the entire model proved impractical; instead, I opted to stream the model components directly from the cloud. This approach, while operational, resulted in slow interaction speeds—about one minute per token once the system warmed up.
The model’s architecture employs a mixture-of-experts paradigm, with approximately 114GB of non-expert weights stored on disk in 8-bit integer format. During inference, for each token, the system dynamically selects 16 “experts” out of 896 per layer. These selected experts are fetched from repositories like HuggingFace via single-range HTTP requests and cached locally to optimize subsequent access. Over time, this caching strategy helps reduce latency by leveraging the most frequently accessed model components.
Performance Optimization Attempts
Understanding that fetch times and compute bottlenecks heavily influence performance, I took several steps to improve throughput:
-
Reducing Download Latency: Initially, prefetching the entire 1.45TB of expert parameters to local storage was a significant step. After this “pre-fill,” the token generation speed improved dramatically—from approximately 2,429 seconds to just 40 seconds for generating a conversation—highlighting the impact of local data availability.
-
Profiling Bottlenecks: Expectations pointed towards matrix multiplications within the model’s experts as the primary bottleneck. To this end, I developed a custom Metal kernel optimized for Apple silicon, achieving a 9.5-fold speed-up over CPU-based tensor operations. However, performance profiling revealed that these matrix operations only accounted for about 6% of the total token inference time, indicating other factors were at play.
-
Addressing Disk I/O: The most significant discovered limitation was the manner in which expert weights were being accessed. Initially, I used NumPy’s
np.memmapto handle expert weights, which led to demand paging—fetching data page-by-page during computation. This resulted in a throughput of approximately 0.87 GB/s, vastly inferior to the disk’s potential, where threaded pread operations could reach about 6.85 GB/s. Optimizing disk I/O by switching to threaded pread and disabling OS-level caching boosted effective data throughput, substantially reducing inference times.
Achievements and Future Directions
Through these combined efforts, I managed to bring the inference time down to approximately 16 seconds per token on my MacBook—a significant improvement from the initial one-minute per token rate. While still not real-time, this progress demonstrates the potential for running large models locally given appropriate optimization.
Additionally, I set up an OpenAI-compatible server interface, allowing interaction through standard chat interfaces—a helpful feature for testing and deployment.
Looking Ahead
These experiments, conducted on an M1 Max with 64GB of RAM—arguably the slowest hardware among potential testers—highlight the scalability of such efforts. I am eager to see how newer hardware with greater memory capacity (such as M3, M4, or M5 chips with 128GB RAM) could further accelerate these processes.
Final Notes
Running Kimi K3 locally is a complex but rewarding endeavor. It requires a nuanced understanding of both hardware and software optimization, particularly in managing disk I/O and memory usage. The recent breakthroughs affirm that, with careful engineering, even large language models can be made more accessible for local experimentation.
For those interested in the technical details or seeking to replicate this setup, I’ve shared the project on GitHub: https://github.com/gavamedia/deltafin.
Disclaimer: Performance may vary significantly depending on hardware configuration. My setup is an M1 Max with 64GB of RAM, and results could be different on other machines.
