We built this project to save LLM from repetitive compute and increase throughput by 3x. Now it has been adopted by IBM in their LLM serving stack!

This is an automated archive made by the Lemmit Bot.

The original was posted on /r/opensource by /u/Nice-Comfortable-650 on 2025-06-19 00:25:32+00:00.

---

Hi guys, our team has built this open source project, LMCache, to reduce repetitive computation in LLM inference and make systems serve more people (3x more throughput in chat applications) and it has been used in IBM’s open source LLM inference stack.

In LLM serving, the input is computed into intermediate states called KV cache to further provide answers. These data are relatively large (~1-2GB for long context) and are often evicted when GPU memory is not enough. In these cases, when users ask a follow up question, the software needs to recompute for the same KV Cache. LMCache is designed to combat that by efficiently offloading and loading these KV cache to and from DRAM and disk.

Ask us anything!

Github: github.com/LMCache/LMCache