Supercharge Your LLM with the Fastest KV Cache Layer

A Golang implemented Redis Server and Cluster. Go 语言实现的 Redis 服务器和分布式集群

Unified KV Cache Compression Methods for Auto-Regressive Models

LLM KV cache compression made easy

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025)

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

[ICLR'26] The official code implementation for "Cache-to-Cache: Direct Semantic Communication Between Large Language Models"

[Survey] Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of HierarchicalKV is to store key-value feature-embeddings on high-bandwidth memory (HBM) of GPUs and in host memory. It also can be used as a generic key-value storage.

Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)

[NeurIPS'25] KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems

📚 A curated list of Awesome Efficient dLLMs Papers with Codes

Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.

Completion After Prompt Probability. Make your LLM make a choice

This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT) variant. The implementation focuses on the model architecture and the inference process. The code is restructured and heavily commented to facilitate easy understanding of the key parts of the architecture.

Notes about LLaMA 2 model

Easy control for Key-Value Constrained Generative LLM Inference(https://arxiv.org/abs/2402.06262)

KV Cache & LoRA for minGPT

PiKV: KV Cache Management System for Mixture of Experts [Efficient ML System]

KV Cache Steering for Inducing Reasoning in Small Language Models

A high-performance RDMA distributed file system for fast LLM Inference and GPU Training

High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference

Cross-GPU KV Cache Marketplace

Implementation of Flash-DLM (paper: FlashDLM: Accelerating Diffusion Language Models via Efficient KV Caching and Guided Diffusion). Provides training-free methods to accelerate diffusion language model inference.

MemCloud is a distributed in-memory data store written in Rust. It allows nodes (such as macOS, Windows and Linux machines) on a local network to pool their RAM, creating a shared, ephemeral storage cloud.

[NeurIPS'25] KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems