vLLM

A high-throughput and memory-efficient inference and serving engine for LLMs

A framework for efficient model inference with omni-modality models

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

TPU inference for vLLM, with unified JAX and PyTorch support.

Community maintained hardware plugin for vLLM on Apple Silicon

vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

Community maintained hardware plugin for vLLM on Ascend

System Level Intelligent Router for Mixture-of-Models at Cloud, Data Center and Edge

A high-performance and light-weight router for vLLM large scale deployment

Cost-efficient and pluggable Infrastructure components for GenAI inference

The vLLM XPU kernels for Intel GPU

vLLM Daily Summarization of Merged PRs

Community maintained hardware plugin for vLLM on Intel Gaudi

Community maintained hardware plugin for vLLM on Spyre

A safetensors extension to efficiently store sparse quantized tensors on disk

Fast and memory-efficient exact attention

Community maintained hardware plugin for vLLM on AWS Neuron

Agent skills for vLLM

Common recipes to run vLLM

This repo hosts code for vLLM CI & Performance Benchmark infrastructure.

vLLM performance dashboard

Performance dashboard for vLLM

vLLM Model plugin for the encoder-decoder BART model

vLLM Logo Assets