An open-source, production-ready system for serving machine-learning models at scale.
vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs), built to deliver state-of-the-art performance on GPUs with features such as PagedAttention and continuous batching.
Open-source high-performance framework and DSL for serving large language & vision-language models with low-latency, controllable, structured generation.
A lightweight open-source platform for running, managing, and integrating large language models locally via a simple CLI and REST API.
NVIDIA TensorRT is an SDK and tool-suite that compiles and optimizes trained neural-network models for ultra-fast, low-latency inference on NVIDIA GPUs.
