Tag

vLLM is a high-throughput, memory-efficient inference and serving engine for large language models (LLMs), built to deliver state-of-the-art performance on GPUs with features such as PagedAttention and continuous batching.

KTransformers is a flexible framework for experiencing cutting-edge optimizations in LLM inference and fine-tuning, focusing on CPU-GPU heterogeneous computing. It consists of two core modules: kt-kernel for high-performance inference kernels and kt-sft for fine-tuning. The project supports various hardware and models like DeepSeek series, Kimi-K2, achieving significant resource savings and speedups, such as reducing GPU memory for a 671B model to 70GB and up to 28x acceleration.

SGLang is a high-performance serving framework for large language models (LLMs) and vision-language models, designed for low-latency and high-throughput inference across single GPUs to large distributed clusters. Key features include RadixAttention for prefix caching, zero-overhead batch scheduling, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, and quantization (FP4/FP8/INT4/AWQ/GPTQ). It supports a wide range of models like Llama, Qwen, DeepSeek, and hardware from NVIDIA, AMD, Intel, TPUs, with an intuitive frontend for LLM applications.

A lightweight open-source platform for running, managing, and integrating large language models locally via a simple CLI and REST API.

An open-source, production-ready system for serving machine-learning models at scale.

NVIDIA TensorRT is an SDK and tool-suite that compiles and optimizes trained neural-network models for ultra-fast, low-latency inference on NVIDIA GPUs.

ONNX (Open Neural Network Exchange) is an open ecosystem that provides an open source format for AI models, including deep learning and traditional ML. It defines an extensible computation graph model, built-in operators, and standard data types, focusing on inferencing capabilities. Widely supported across frameworks and hardware, it enables interoperability and accelerates AI innovation.

PaddleOCR is an industry-leading, production-ready OCR and document AI engine developed by the PaddlePaddle team. It supports over 100 languages and converts PDFs or image documents into structured AI-friendly data (e.g., JSON and Markdown), bridging the gap between images/PDFs and LLMs. Key features include multilingual support, high accuracy, handwriting recognition, and advanced document parsing for elements like tables, formulas, and charts, with end-to-end tools for training, inference, and deployment.

ExecuTorch is PyTorch’s unified on-device AI deployment solution for mobile, embedded, and edge devices. It enables direct export from PyTorch, ahead-of-time compilation, quantization and hardware partitioning to produce compact runtime programs (.pte) that run across many backends (XNNPACK, Vulkan, CoreML, Qualcomm, etc.). It supports LLMs, vision, speech and multimodal models with a small runtime footprint and production tools for profiling, memory planning, and selective operator builds.

FunASR is an open-source end-to-end speech recognition toolkit (ASR) led by Alibaba DAMO Academy. It supports ASR, voice activity detection (VAD), punctuation restoration, speaker verification/diarization, multi-talker ASR, emotion recognition and more. FunASR provides many industrial-grade pretrained models, inference scripts, and deployment runtimes for research and production use.

LocalAI is a free, Open Source OpenAI alternative. It acts as a drop-in replacement REST API that's compatible with OpenAI API specifications for local AI inferencing. It allows you to run LLMs, generate images, audio, and more locally or on-prem with consumer-grade hardware, supporting multiple model families without requiring a GPU.

BISHENG is an open LLM application DevOps platform for enterprise AI applications. It provides GenAI workflows, RAG, agents, unified model management, evaluation, SFT, dataset management, enterprise-grade system management and observability. Key strengths are a powerful visual workflow/orchestration system (with loops, parallelism, human-in-the-loop), high-precision document parsing, and features targeted at production enterprise deployments.