Tag

PrivateGPT

Wraps a local, OpenAI-compatible inference server in one messages API so you can build private AI apps with no data leaving your network: document ingestion, retrieval with inline citations, and built-in tools (web search, code execution, MCP).

AirLLM

Runs 70B-class LLM inference on a single 4GB GPU without quantization and supports Llama3.1 405B on 8GB VRAM. Uses layer-splitting and block-wise model compression (4/8-bit) to reduce disk load and can speed up inference loading by up to ~3x; integrates with Hugging Face models.

Xorbits Inference (Xinference)

Run any open-source LLM, embedding, speech, image, or multimodal model behind one OpenAI-compatible API — swap GPT for an open model in a single line. Routes across vLLM, llama.cpp, GGML, and TensorRT, scaling from a laptop to a multi-node GPU cluster.

LMDeploy

Compresses, deploys, and serves LLMs via two engines: TurboMind for raw speed, a PyTorch engine for flexibility. Claims ~1.8x vLLM throughput through persistent batching, blocked KV cache, and split-and-fuse; ships 4-bit AWQ and KV-cache quantization.

Generative Models by Stability AI

Reference implementation for Stability AI's diffusion models: SDXL base/refiner/Turbo for text-to-image, plus Stable Video Diffusion, SV3D, and SV4D for image-to-video and 4D synthesis. A modular engine separates samplers, guiders, and conditioners.

3D Gaussian Splatting for Real-Time Radiance Field Rendering

Enables real-time (≥30 fps) 1080p novel-view synthesis by representing scenes as optimized anisotropic 3D Gaussians plus a visibility-aware splatting renderer; provides the paper's reference implementation, pretrained models and viewers — high-quality training requires CUDA GPU and significant VRAM.

stable-diffusion.cpp

stable-diffusion.cpp is a pure C/C++ implementation for diffusion model inference, based on ggml, supporting models like Stable Diffusion (SD1.x, SD2.x, SDXL), Flux, Wan, Qwen Image, Z-Image, and more. It's lightweight with no external dependencies, supports backends like CPU, CUDA, Vulkan, Metal, and features like LoRA, ControlNet, LCM for efficient local image generation on platforms including Linux, Mac, Windows, and Android.

LiteLLM

Calls 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure — through one OpenAI-compatible API, as a Python SDK or self-hosted proxy. The proxy adds virtual keys, spend tracking, rate limits, and load balancing across models and providers.

ms-swift

Fine-tunes and deploys 600+ LLMs and 400+ multimodal models in one framework, with SFT, pretraining, RLHF (DPO, PPO, GRPO), and lightweight methods like LoRA and QLoRA. Adds Megatron parallelism, vLLM/SGLang/LMDeploy inference, and a training web UI.

Magika

Detects file content types with a compact deep‑learning model that runs in milliseconds on a single CPU. Trained on ~100M samples across 200+ content types; offered as a Rust CLI plus Python, JS, and Go bindings for large‑scale security and file‑routing use.

BISHENG

Self-hostable platform for building enterprise GenAI apps with visual workflow orchestration — loops, parallelism, human-in-the-loop — plus RAG, agents, unified model management, and in-house OCR for handwriting and rare characters.

RealtimeSTT

Converts microphone or streamed audio to text with sub-second latency, pairing WebRTC/Silero voice-activity detection and wake-word activation with swappable local backends — faster-whisper by default, plus whisper.cpp, Moonshine, and sherpa-onnx.