AI Deploy2023
Open-source LLM inference and serving engine built around PagedAttention, which manages the KV cache like OS virtual memory to cut waste and raise throughput. Supports continuous batching, KV cache sharing, quantization, and an OpenAI-compatible API.
AI Audio2023
Reimplements OpenAI's Whisper speech-to-text on the CTranslate2 inference engine, running up to 4x faster at the same accuracy while using less memory. Adds a batched pipeline, 8-bit quantization, VAD filtering, and word-level timestamps.
AI Infra2023
Runs AI-generated code in secure, isolated cloud sandboxes you control via Python or JavaScript SDKs; supports self-hosting (Terraform) and AWS/GCP, enabling agents and code-interpreting workflows to execute real-world tools safely.
AI Infra2023
Runs large language models entirely in C/C++ with no external dependencies, using 1.5-to-8-bit integer quantization and CPU+GPU hybrid inference to fit models larger than available VRAM. Backs Ollama, LM Studio, and most local-inference tooling.
AI Agent2023
Builds production RAG systems around deep document understanding, explainable chunking, hybrid retrieval, citations, and agent workflows for messy enterprise documents.
AI Infra2023
Puts OpenAI-, Anthropic- and Ollama-compatible endpoints in front of 60+ inference backends, so existing client code runs unchanged against local models for text, vision, audio, image and embeddings. Runs CPU-only or accelerated, data stays local.
AI Infra2023
Probes LLMs for failure modes — prompt injection, jailbreaks, data leakage, toxicity, hallucination — the way nmap scans a network. Ships 20+ attack probes that run against Hugging Face, OpenAI, Bedrock, Cohere, or any REST endpoint.
AI Infra2023
Tracks, evaluates, and debugs LLM applications with traces, prompt management, datasets, playgrounds, and observability that can run in cloud or self-hosted setups.
AI Infra2023
Runs 70B-class LLM inference on a single 4GB GPU without quantization and supports Llama3.1 405B on 8GB VRAM. Uses layer-splitting and block-wise model compression (4/8-bit) to reduce disk load and can speed up inference loading by up to ~3x; integrates with Hugging Face models.
AI Image2023
Reference implementation for Stability AI's diffusion models: SDXL base/refiner/Turbo for text-to-image, plus Stable Video Diffusion, SV3D, and SV4D for image-to-video and 4D synthesis. A modular engine separates samplers, guiders, and conditioners.
AI Image2023
stable-diffusion.cpp is a pure C/C++ implementation for diffusion model inference, based on ggml, supporting models like Stable Diffusion (SD1.x, SD2.x, SDXL), Flux, Wan, Qwen Image, Z-Image, and more. It's lightweight with no external dependencies, supports backends like CPU, CUDA, Vulkan, Metal, and features like LoRA, ControlNet, LCM for efficient local image generation on platforms including Linux, Mac, Windows, and Android.
AI Train2023
Trains LLMs with RLHF at scale by splitting actor, critic, reward, and reference models across separate GPU groups via Ray, with vLLM-accelerated generation and DeepSpeed ZeRO-3. Supports PPO, GRPO, REINFORCE++, DPO, plus async and agentic multi-turn RL.