AI Infra2024
Connects multiple Macs and Linux machines into one cluster to run models too large for any single machine. Auto-discovers peers, shards a model across them via tensor parallelism, and exposes OpenAI-, Claude-, and Ollama-compatible APIs.
AI Infra2024
Disaggregated LLM serving architecture that splits prefill and decode into separate clusters and pools spare CPU, DRAM, and SSD into a distributed KVCache. Powers Kimi in production, handling 75% more requests under the same SLOs.
AI Infra2024
kvcache-ai (maintainers / MADSys group, Tsinghua University & contributors)MADSys Lab, Tsinghua University, Approaching.AI +1
Runs huge mixture-of-experts LLMs like DeepSeek-R1/V3 on a single 24GB GPU plus CPU DRAM by keeping attention on the GPU and offloading expert weights to CPU. Reports 3-28x speedups via Intel AMX/AVX512 kernels and fits 139K context in 24GB VRAM.
AI Infra2024
Official inference framework for 1-bit and ternary (1.58-bit) LLMs such as BitNet b1.58, with optimized CPU kernels. Delivers 1.37x-6.17x speedups and 55-82% lower energy on x86 and ARM, and runs a 100B model on a single CPU at 5-7 tokens/sec.
AI Audio2024
Chains four swappable open modules — voice activity detection, speech-to-text, an LLM, and text-to-speech — into a local voice agent that needs no proprietary APIs. Runs on CUDA, Apple Silicon, or Docker, with an OpenAI-compatible realtime WebSocket mode.
AI Infra2024
Runs local LLM, vision-language, ASR, OCR, and image-generation models across NPU, GPU, and CPU from one command. Differs from Ollama and llama.cpp with first-class Qualcomm Hexagon NPU support and day-0 coverage of new models like Qwen3-VL.
AI Deploy2024
Runs and optimizes ML and generative-AI models on-device across mobile, desktop, web, and IoT. Successor to TensorFlow Lite, it adds automated GPU/NPU accelerator selection and zero-copy buffer interop to cut latency without cloud round-trips.
AI Client2024
Runs AI models on user devices with native SDKs, optimized model management, hardware acceleration, and OpenAI-compatible APIs for apps that need offline, private inference.
MCP Server2024
Exposes a managed cloud browser to an LLM as MCP tools, letting an agent open sessions, navigate, click, read page elements, and pull data from live websites. Built on Stagehand, so steps are written in plain language, not brittle CSS selectors.
AI Infra2024
Full-stack AI red‑teaming platform that fingerprints AI infrastructure for known CVEs, audits MCP servers and agent skills with LLM-driven analysis, and runs cross-model jailbreak evaluations; designed for hands-on security assessment of AI deployments.
Embodied AI2024
Provides an open platform of omnimodal world models, datasets, and tools to build Physical AI — joint perception, generation, and action reasoning for robots, autonomous vehicles, and smart infrastructure. Supports images, video, audio, and action-conditioned workflows.
AI Deploy2025
Provides a hardware plugin that runs vLLM on Huawei Ascend NPUs by mapping vLLM execution and memory management to the Ascend runtime. Key features: support for Transformer/MoE/embedding/multimodal models, official docs, CI-backed release branches and community maintenance.