AI Infra2024
Cloud-native control plane that scales vLLM on Kubernetes, adding the routing, autoscaling, and fault tolerance single-instance serving lacks. Brings high-density LoRA management, an LLM gateway, distributed KV cache reuse, and SLO-aware GPU serving.
AI Infra2024
Disaggregated LLM serving architecture that splits prefill and decode into separate clusters and pools spare CPU, DRAM, and SSD into a distributed KVCache. Powers Kimi in production, handling 75% more requests under the same SLOs.
AI Audio2024
Open-source TTS that clones a voice from 3-10s of audio and synthesizes cross-lingual speech in 9 languages and 18+ Chinese dialects. Supports streaming at ~150ms latency with instruction control over emotion, speed, and accent.
AI Train2024
Open-source HybridFlow implementation for RL post-training of LLMs. Decouples control flow from compute so PPO, GRPO, GSPO and DAPO share one dataflow; pairs FSDP/Megatron with vLLM/SGLang rollout and reports 1.5-20x throughput over prior RLHF stacks.
AI Train2024
A 671B-parameter Mixture-of-Experts language model (37B activated) trained on 14.8T tokens with 128K context, FP8-first training, a Multi-Token Prediction module, and Hugging Face weights—focused on efficient MoE training and long-context use cases.
Embodied AI2024
Provides an open platform of omnimodal world models, datasets, and tools to build Physical AI — joint perception, generation, and action reasoning for robots, autonomous vehicles, and smart infrastructure. Supports images, video, audio, and action-conditioned workflows.
AI Deploy2025
Provides a hardware plugin that runs vLLM on Huawei Ascend NPUs by mapping vLLM execution and memory management to the Ascend runtime. Key features: support for Transformer/MoE/embedding/multimodal models, official docs, CI-backed release branches and community maintenance.
AI Train2025
An asynchronous, high-throughput framework for large-scale reinforcement learning and agentic training that scales to 1T+ MoE models and 1000+ GPUs, with native verifiers integration, end-to-end SFT/RL/evals, and Slurm/Kubernetes deployment; requires NVIDIA GPUs.
AI Train2025
Trains multi-step LLM agents with reinforcement learning (GRPO) on your own tasks, wrapping existing agent code behind an OpenAI-compatible client. Its RULER mode scores trajectories with an LLM judge, so there's no reward function to hand-write.
AI Deploy2025
Splits LLM inference into separate prefill and decode GPU pools, then routes requests with KV-cache awareness to cut redundant recomputation. Reports up to 30x throughput on DeepSeek-R1 (GB200 NVL72) and works across TensorRT-LLM, vLLM, and SGLang.
AI Infra2025
Reimplements the vLLM inference engine from scratch in ~1,200 lines of readable Python, matching its offline throughput on small models. Prefix caching, tensor parallelism, torch.compile, and CUDA graphs are all kept legible.
AI Infra2025
Model-compression toolkit for large LLMs/VLMs that integrates quantization (FP8/INT4/etc.), speculative decoding, token pruning and deployment hooks—designed for end-to-end performance on single/multi-GPU inference workflows and research-to-prod model optimization.