AI Infra2020
Optimizes distributed PyTorch training and inference for very large models with ZeRO memory partitioning, parallelism, MoE, offload, and compression. Best when GPU memory, training cost, or cluster throughput is the bottleneck.
AI Deploy2021
Deploys trained SavedModels behind gRPC and REST endpoints, with hot-swappable versioning so new weights load without downtime. Built around servables, loaders, sources, and a manager, plus request batching to cut accelerator cost.
AI Infra2021
Runs, manages, and scales AI workloads across 20+ clouds, Kubernetes, Slurm, and on-prem from one YAML or Python spec. Auto-provisions GPUs/TPUs, fails over across regions and providers when capacity is short, and routes jobs to the cheapest option.
AI Deploy2021
Serves predictive and generative ML models on Kubernetes via a single InferenceService CRD, with scale-to-zero, canary rollouts, and an OpenAI-compatible LLM path on vLLM. One autoscaling abstraction over PyTorch, XGBoost, ONNX, and HuggingFace.
AI Train2021
HPC-AI Technology Inc. (Colossal-AI team), Shenggui Li +1HPC-AI Technology Inc., National University of Singapore
Scales a single-GPU training script to thousands of GPUs through a unified interface, combining data, pipeline, tensor, and sequence parallelism. Its Gemini memory manager offloads tensors across GPU, CPU, and NVMe so models far larger than VRAM still fit.
AI Image2021
Library for benchmarking, developing, and deploying deep-learning visual anomaly-detection models — includes ready-to-use model implementations (PatchCore, DINO-based), experiment/HPO tooling, OpenVINO export for edge inference, and a low-code Studio for deployment.
MLOps2021
Orchestrates ML training pipelines and production agent workflows from one Python codebase that runs unchanged from a laptop to Kubernetes or any cloud. Auto-versions artifacts, models, and agent checkpoints, with no orchestrator or framework lock-in.
AI Deploy2022
Deploys PyTorch models directly on phones, microcontrollers, and embedded hardware via ahead-of-time compilation to a ~50KB C++ runtime. Delegates subgraphs to 12+ backends (XNNPACK, CoreML, Qualcomm, ARM Ethos-U) with torchao quantization.
AI Model2022
Transformer-based foundation model for tabular data that provides pre-trained checkpoints for fast classification and regression, with GPU-accelerated local inference and an optional cloud client. Best suited for small-to-medium datasets (~≤50k rows).
AI Infra2022
Smart model router for personal AI agents that sends each request to the cheapest model capable of handling it — cutting API costs by up to ~70%. Uses a fast 23-dimension scorer, automatic fallbacks, per-tier controls, and supports local Docker self-hosting or a cloud app; ideal for cost-sensitive personal agents.
AI Model2022
Rust-and-Python toolkit that serves open-source LLMs (Llama, Falcon, Mixtral, StarCoder) over HTTP/gRPC with tensor parallelism, continuous batching, Flash/Paged Attention and quantization. Now in maintenance mode, pointing users toward vLLM and SGLang.