AI Deploy2021
Deploys trained SavedModels behind gRPC and REST endpoints, with hot-swappable versioning so new weights load without downtime. Built around servables, loaders, sources, and a manager, plus request batching to cut accelerator cost.
AI Infra2021
Runs, manages, and scales AI workloads across 20+ clouds, Kubernetes, Slurm, and on-prem from one YAML or Python spec. Auto-provisions GPUs/TPUs, fails over across regions and providers when capacity is short, and routes jobs to the cheapest option.
AI Deploy2021
Serves predictive and generative ML models on Kubernetes via a single InferenceService CRD, with scale-to-zero, canary rollouts, and an OpenAI-compatible LLM path on vLLM. One autoscaling abstraction over PyTorch, XGBoost, ONNX, and HuggingFace.
AI Train2021
HPC-AI Technology Inc. (Colossal-AI team), Shenggui Li +1HPC-AI Technology Inc., National University of Singapore
Scales a single-GPU training script to thousands of GPUs through a unified interface, combining data, pipeline, tensor, and sequence parallelism. Its Gemini memory manager offloads tensors across GPU, CPU, and NVMe so models far larger than VRAM still fit.
AI Image2021
Library for benchmarking, developing, and deploying deep-learning visual anomaly-detection models — includes ready-to-use model implementations (PatchCore, DINO-based), experiment/HPO tooling, OpenVINO export for edge inference, and a low-code Studio for deployment.
AI Deploy2022
Deploys PyTorch models directly on phones, microcontrollers, and embedded hardware via ahead-of-time compilation to a ~50KB C++ runtime. Delegates subgraphs to 12+ backends (XNNPACK, CoreML, Qualcomm, ARM Ethos-U) with torchao quantization.
AI Infra2022
Fused CUDA kernels that compute exact attention without ever writing the full N×N score matrix to GPU memory, cutting memory from quadratic to linear and speeding up training and inference on A100/H100. Ships FlashAttention-2/3 plus KV-cache decode paths.
AI Model2022
Transformer-based foundation model for tabular data that provides pre-trained checkpoints for fast classification and regression, with GPU-accelerated local inference and an optional cloud client. Best suited for small-to-medium datasets (~≤50k rows).
AI Image2022
Unifies successive YOLO generations — YOLOv8, YOLO11, YOLOv3 and newer — under one package and a single `YOLO` API spanning detection, segmentation, classification, pose, oriented boxes and tracking, plus one-line export to ONNX, TensorRT and CoreML.
AI Infra2022
Smart model router for personal AI agents that sends each request to the cheapest model capable of handling it — cutting API costs by up to ~70%. Uses a fast 23-dimension scorer, automatic fallbacks, per-tier controls, and supports local Docker self-hosting or a cloud app; ideal for cost-sensitive personal agents.
AI Model2022
Rust-and-Python toolkit that serves open-source LLMs (Llama, Falcon, Mixtral, StarCoder) over HTTP/gRPC with tensor parallelism, continuous batching, Flash/Paged Attention and quantization. Now in maintenance mode, pointing users toward vLLM and SGLang.
AI Agent2022
Build, run, and monitor LLM agents across one stack: an open framework for chaining models and tools, LangGraph for stateful agent orchestration, and LangSmith for tracing, evaluation, and deployment in production.