AI Infra2015
Builds and deploys machine learning models across research, production, web, mobile, and edge environments. Its ecosystem spans Keras, TFX, LiteRT, TensorFlow.js, datasets, model hubs, and visualization tools.
AI Deploy2018
Serves machine learning and deep learning models for cloud, data center, edge and embedded environments. Supports multiple frameworks and backends, dynamic and sequence batching, HTTP/gRPC APIs, Docker deployment and NVIDIA-optimized runtimes.
AI Infra2018
Runs ONNX models faster on CPU, GPU, and NPU by routing graph subgraphs to backend execution providers (CUDA, TensorRT, OpenVINO, DirectML, CoreML). One engine serves the same model across cloud, browser, mobile, and edge, for both inference and training.
AI Deploy2021
Serves predictive and generative ML models on Kubernetes via a single InferenceService CRD, with scale-to-zero, canary rollouts, and an OpenAI-compatible LLM path on vLLM. One autoscaling abstraction over PyTorch, XGBoost, ONNX, and HuggingFace.
AI Model2022
Rust-and-Python toolkit that serves open-source LLMs (Llama, Falcon, Mixtral, StarCoder) over HTTP/gRPC with tensor parallelism, continuous batching, Flash/Paged Attention and quantization. Now in maintenance mode, pointing users toward vLLM and SGLang.
AI Deploy2023
Run any open-source LLM, embedding, speech, image, or multimodal model behind one OpenAI-compatible API — swap GPT for an open model in a single line. Routes across vLLM, llama.cpp, GGML, and TensorRT, scaling from a laptop to a multi-node GPU cluster.
AI Deploy2024
GPU kernel library for LLM inference attention, sampling, and KV-cache, built on block-sparse formats with JIT-compiled customizable templates. Reports 29-69% inter-token-latency cuts vs compiler backends; powers SGLang, vLLM, and MLC-Engine.
AI Model2024
A research codebase and model family for vision–language models that experiments with data‑centric post‑training strategies and long‑context multimodal reasoning. Includes model reports, released research weights (non‑commercial), grounding tools (LocateAnything) and integrations for inference/optimization.
AI Train2024
A 671B-parameter Mixture-of-Experts language model (37B activated) trained on 14.8T tokens with 128K context, FP8-first training, a Multi-Token Prediction module, and Hugging Face weights—focused on efficient MoE training and long-context use cases.
Embodied AI2025
A vision-language-action foundation model and reference stack for generalized humanoid and cross-embodiment robot manipulation. Provides pretrained checkpoints, demo datasets, and tooling for fine-tuning, evaluation, and deployment (ONNX/TensorRT); released as Early Access.
AI Deploy2025
Splits LLM inference into separate prefill and decode GPU pools, then routes requests with KV-cache awareness to cut redundant recomputation. Reports up to 30x throughput on DeepSeek-R1 (GB200 NVL72) and works across TensorRT-LLM, vLLM, and SGLang.