AI Infra2023
Runs 70B-class LLM inference on a single 4GB GPU without quantization and supports Llama3.1 405B on 8GB VRAM. Uses layer-splitting and block-wise model compression (4/8-bit) to reduce disk load and can speed up inference loading by up to ~3x; integrates with Hugging Face models.
AI Deploy2023
Run any open-source LLM, embedding, speech, image, or multimodal model behind one OpenAI-compatible API — swap GPT for an open model in a single line. Routes across vLLM, llama.cpp, GGML, and TensorRT, scaling from a laptop to a multi-node GPU cluster.
AI Deploy2023
Compresses, deploys, and serves LLMs via two engines: TurboMind for raw speed, a PyTorch engine for flexibility. Claims ~1.8x vLLM throughput through persistent batching, blocked KV cache, and split-and-fuse; ships 4-bit AWQ and KV-cache quantization.
AI API2023
Calls 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure — through one OpenAI-compatible API, as a Python SDK or self-hosted proxy. The proxy adds virtual keys, spend tracking, rate limits, and load balancing across models and providers.
AI Train2023
Fine-tunes and deploys 600+ LLMs and 400+ multimodal models in one framework, with SFT, pretraining, RLHF (DPO, PPO, GRPO), and lightweight methods like LoRA and QLoRA. Adds Megatron parallelism, vLLM/SGLang/LMDeploy inference, and a training web UI.
MLOps2023
Self-hostable platform for building enterprise GenAI apps with visual workflow orchestration — loops, parallelism, human-in-the-loop — plus RAG, agents, unified model management, and in-house OCR for handwriting and rare characters.
AI Deploy2023
Compiles LLMs into optimized TensorRT inference engines for NVIDIA GPUs via a Python API. Layers in kernel fusion, quantization, paged attention, KV caching, and continuous in-flight batching, scaling from a single GPU to multi-node deployments.
AI Infra2023
Provides a self-hosted inference engine that serves all models an agent needs—embeddings, retrieval/reranking, document-to-markdown OCR, structured extraction, content-safety scoring, and LLM generation—through an OpenAI-compatible API. Bundles a 100+ model catalog, SDKs, and production deployment tooling.
AI Image2023
Provides a diffusion-model studio for image, video, audio-video, editing, LoRA, and full training workflows so many model families share one inference and training framework.
AI Deploy2023
Builds custom AI inference servers in pure Python on top of FastAPI, keeping full control over request logic while batching, GPU autoscaling, streaming, and OpenAI-spec endpoints come built in. Claims a 2x+ throughput edge over plain FastAPI.
AI Infra2024
Serves large language and multimodal models with low latency and high throughput using RadixAttention, continuous batching, structured outputs, parallelism, quantization, and broad accelerator support.