AI Deploy2023
Run any open-source LLM, embedding, speech, image, or multimodal model behind one OpenAI-compatible API — swap GPT for an open model in a single line. Routes across vLLM, llama.cpp, GGML, and TensorRT, scaling from a laptop to a multi-node GPU cluster.
AI Deploy2023
Compresses, deploys, and serves LLMs via two engines: TurboMind for raw speed, a PyTorch engine for flexibility. Claims ~1.8x vLLM throughput through persistent batching, blocked KV cache, and split-and-fuse; ships 4-bit AWQ and KV-cache quantization.
AI API2023
Calls 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure — through one OpenAI-compatible API, as a Python SDK or self-hosted proxy. The proxy adds virtual keys, spend tracking, rate limits, and load balancing across models and providers.
AI Train2023
Fine-tunes and deploys 600+ LLMs and 400+ multimodal models in one framework, with SFT, pretraining, RLHF (DPO, PPO, GRPO), and lightweight methods like LoRA and QLoRA. Adds Megatron parallelism, vLLM/SGLang/LMDeploy inference, and a training web UI.
MLOps2023
Self-hostable platform for building enterprise GenAI apps with visual workflow orchestration — loops, parallelism, human-in-the-loop — plus RAG, agents, unified model management, and in-house OCR for handwriting and rare characters.
AI Deploy2023
Compiles LLMs into optimized TensorRT inference engines for NVIDIA GPUs via a Python API. Layers in kernel fusion, quantization, paged attention, KV caching, and continuous in-flight batching, scaling from a single GPU to multi-node deployments.
AI Image2023
Provides a diffusion-model studio for image, video, audio-video, editing, LoRA, and full training workflows so many model families share one inference and training framework.
AI Deploy2023
Builds custom AI inference servers in pure Python on top of FastAPI, keeping full control over request logic while batching, GPU autoscaling, streaming, and OpenAI-spec endpoints come built in. Claims a 2x+ throughput edge over plain FastAPI.
AI Infra2024
Serves large language and multimodal models with low latency and high throughput using RadixAttention, continuous batching, structured outputs, parallelism, quantization, and broad accelerator support.
AI Audio2024
Lets AI agents place and answer business phone calls, holding spoken conversations to collect structured data, answer questions, and escalate to humans. Built on Azure Communication Services and Azure OpenAI, with RAG over your own documents.
AI Model2024
Performs document OCR, layout analysis, reading-order detection and table recognition across 90+ languages using a ~650M-parameter vision–language model; offers per-page and per-block modes and supports GPU (vllm) and CPU/Apple Silicon backends.
AI Deploy2024
GPU kernel library for LLM inference attention, sampling, and KV-cache, built on block-sparse formats with JIT-compiled customizable templates. Reports 29-69% inter-token-latency cuts vs compiler backends; powers SGLang, vLLM, and MLC-Engine.