AI Deploy2023
Compresses, deploys, and serves LLMs via two engines: TurboMind for raw speed, a PyTorch engine for flexibility. Claims ~1.8x vLLM throughput through persistent batching, blocked KV cache, and split-and-fuse; ships 4-bit AWQ and KV-cache quantization.
AI Image2023
stable-diffusion.cpp is a pure C/C++ implementation for diffusion model inference, based on ggml, supporting models like Stable Diffusion (SD1.x, SD2.x, SDXL), Flux, Wan, Qwen Image, Z-Image, and more. It's lightweight with no external dependencies, supports backends like CPU, CUDA, Vulkan, Metal, and features like LoRA, ControlNet, LCM for efficient local image generation on platforms including Linux, Mac, Windows, and Android.
AI API2023
Calls 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure — through one OpenAI-compatible API, as a Python SDK or self-hosted proxy. The proxy adds virtual keys, spend tracking, rate limits, and load balancing across models and providers.
MLOps2023
Self-hostable platform for building enterprise GenAI apps with visual workflow orchestration — loops, parallelism, human-in-the-loop — plus RAG, agents, unified model management, and in-house OCR for handwriting and rare characters.
AI Deploy2023
Compiles LLMs into optimized TensorRT inference engines for NVIDIA GPUs via a Python API. Layers in kernel fusion, quantization, paged attention, KV caching, and continuous in-flight batching, scaling from a single GPU to multi-node deployments.
AI Deploy2023
Builds custom AI inference servers in pure Python on top of FastAPI, keeping full control over request logic while batching, GPU autoscaling, streaming, and OpenAI-spec endpoints come built in. Claims a 2x+ throughput edge over plain FastAPI.
AI Audio2024
Lets AI agents place and answer business phone calls, holding spoken conversations to collect structured data, answer questions, and escalate to humans. Built on Azure Communication Services and Azure OpenAI, with RAG over your own documents.
Chatbot2024
Creates personalized digital avatars (AI twins) by fine-tuning LLMs on users' chat history and binding them to chatbots. Provides an end-to-end pipeline — chat export, preprocessing with privacy filters, SFT/LoRA training, and deployment (Telegram/Discord/Slack). Best with larger models and substantial chat data.
AI Deploy2024
GPU kernel library for LLM inference attention, sampling, and KV-cache, built on block-sparse formats with JIT-compiled customizable templates. Reports 29-69% inter-token-latency cuts vs compiler backends; powers SGLang, vLLM, and MLC-Engine.
AI Infra2024
Runs AI-generated code in isolated, elastic sandboxes with SDK, API, and CLI access for agent workflows that need stateful execution and environment control.
AI Train2024
Accelerates video generation with a unified framework for inference, finetuning, LoRA, distillation, sparse attention, and distributed execution for research and demos.