AI Infra2024
Cloud-native control plane that scales vLLM on Kubernetes, adding the routing, autoscaling, and fault tolerance single-instance serving lacks. Brings high-density LoRA management, an LLM gateway, distributed KV cache reuse, and SLO-aware GPU serving.
AI Client2024
Register React components with Zod schemas so an LLM agent can select, fill, and stream their props from user requests, turning chat into live interactive UI. Works with OpenAI, Anthropic, Gemini, and Mistral, plus MCP servers like Linear.
A 236B-parameter Mixture-of-Experts LLM that activates only 21B parameters per token, cutting training cost 42.5% versus a dense 67B model and shrinking the KV cache 93.3% via Multi-head Latent Attention, with 128K context.
AI Agent2024
Provides a Python framework for building generative-AI agents and workflows with Pydantic-style type safety and composable capabilities. Model-agnostic provider support, built-in observability, human-in-the-loop tool approval, and durable execution for production use cases.
AI Infra2024
Connects multiple Macs and Linux machines into one cluster to run models too large for any single machine. Auto-discovers peers, shards a model across them via tensor parallelism, and exposes OpenAI-, Claude-, and Ollama-compatible APIs.
AI Client2024
Visually edit Next.js + Tailwind projects in the browser like Figma, with every change written straight back to your real React code. Pairs a DOM-level visual canvas with AI chat that scaffolds and edits components, plus branching and one-click deploy.
AI Infra2024
Disaggregated LLM serving architecture that splits prefill and decode into separate clusters and pools spare CPU, DRAM, and SSD into a distributed KVCache. Powers Kimi in production, handling 75% more requests under the same SLOs.
Official code companion to the O'Reilly book by Jay Alammar and Maarten Grootendorst: 12 chapters of runnable notebooks on tokens, embeddings, Transformers, text classification, clustering, prompt engineering, semantic search, RAG, and fine-tuning.
Publishes a structured open textbook on large language model foundations, covering language modeling, LLM architectures, prompt engineering, PEFT, model editing, and RAG.
AI API2024
aisuite is a lightweight Python library that provides a unified API for working with multiple Generative AI providers. It supports models from OpenAI, Anthropic, Google, Hugging Face, AWS, Cohere, Mistral, Ollama, and others—abstracting away SDK differences, authentication details, and parameter variations. Modeled after OpenAI’s API style, it enables developers to build LLM-based or agentic applications across providers with minimal setup.
AI Audio2024
Open-source TTS that clones a voice from 3-10s of audio and synthesizes cross-lingual speech in 9 languages and 18+ Chinese dialects. Supports streaming at ~150ms latency with instruction control over emotion, speed, and accent.
AI Infra2024
Converts PDFs, images, and Office documents into Markdown or JSON for retrieval, extraction, and agent workflows, with OCR, layout analysis, formula handling, and multiple runtime modes.