AI Infra2024
Stores and reuses LLM key-value caches across GPU, CPU, disk, and remote backends so vLLM and SGLang skip recomputing repeated context. Non-prefix reuse (CacheBlend) and PD disaggregation cut time-to-first-token for long-context and RAG serving.
AI Train2024
A community speedrun to train a 124M GPT as fast as possible on 8 H100s, all chasing a fixed 3.28 FineWeb loss. Successive records cut the run from llm.c's 45 minutes to under 1.4, mostly via the new Muon optimizer rather than more hardware.
AI API2024
Ingests documents, images, audio, video and web pages and converts them into structured, LLM-friendly markdown and parsed data. Runs locally (fits on a T4 GPU), supports ~20 file types, offers OCR, transcription, table extraction and a Gradio UI; deployable via Docker/Skypilot. Licensed under GPL-3.0; some model weights carry cc-by-nc-sa restrictions for commercial use.
AI Infra2024
Cloud-native control plane that scales vLLM on Kubernetes, adding the routing, autoscaling, and fault tolerance single-instance serving lacks. Brings high-density LoRA management, an LLM gateway, distributed KV cache reuse, and SLO-aware GPU serving.
AI Client2024
Register React components with Zod schemas so an LLM agent can select, fill, and stream their props from user requests, turning chat into live interactive UI. Works with OpenAI, Anthropic, Gemini, and Mistral, plus MCP servers like Linear.
MCP Server2024
Continuously records your screen and audio 24/7 to a local, searchable timeline you can query in natural language. Stores screenshots with accessibility data in SQLite, and a plugin system runs scheduled AI agents on what it captures.
AI Agent2024
Builds real-time multimodal conversational AI agents with voice-assistant examples, VAD, turn detection, RTC/WebSocket transport, avatars, transcription, and edge-device demos.
AI Infra2024
Connects multiple Macs and Linux machines into one cluster to run models too large for any single machine. Auto-discovers peers, shards a model across them via tensor parallelism, and exposes OpenAI-, Claude-, and Ollama-compatible APIs.
AI Client2024
Visually edit Next.js + Tailwind projects in the browser like Figma, with every change written straight back to your real React code. Pairs a DOM-level visual canvas with AI chat that scaffolds and edits components, plus branching and one-click deploy.
AI Model2024
Provides code, pretrained weights, and tooling for protein language models and structure prediction — including ESMC, ESMFold2, sparse autoencoders (SAEs), and the ESM Atlas. Includes model checkpoints, tutorials, Hugging Face & Biohub integration, and an MIT license.
AI Infra2024
Disaggregated LLM serving architecture that splits prefill and decode into separate clusters and pools spare CPU, DRAM, and SSD into a distributed KVCache. Powers Kimi in production, handling 75% more requests under the same SLOs.