AI Infra2025
Reimplements the vLLM inference engine from scratch in ~1,200 lines of readable Python, matching its offline throughput on small models. Prefix caching, tensor parallelism, torch.compile, and CUDA graphs are all kept legible.
AI Audio2025
A toolkit and open-weights system for real-time streaming music generation — offers two model sizes (230M / 2.4B), a Python inference library (JAX/MLX), and a C++ engine optimized for Apple Silicon for embedding into DAWs and apps; real-time streaming requires M‑series chips.
AI Infra2025
Offline-first knowledge server that bundles local AI chat (Ollama + vector RAG), offline Wikipedia/education/maps, and utility tools behind a Dockerized management UI — designed to keep searchable knowledge available without cloud access.
AI Infra2025
Stores a pruned proximity graph instead of all embeddings, recomputing vectors on demand at query time. A 60M-doc index takes 6GB, not 201GB (97% less), at comparable recall. Powers private local RAG over files, mail, chat, and browser history.
AI Train2025
Forecasts financial candlesticks (OHLCV K-lines) with a decoder-only transformer pre-trained on 12B+ records from 45 exchanges. A tokenizer turns market data into discrete tokens, enabling price/volatility forecasting and synthetic K-line generation.
AI Infra2025
Model-compression toolkit for large LLMs/VLMs that integrates quantization (FP8/INT4/etc.), speculative decoding, token pruning and deployment hooks—designed for end-to-end performance on single/multi-GPU inference workflows and research-to-prod model optimization.
AI Agent2025
Unifies agentic tasks, reasoning, and coding in a single MoE model with 355B total / 32B active parameters and a switchable thinking mode. A lighter 106B-param Air variant trades scale for efficiency; both ship MIT-licensed.
AI Client2025
Cross‑platform AI client for web, desktop, and mobile that lets teams pick model providers, run local or on‑prem inference, and keep data self‑hosted — aimed at enterprise self‑deployment to avoid vendor lock‑in.
AI Deploy2025
Turns OpenAI Whisper into a live streaming transcriber: audio flows in over WebSocket and text returns word-by-word instead of after full utterances. Adds SimulStreaming and LocalAgreement decoding, Silero VAD, and speaker diarization, all self-hosted.
AI Deploy2025
A ~5,000-line Python LLM inference engine that re-implements SGLang's serving optimizations — radix KV-cache reuse, chunked prefill, overlap scheduling, tensor parallelism — as a fully type-annotated reference instead of a black box.
AI Infra2025
Extends vLLM beyond text to serve omni-modal models — Qwen3-Omni, TTS like CosyVoice3, and diffusion image/video/audio generators — in one engine, adding the non-autoregressive Diffusion Transformer support the core project never targeted.
AI Infra2025
Provides an NVFP4‑optimized training and inference infrastructure for long-form video diffusion models — supports multi-shot AR training, KV-cache and NVFP4 quantized inference, sequence-parallelism and async decoding for higher FPS and longer outputs.