AI Infra2025
Runs and fine-tunes LLMs locally on Apple silicon via the MLX framework, pulling thousands of Hugging Face models with one command. Adds 4- and 8-bit quantization, LoRA and full fine-tuning, prompt caching, and distributed inference across Macs.
AI Deploy2025
Splits LLM inference into separate prefill and decode GPU pools, then routes requests with KV-cache awareness to cut redundant recomputation. Reports up to 30x throughput on DeepSeek-R1 (GB200 NVL72) and works across TensorRT-LLM, vLLM, and SGLang.
AI Image2025
Real-time DETR detector on a DINOv2 backbone, covering detection, segmentation, and keypoints. Ships in six sizes (Nano to 2XL), beats YOLO on the COCO speed-accuracy curve, and transfers better to non-COCO real-world domains.
AI Deploy2025
Run large-language and multimodal models locally on edge devices (Android, iOS, desktop, web, Raspberry Pi) with hardware acceleration, function-calling, and multi-language SDKs—designed for low-latency, privacy-sensitive on-device inference.
AI Infra2025
Measures generative AI inference performance with token-level metrics (TTFT, inter-token latency), latency, and throughput under realistic traffic patterns. Provides a multiprocess engine, real-time TUI dashboard, extensible plugins, and integrations for telemetry and result uploads, aimed at inference benchmarking and capacity planning.
AI Image2025
Converts document images—scans, photos, born-digital PDFs—into structured text in two stages: first map layout and reading order, then parse each element (text, tables, formulas, figures) in parallel, each guided by its own task prompt.
AI Infra2025
Parses the local JSONL logs that coding-agent CLIs write and turns them into token and cost reports, no API keys or telemetry. Breaks spend down by day, month, session, and Claude's 5-hour billing windows across Claude Code, Codex, Gemini CLI and more.
AI Deploy2025
Turns commodity WiFi Channel State Information into spatial sensing: 17-keypoint pose estimation, presence detection, and contactless breathing/heart-rate monitoring through walls, with no camera. Runs on a mesh of ESP32-S3 nodes (~$9 each).
AI Infra2025
Reimplements the vLLM inference engine from scratch in ~1,200 lines of readable Python, matching its offline throughput on small models. Prefix caching, tensor parallelism, torch.compile, and CUDA graphs are all kept legible.
AI Audio2025
A toolkit and open-weights system for real-time streaming music generation — offers two model sizes (230M / 2.4B), a Python inference library (JAX/MLX), and a C++ engine optimized for Apple Silicon for embedding into DAWs and apps; real-time streaming requires M‑series chips.
AI Infra2025
Offline-first knowledge server that bundles local AI chat (Ollama + vector RAG), offline Wikipedia/education/maps, and utility tools behind a Dockerized management UI — designed to keep searchable knowledge available without cloud access.
AI Infra2025
Stores a pruned proximity graph instead of all embeddings, recomputing vectors on demand at query time. A 60M-doc index takes 6GB, not 201GB (97% less), at comparable recall. Powers private local RAG over files, mail, chat, and browser history.