AI Deploy2024
GPU kernel library for LLM inference attention, sampling, and KV-cache, built on block-sparse formats with JIT-compiled customizable templates. Reports 29-69% inter-token-latency cuts vs compiler backends; powers SGLang, vLLM, and MLC-Engine.
AI Infra2024
Chains pre-trained AI weather and climate models like GraphCast, Pangu, and FourCastNet into composable inference pipelines. Swap prognostic or diagnostic components, plug in reanalysis sources, and add ensemble perturbations or in-loop metrics.
AI Agent2024
React components for building LLM chat and agent interfaces: message bubbles, prompt sets, conversation lists, and sender inputs under a RICH interaction paradigm, plus a streaming Markdown renderer and hooks for wiring UI to model data streams.
AI Infra2024
Stores and reuses LLM key-value caches across GPU, CPU, disk, and remote backends so vLLM and SGLang skip recomputing repeated context. Non-prefix reuse (CacheBlend) and PD disaggregation cut time-to-first-token for long-context and RAG serving.
AI Train2024
A community speedrun to train a 124M GPT as fast as possible on 8 H100s, all chasing a fixed 3.28 FineWeb loss. Successive records cut the run from llm.c's 45 minutes to under 1.4, mostly via the new Muon optimizer rather than more hardware.
AI API2024
aisuite is a lightweight Python library that provides a unified API for working with multiple Generative AI providers. It supports models from OpenAI, Anthropic, Google, Hugging Face, AWS, Cohere, Mistral, Ollama, and others—abstracting away SDK differences, authentication details, and parameter variations. Modeled after OpenAI’s API style, it enables developers to build LLM-based or agentic applications across providers with minimal setup.
AI Infra2024
Converts PDFs, images, and Office documents into Markdown or JSON for retrieval, extraction, and agent workflows, with OCR, layout analysis, formula handling, and multiple runtime modes.
AI Infra2024
kvcache-ai (maintainers / MADSys group, Tsinghua University & contributors)MADSys Lab, Tsinghua University, Approaching.AI +1
Runs huge mixture-of-experts LLMs like DeepSeek-R1/V3 on a single 24GB GPU plus CPU DRAM by keeping attention on the GPU and offloading expert weights to CPU. Reports 3-28x speedups via Intel AMX/AVX512 kernels and fits 139K context in 24GB VRAM.
AI Infra2024
Official inference framework for 1-bit and ternary (1.58-bit) LLMs such as BitNet b1.58, with optimized CPU kernels. Delivers 1.37x-6.17x speedups and 55-82% lower energy on x86 and ARM, and runs a 100B model on a single CPU at 5-7 tokens/sec.
AI Train2024
Trains a sub-100M-parameter LLM from scratch — pretraining, SFT, LoRA, DPO/RLHF, and distillation, sized from ~26M up to ~100M-plus dense and MoE. Headline figure: the ~64M minimind-3 variant's SFT stage runs 1 epoch in ~2h and ~3 RMB on one NVIDIA 3090.
AI Agent2024
Stores agent memory as human- and agent-readable Markdown files with wikilinks instead of an opaque vector DB. Auto Memory/Resource/Dream jobs distill conversations into long-term notes, and hybrid wikilink + BM25 + embedding search retrieves them.