Tag

LMCache

Stores and reuses LLM key-value caches across GPU, CPU, disk, and remote backends so vLLM and SGLang skip recomputing repeated context. Non-prefix reuse (CacheBlend) and PD disaggregation cut time-to-first-token for long-context and RAG serving.

modded-nanogpt

A community speedrun to train a 124M GPT as fast as possible on 8 H100s, all chasing a fixed 3.28 FineWeb loss. Successive records cut the run from llm.c's 45 minutes to under 1.4, mostly via the new Muon optimizer rather than more hardware.

aisuite

aisuite is a lightweight Python library that provides a unified API for working with multiple Generative AI providers. It supports models from OpenAI, Anthropic, Google, Hugging Face, AWS, Cohere, Mistral, Ollama, and others—abstracting away SDK differences, authentication details, and parameter variations. Modeled after OpenAI’s API style, it enables developers to build LLM-based or agentic applications across providers with minimal setup.

MinerU

Converts PDFs, images, and Office documents into Markdown or JSON for retrieval, extraction, and agent workflows, with OCR, layout analysis, formula handling, and multiple runtime modes.

KTransformers

Runs huge mixture-of-experts LLMs like DeepSeek-R1/V3 on a single 24GB GPU plus CPU DRAM by keeping attention on the GPU and offloading expert weights to CPU. Reports 3-28x speedups via Intel AMX/AVX512 kernels and fits 139K context in 24GB VRAM.

bitnet.cpp (BitNet)

Official inference framework for 1-bit and ternary (1.58-bit) LLMs such as BitNet b1.58, with optimized CPU kernels. Delivers 1.37x-6.17x speedups and 55-82% lower energy on x86 and ARM, and runs a 100B model on a single CPU at 5-7 tokens/sec.

minimind

Trains a sub-100M-parameter LLM from scratch — pretraining, SFT, LoRA, DPO/RLHF, and distillation, sized from ~26M up to ~100M-plus dense and MoE. Headline figure: the ~64M minimind-3 variant's SFT stage runs 1 epoch in ~2h and ~3 RMB on one NVIDIA 3090.

ReMe

Stores agent memory as human- and agent-readable Markdown files with wikilinks instead of an opaque vector DB. Auto Memory/Resource/Dream jobs distill conversations into long-term notes, and hybrid wikilink + BM25 + embedding search retrieves them.

Docling

Converts PDFs, Office files, HTML, images and audio into one structured DoclingDocument, with deep PDF layout, reading order, table-structure and formula recognition, OCR, and native LangChain/LlamaIndex/Haystack integrations for RAG pipelines.

cuTile Python

Lets Python developers write tile-based parallel kernels for NVIDIA GPUs, generating CUDA Tile IR while staying close to Python syntax for custom GPU operations.

Tile Language (tile-lang)

A compact domain-specific language for writing high-performance GPU/CPU kernels (GEMM, FlashAttention, sparse kernels) with Python-like syntax. It provides tiling/pipelining primitives, a TVM-based compiler and multiple backends (CUDA/CuTeDSL, NVRTC, WebGPU, Metal, Ascend) for operator-level performance work.

Agent S

Enables agents to autonomously operate GUIs and complete complex computer tasks — includes the Agent S papers and the gui-agents SDK, grounding-model support, and runnable S3 agent implementations for Windows/macOS/Linux.