Category

Unstract

Extracts structured JSON from unstructured documents — PDFs, scans, images — by defining extraction schemas as natural-language prompts, then shipping them as REST APIs or ETL pipelines. Swappable across LLM providers and vector DBs.

NVIDIA Model Optimizer

Applies quantization, pruning, distillation, speculative decoding and sparsity to compress and accelerate deep learning models, producing exportable checkpoints ready for deployment on TensorRT‑LLM, TensorRT, vLLM and similar inference runtimes.

FastVideo

Accelerates video generation with a unified framework for inference, finetuning, LoRA, distillation, sparse attention, and distributed execution for research and demos.

Oumi

Streamlines the full lifecycle of foundation models — data prep, fine-tuning (SFT/LoRA/QLoRA/GRPO), evaluation, and deployment — with ready-to-run recipes, multi-engine inference support, and cloud/CLI workflows for both laptop experiments and large-scale runs.

LMCache

Stores and reuses LLM key-value caches across GPU, CPU, disk, and remote backends so vLLM and SGLang skip recomputing repeated context. Non-prefix reuse (CacheBlend) and PD disaggregation cut time-to-first-token for long-context and RAG serving.

Mooncake

Disaggregated LLM serving architecture that splits prefill and decode into separate clusters and pools spare CPU, DRAM, and SSD into a distributed KVCache. Powers Kimi in production, handling 75% more requests under the same SLOs.

ElizaOS

Local-first runtime for autonomous AI agents that run on-device and stay model-agnostic across OpenAI, Anthropic, Gemini, Grok, and local models. A plugin system adds chat platforms (Discord, Telegram, X), voice, browser automation, RAG, and wallets.

NexaSDK

Runs local LLM, vision-language, ASR, OCR, and image-generation models across NPU, GPU, and CPU from one command. Differs from Ollama and llama.cpp with first-class Qualcomm Hexagon NPU support and day-0 coverage of new models like Qwen3-VL.

LiteRT

Runs and optimizes ML and generative-AI models on-device across mobile, desktop, web, and IoT. Successor to TensorFlow Lite, it adds automated GPU/NPU accelerator selection and zero-copy buffer interop to cut latency without cloud round-trips.

Cloudflare Vibe SDK

Generates and deploys full-stack React apps from natural-language prompts on Cloudflare’s platform, combining AI code generation, previews, Workers, Durable Objects, and containers.

Foundry Local

Runs AI models on user devices with native SDKs, optimized model management, hardware acceleration, and OpenAI-compatible APIs for apps that need offline, private inference.

Open Notebook

Self-hostable alternative to Google NotebookLM: organize PDFs, videos, audio, web pages, and Office docs, then chat over them, take AI-assisted notes, and search via full-text and vector. Routes to 18+ model providers and generates 1-4 speaker podcasts.