Tag
Explore by tags
Runs AI-generated code in isolated, elastic sandboxes with SDK, API, and CLI access for agent workflows that need stateful execution and environment control.
Runs a privacy-first, self-hosted answering engine that combines web retrieval with local and cloud LLMs to produce cited answers. Supports SearxNG search, file uploads, image/video search, and mix-and-match models with Speed/Balanced/Quality modes.
Provides local inference, fine-tuning, and a server/CLI for vision–language and omni (image/audio/video) models via MLX. Supports multi-image chat, audio/video inputs, activation quantization (CUDA), TurboQuant KV cache, and LoRA/QLoRA fine-tuning for on-device workflows.
Runs open LLMs entirely on your own machine — discover and download models from Hugging Face, chat in a desktop GUI, or expose an OpenAI-compatible local server. Native Apple MLX and llama.cpp backends; headless deploy via llmster.
Segments each PDF page into 11 labeled regions — titles, tables, formulas, figures, footnotes and more — and recovers reading order. Offers two engines: an accurate VGT visual model (~0.96 F1) or a faster CPU-only LightGBM ensemble.
Stores and reuses LLM key-value caches across GPU, CPU, disk, and remote backends so vLLM and SGLang skip recomputing repeated context. Non-prefix reuse (CacheBlend) and PD disaggregation cut time-to-first-token for long-context and RAG serving.
Cloud-native control plane that scales vLLM on Kubernetes, adding the routing, autoscaling, and fault tolerance single-instance serving lacks. Brings high-density LoRA management, an LLM gateway, distributed KV cache reuse, and SLO-aware GPU serving.
Connects multiple Macs and Linux machines into one cluster to run models too large for any single machine. Auto-discovers peers, shards a model across them via tensor parallelism, and exposes OpenAI-, Claude-, and Ollama-compatible APIs.
Disaggregated LLM serving architecture that splits prefill and decode into separate clusters and pools spare CPU, DRAM, and SSD into a distributed KVCache. Powers Kimi in production, handling 75% more requests under the same SLOs.
Runs huge mixture-of-experts LLMs like DeepSeek-R1/V3 on a single 24GB GPU plus CPU DRAM by keeping attention on the GPU and offloading expert weights to CPU. Reports 3-28x speedups via Intel AMX/AVX512 kernels and fits 139K context in 24GB VRAM.
Official inference framework for 1-bit and ternary (1.58-bit) LLMs such as BitNet b1.58, with optimized CPU kernels. Delivers 1.37x-6.17x speedups and 55-82% lower energy on x86 and ARM, and runs a 100B model on a single CPU at 5-7 tokens/sec.