Provides an open-source Mixture-of-Experts coding LLM (397B) optimized for agentic, tool-enabled coding workflows with a 262,144-token context window, OpenAI-compatible API, serving recipes (vLLM/SGLang), and published coding-benchmark results.
Streams routed experts from disk and treats storage, RAM and VRAM as a single inference hierarchy so MoE models (hundreds of billions to trillions of parameters) can run on consumer or heterogeneous hardware; single-file C engines with zero runtime deps and optional Vulkan/CUDA/Metal backends, speed governed by placement and disk I/O.
Provides a portable C++ inference runtime to deploy embodied AI models (vision–language–action and world–action) on heterogeneous robot hardware, enabling latency-first batch-1 closed-loop control. Key features include modular multi-rate layers, fused low-latency inference, and extensible head/IO plugins.
Runs large open-weight MoE models locally by coordinating CPU and GPU resources, caching experts, and reallocating VRAM at runtime. It targets consumer hardware, from laptops to workstation GPUs.
Enables interactive serving of large Mixture-of-Experts (MoE) models on personal machines by adapting offload and execution to measured device bandwidth and agentic workload patterns. Key features include bandwidth-adaptive execution, semantic-aware caching of recurrent state, and an elastic GPU expert cache; supports 20+ MoE models and runs models from ~35B to 753B on consumer/workstation GPUs.
Evaluates multi-field JSON schemas in parallel to extract boolean or categorical field values from text, producing guaranteed-valid JSON and per-field calibrated confidences. Uses KV-cache broadcasting, sub-vocabulary logit slicing and token-tree disambiguation to cut latency (5.6x–7.0x on M4 Max) versus autoregressive decoding; requires Apple Silicon and MLX.
Provides GGUF quantizations of DeepSeek-V4.1-Flash-1017 for local inference, with swappable Engram n-gram tables. Even the smallest quant requires about 128 GB of memory; SSD offloading can reduce the amount kept resident.