Long-context, multimodal models are increasingly necessary for agentic workflows and document-scale understanding, yet few open-weight options support both extremely long contexts and multi-modal inputs. dots3-note Preview aims to fill that gap by offering an inference-first Mixture-of-Experts (MoE) multimodal causal LM that exposes long-context (up to 512K tokens) multimodal inference at a 280B-parameter scale while keeping per-request activated compute lower via MoE routing and FP8 support.
Key Capabilities
- Multimodal input → text output: accepts text, images, video (with audio tracks) and audio, enabling single-model pipelines for transcription, image-to-text, video understanding and multimodal question answering — so you can avoid stitching separate encoders and LLMs for many applications.
- MoE efficiency and scale trade-off: 280B total parameters with ~16B activated parameters and 256 routed experts (top-8 routing) — this reduces activation compute for typical inferences compared with dense models of comparable total size, making very large-capacity models more practical for inference.
- Very long context and vocab: supports up to 512K tokens and a 152K vocabulary, useful for long-document summarization, codebase reasoning, and multi-turn agent memory tasks where context retention matters.
- Production-oriented checkpoints and deployment paths: provides BF16 and FP8 checkpoints (FP8 tuned for inference), and has native-serving recipes or integrations mentioned for vLLM, SGLang and Transformers PRs; designed for multi-GPU inference with expert-parallel/tensor-parallel configurations.
Who it's for & trade-offs
Great fit if you need a single-model multimodal inference stack that can handle extremely long contexts (hundreds of thousands of tokens) and mixed media (image/video/audio) for agentic workflows, long-document analysis, or multimodal QA, and you can provision multi-GPU H100-class resources and advanced serving stacks (vLLM/SGLang/Transformers PRs).
Look elsewhere if you need a compact on-device model, lightweight CPU inference, or a model specifically tuned for highest-accuracy dense decoding on commodity GPUs — dots3-note Preview is inference-focused, expects high-memory multi-GPU setups (FP8 recommended for cost/memory), and operationalizing MoE routing and expert-parallelism adds infrastructure complexity.
Where it fits
Use dots3-note Preview when you want an open-weight alternative for production multimodal agents that needs long context and tool/agent workflows. For lower-cost or single-GPU deployments prefer smaller dense multimodal models; for research into MoE training you may need additional tooling beyond this inference-targeted release.
