Multimodal search and recommendation increasingly require a single embedding space that preserves fine-grained semantics across text, images, video and document-like visuals while remaining efficient at billion-scale indexing. The core insight: training a family of scalable encoders (2B/4B/9B) with a two‑stage curriculum yields embeddings that are both serviceable in production and competitive on open benchmarks.
Key Findings
- Two-stage training: large-scale multimodal alignment followed by a refinement phase using curated data, fine-grained relevance supervision, and cross-scale knowledge transfer — this combination improves discrimination without sacrificing broad coverage. This means embeddings better match hard retrieval cases while retaining generality.
- Matryoshka embeddings and configurable dimensions: models expose multiple output sizes (e.g., 64–4096) so teams can truncate to lower dims (e.g., 256) to cut storage/compute with modest accuracy loss. This enables practical deployment trade-offs for latency- and cost-sensitive systems.
- Empirical results: across public benchmarks the family leads or matches state-of-the-art; the 9B variant achieves an overall MMEB-v2 score of 80.6 and produces 4096‑D L2‑normalized vectors. WeChat-internal evaluations and multiple online A/B tests show production gains in recommendation and search.
- Engineering posture: weights and code are released under Apache-2.0, calling out support for common inference stacks (Transformers/SentenceTransformers, vLLM) and Hugging Face hosting, which lowers integration friction for practitioners.
Who it fits and trade-offs
Great fit if you need a unified, production-ready multimodal embedding backbone that supports cross-modal retrieval and can be dimension‑truncated to manage cost. Prefer it when you want an off-the-shelf family with released weights, benchmarked performance, and documented deployment recipes.
Look elsewhere if you require audio input support (audio is not supported) or if you need a custom task-specific embedding trained on proprietary domain data without fine-tuning; extremely constrained-device scenarios might still favor specialized lightweight encoders despite Matryoshka truncation.