AIAny
AI Deploy2023
Icon for item

Modular Platform

Provides a unified platform for AI development and deployment, including the MAX serving framework and the Mojo systems programming language for writing kernels. Offers an OpenAI-compatible inference endpoint, Mojo-written CPU/GPU kernels, and tools to run hundreds of open models across diverse hardware without vendor lock-in.

Introduction

Deploying open LLMs at production scale often forces trade-offs: large vendor-specific runtimes (CUDA/ROCm), heavy containers, and brittle portability across hardware. This project addresses that operational friction by combining a Python-based inference server with a kernel-focused systems language (Mojo) so teams can optimize at the kernel level while keeping a single, OpenAI-compatible serving API.

What Sets It Apart
  • Kernel-first portability: Kernels for CPUs and GPUs are written in Mojo, enabling low-level performance tuning that can be targeted to multiple hardware backends without binding to CUDA or ROCm — so you can move workloads across vendors with fewer changes.
  • OpenAI-compatible serving with MAX: A production-oriented Python inference server exposes OpenAI-style endpoints and model pipelines, simplifying adoption for services already built around that API.
  • Smaller, faster containers and tooling: The stack emphasizes minimal runtime dependencies (no PyTorch/CUDA bundling), which reduces container sizes and cold-start costs for inference fleets.
  • End-to-end extensibility: From high-level Python pipelines to low-level Mojo kernels, the repo provides reference implementations and examples to extend models, ops, and hardware targets.
Who It's For & Trade-offs

Great fit if you build or operate LLM inference at scale and need hardware-agnostic deployment, kernel-level performance tuning, or an OpenAI-compatible endpoint across heterogeneous clusters. It suits infra engineers and model-serving teams who can invest in custom kernels or integrate MAX into orchestration flows.

Look elsewhere if you need a drop-in PyTorch/CUDA workflow, require an exclusively permissive license for distribution of all artifacts, or prefer a lightweight client-only SDK — this repo is a full-stack serving and kernel library with a nontrivial codebase and mixed licensing (Apache 2.0 + Modular Community License for some usage).

Information

  • Websitegithub.com
  • OrganizationsModular
  • Published date2023/04/28

More Items

Evaluates how well models and coding agents optimize operators for real Ascend NPU workloads, accounting for both speedups and token cost. Test cases preserve tensor shapes and data types from real model execution contexts.

Details the Xing 4 model architecture, Ascend-scale pretraining systems, and agentic post-training, including a 285B-parameter training run on 8,192 Ascend 910C cards.

GitHub
AI Infra2026

Create and manage Google Colab runtimes from the command line, then run local scripts or notebooks on remote CPUs, GPUs, or TPUs. It also handles file transfers and one-off jobs; supported on Linux and macOS.