Tag

Monty

Runs LLM-generated Python in a Rust sandbox that starts in tens of microseconds (~60µs), with no container overhead. Filesystem, network, and environment access are blocked, and state serializes for pause/resume with per-run resource limits.

AirLLM

Runs 70B-class LLM inference on a single 4GB GPU without quantization and supports Llama3.1 405B on 8GB VRAM. Uses layer-splitting and block-wise model compression (4/8-bit) to reduce disk load and can speed up inference loading by up to ~3x; integrates with Hugging Face models.

Xorbits Inference (Xinference)

Run any open-source LLM, embedding, speech, image, or multimodal model behind one OpenAI-compatible API — swap GPT for an open model in a single line. Routes across vLLM, llama.cpp, GGML, and TensorRT, scaling from a laptop to a multi-node GPU cluster.

LMDeploy

Compresses, deploys, and serves LLMs via two engines: TurboMind for raw speed, a PyTorch engine for flexibility. Claims ~1.8x vLLM throughput through persistent batching, blocked KV cache, and split-and-fuse; ships 4-bit AWQ and KV-cache quantization.

MetaGPT

Orchestrates LLM-based roles (product managers, architects, engineers) to turn a one-line requirement into user stories, APIs and a starter code repo. SOP-driven multi-agent workflows with CLI and library APIs for prototype generation and agentic development.

Open Interpreter

Lets LLMs run code and control a user’s computer via natural language (Python, JavaScript, Shell, etc.) with interactive approval. Supports local or hosted models, terminal and Colab/Codespaces integrations, streaming output, and configurable safety/auto-run options.

LiteLLM

Calls 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure — through one OpenAI-compatible API, as a Python SDK or self-hosted proxy. The proxy adds virtual keys, spend tracking, rate limits, and load balancing across models and providers.

OpenRLHF

Trains LLMs with RLHF at scale by splitting actor, critic, reward, and reference models across separate GPU groups via Ray, with vLLM-accelerated generation and DeepSpeed ZeRO-3. Supports PPO, GRPO, REINFORCE++, DPO, plus async and agentic multi-turn RL.

DeepEval

Framework for unit-testing, evaluating and benchmarking LLM systems with ready-made metrics (G‑Eval, hallucination, task completion), support for local judge models and synthetic datasets, plus CI-friendly integrations for LangChain/OpenAI/Anthropic.

GeoAI

Applies deep learning workflows to geospatial data, covering imagery search, dataset preparation, model training, inference, visualization, and QGIS integration for remote sensing.

Claude Cookbooks

Runnable Jupyter notebooks for building with the Claude API: tool use, RAG, vision, prompt caching, sub-agents, classification, summarization, and integrations like Pinecone and Voyage embeddings. Copy-paste recipes that drop into real projects.

kotaemon

Chat with your documents via retrieval-augmented generation; each answer carries inline citations and a built-in viewer highlights the cited PDF passage. Pairs full-text with vector search and runs on OpenAI, Azure, Cohere, Ollama, or local models.