AIAny - AI Infra

LLM Transparency Tool

2023

facebookresearch, Igor Tufanov +3

LLM Transparency Tool (LLM-TT) is an open-source interactive toolkit from Facebook Research for analyzing the internal workings of Transformer-based language models. It lets users run inferences, explore contribution graphs tied to selected tokens, inspect representations after any block, and drill down to attention heads, FFN blocks and individual neurons to see how they promote or suppress output tokens. The project provides Docker and local installation instructions and a live demo hosted on Hugging Face.

github llm NLP xai ai-tools+3

Claude-Mem

2024

Alex Newman, thedotmack

Claude-Mem is a persistent memory compression system built for Claude Code. It automatically captures tool usage observations during coding sessions, generates semantic summaries using Claude's agent-sdk, and injects relevant context into future sessions to maintain continuity of project knowledge.

claude ai-coding ai-agent plugin ai-tools+3

Daytona

2024

Daytona (daytonaio)

Daytona is a secure and elastic infrastructure for running AI-generated code. It provides fast, isolated sandboxes (sub-90ms creation), SDKs for Python and TypeScript, programmatic control (file, Git, LSP, execute APIs), OCI/Docker compatibility, and persistent sandboxes. Designed to safely execute model-generated or untrusted code at scale, Daytona emphasizes isolation, performance, and developer-friendly APIs. It is open-source under the AGPL-3 license.

mlops ai-inference ai-serving ai-workflow ai-client+5

FastVideo

2024

The FastVideo Team, hao-ai-lab

FastVideo is a unified post-training and inference framework for accelerated video generation. It supports end-to-end pipelines for diffusion models, from data preprocessing to training, finetuning, distillation, and inference. Key features include sparse distillation for over 50x denoising speedup, optimizations like Video Sparse Attention, and support for various hardware and OS.

github ai-video pytorch huggingface ai-tools+3

LMCache

2024

LMCache

LMCache is an open-source, high-performance KV (key-value) cache layer designed to accelerate LLM serving and inference, especially for long-context scenarios. By storing and reusing KV caches across GPU, CPU DRAM and local disk, and enabling cross-instance sharing, LMCache reduces time-to-first-token (TTFT) and GPU usage. It integrates tightly with vLLM, supports P2P cache sharing, non-prefix caches, multiple storage backends (CPU, disk, NIXL), and is distributed under Apache-2.0.

LLM llm ai-serving ai-inference ai-library+4

TEN Framework

2024

TEN-framework

TEN Framework is an open-source framework for building real-time multimodal conversational voice AI agents. It supports low-latency, high-quality interactions with components like STT, LLM, and TTS, and includes extensible agent examples such as voice assistants, lip-sync avatars, and speech diarization. It integrates with services like Agora, OpenAI, and Deepgram, and its ecosystem features VAD, turn detection, and portal tools for real-time communication and hardware integration.

ai-framework ai-agent chatbot audio ASR+3

EXO

2024

exo-explore (GitHub), exo labs

EXO is an open-source project that connects everyday devices into a local AI cluster for running and accelerating large models. It features automatic device discovery, topology-aware automatic parallelism, tensor sharding, MLX-based distributed inference, and Day-0 support for RDMA over Thunderbolt to reduce inter-device latency.

github mlops ai-inference ai-serving ai-development+1

MinerU

2024

OpenDataLab

MinerU is an open-source tool that transforms complex documents like PDFs into LLM-ready markdown or JSON formats. It excels in layout analysis, text recognition, formula and table parsing, removing headers/footers, supporting multilingual OCR, converting formulas to LaTeX, and tables to HTML, ideal for AI agentic workflows.

github ai-tools llm ai-development ai-library

BitNet (bitnet.cpp)

2024

Microsoft

BitNet (bitnet.cpp) is Microsoft's open-source inference framework for 1-bit large language models (LLMs). It provides optimized kernels for fast, lossless inference of 1.58-bit models on CPU (and GPU support added later), delivering substantial speed and energy improvements on ARM and x86. It integrates with Hugging Face models, includes build/run/benchmark tools, and aims to enable running large low-bit models locally (e.g., a 100B BitNet model on a single CPU at human-reading speeds).

microsoft github llm ai-inference ai-serving+4

NexaSDK

2024

NexaAI

NexaSDK is a cross‑platform developer toolkit and low‑level inference engine (NexaML) for running AI models locally on NPUs, GPUs and CPUs. It supports GGUF, MLX and .nexa model formats, provides Day‑0 support for new architectures, multimodal capabilities (text, vision, audio), mobile SDKs (Android/iOS), OpenAI‑compatible APIs, and optimized NPU support.

github ai-inference ai-serving ai-client ai-framework+5

Foundry Local

2024

Microsoft

Foundry Local is an open-source tool from Microsoft that enables running generative AI models on local devices without an Azure subscription. It supports on-device processing for privacy and security, integrates models via an OpenAI-compatible API, and optimizes performance using ONNX Runtime and hardware acceleration.

microsoft ai-inference ai-serving llm ai-client+2

cuTile Python

2024

NVIDIA CORPORATION

cuTile Python is a programming model for writing parallel kernels for NVIDIA GPUs, built primarily in Python with a C++ extension. It enables efficient GPU programming, requires CUDA Toolkit 13.1+, and supports installation via PyPI or from source.

nvidia ai-development ai-library ai-train

Category

Explore by categories

All

AI Leaderboard

AI Agent Tutorials

AI Coding Tutorials

AI Agent Papers

Chatbot

Machine Learning Foundation Books

AI Train

AI Deploy

AI Client

Machine Learning Foundation Papers

Machine Learning Foundation Tutorials

AI Image Demos

AI Agent

Large Language Model Tutorials

Large Language Model Papers

Machine Learning Engineering Papers

Computer Vision Tutorials

Computer Vision Papers

Natural Language Processing Papers

Reinforcement Learning Papers

Speech Technology Papers

AI API

AI Coding

AI Image

AI Video

MLOps

MCP Client

MCP Server

AI Video Papers

AI Audio

AI Infra

LLM Transparency Tool

Claude-Mem

Daytona

FastVideo

LMCache

TEN Framework

EXO

MinerU

BitNet (bitnet.cpp)

NexaSDK

Foundry Local

cuTile Python