Category

Pathway

Unified Python framework where the same code runs on batch and streaming data, backed by a Rust engine on Differential Dataflow for incremental computation. Aimed at ETL, analytics, and live RAG pipelines over Kafka and 300+ connectors.

Activepieces

Builds no-code automations with TypeScript-based integrations, AI pieces, human-in-the-loop steps, and MCP exposure for community and product workflows.

OpenObserve

Centralizes logs, metrics, traces, frontend RUM and LLM observability into one self-hostable platform, using Parquet + S3-native storage and SQL/PromQL querying to reduce long‑term storage costs and unify telemetry analysis.

Dify: Leading Agentic Workflow Builder

Build LLM apps by chaining nodes on a visual canvas — prompts, branching, RAG, agents, tools — and ship the same graph as an API or hosted app. Bundles a plugin marketplace, model routing across hosted and local providers, and built-in observability.

promptfoo

Declarative CLI and library to evaluate and red-team LLM apps: run test cases against prompts and models, compare providers side-by-side, and scan for jailbreaks, prompt injection, and data leaks — with CI/CD and pull-request code scanning built in.

Ragas

Evaluates and tests LLM apps — RAG pipelines, agents, and workflows — using objective metrics that mix LLM-as-judge scoring with deterministic measures. Auto-generates synthetic test datasets and integrates with LangChain and tracing tools.

Opik

Provides end-to-end observability, evaluation, and optimization for LLM-based applications by tracing model calls, running automated evaluations, and surfacing production metrics. Ships SDKs, broad framework integrations, LLM-as-a-judge metrics, and dashboards to support development, CI, and production monitoring.

Langfuse

Tracks, evaluates, and debugs LLM applications with traces, prompt management, datasets, playgrounds, and observability that can run in cloud or self-hosted setups.

Continue

Open-source AI coding assistant for VS Code and JetBrains that bundles autocomplete, chat, inline edit, and an agent mode behind one config, letting each capability use any model provider rather than a single locked-in vendor.

DeepEval

Framework for unit-testing, evaluating and benchmarking LLM systems with ready-made metrics (G‑Eval, hallucination, task completion), support for local judge models and synthetic datasets, plus CI-friendly integrations for LangChain/OpenAI/Anthropic.

BISHENG

Self-hostable platform for building enterprise GenAI apps with visual workflow orchestration — loops, parallelism, human-in-the-loop — plus RAG, agents, unified model management, and in-house OCR for handwriting and rare characters.

VLMEvalKit

Runs one-command evaluation of vision-language models across 80+ multimodal benchmarks, handling data download, inference, and metric scoring in a single pass. Supports 220+ LMMs; adding a new model means writing one generate_inner() function.