Most medical benchmarks score only final answers; this makes it easy for models to appear clinically competent while skipping the operational steps that keep patients safe. Digital Hospital shifts the focus from single-shot correctness to stepwise clinical work: triage, chart review, targeted investigations, mandatory evidence checks, role-appropriate tool calls, cross-role communication, and an auditable trajectory that the grader evaluates deterministically.
What Sets It Apart
- Emphasis on process over outcome: rewards and final grades incorporate mandatory evidence use, critical-finding flags, tool-permission compliance, and diminishing returns for repeated low-value calls—so a correct diagnosis without the right workflow scores poorly.
- Runtime + data, not a prompt: includes a FastAPI server, role state machine, 47 patient cases with hidden answer keys, 550 MCQs, 55 tools, and deterministic graders so evaluations are reproducible and suitable for rollout collection.
- Trajectory-first contract: every model action, observation, reward and error is recorded in an explicit schema for offline RL, supervised fine-tuning, preference learning, and failure analysis.
- Multi-role, multi-agent realism: 11 roles (specialists, research, director) with distinct permissions and cross-role inboxes captures handoffs, escalation, and organisational context often missing from vignette benchmarks.
Who it's for and trade-offs
Great fit if you need a controlled, reproducible environment to benchmark or improve agentic clinical behaviour (triage, tool use, evidence synthesis) with dense, step-level feedback. Researchers building offline RL datasets, process-supervision pipelines, or multi-agent hospital workflows will get the most value.
Look elsewhere if you need a clinical decision-support tool for real patients, very large-scale patient corpora, or free-form natural-language-only evaluations—Digital Hospital is synthetic, deliberately prescriptive, and designed as a research benchmark, not a deployment system.
Where it fits
Use it when you want to evaluate whether a model can operate like a clinician under operational constraints (role permissions, step budgets, mandatory investigations) and to generate reproducible rollouts for policy improvement, not just aggregate accuracy numbers.
