Scientific software encodes domain knowledge and data semantics that make simple patch-based evaluation misleading: a passing unit test can still break scientific contracts. SWE-bench Science surfaces this gap by forcing coding agents to edit real repositories while preserving units, file formats, numerical invariants, and end-to-end behavior.
Key Findings
- Benchmark scope and setup: 119 tasks drawn from 98 GitHub repositories spanning 20 scientific domains, organized into three paradigms (Issue-driven, Expert-exploratory, Engineering-integration). Each task ships a baseline environment image and a separate verifier image; images are immutable and pinned by digest for reproducible evaluation.
- Performance and difficulty: Even the best agent evaluated (Claude Code with Opus-5 (max)) achieves pass@1 below 50%, showing that repository-level scientific repairs remain challenging for current coding agents.
- Failure mechanisms: Four recurring modes are identified — lack of scientific abstraction/knowledge, surface-level or misguided exploration, incomplete system-level integration or coverage, and failures to generalize scientific reasoning beyond observed examples.
- Ablation insight: Removing explicit scientific guidance (while preserving repository and execution context) yields mixed effects — accurate, well-aligned scientific guidance can constrain search and improve efficiency, but misaligned guidance can anchor agents and hurt exact-repair success.
Who it's for and tradeoffs
Great fit if you research coding-agent capabilities, evaluate LLM-driven code repair in domain-rich settings, or need a repository-level benchmark that stresses scientific correctness beyond unit tests. Look elsewhere if you need micro-level function-completion tasks or synthetic toy benchmarks; SWE-bench Science emphasizes end-to-end engineering context, reproducible images, and held-out verifiers rather than isolated function synthesis. The benchmark favors experiments that integrate execution, pinned environments (Docker images), and domain-aware verification over purely prompt-based code generation.