Why this matters
Training coding agents to build programs from scratch requires environments that mirror real software engineering: multiple files, dependencies, runtime sandboxes, and behavioral tests. ProgramGym addresses this gap by automating the conversion of real GitHub repositories into verifiable training environments and by providing an agent workflow designed for long-horizon development.
Key Findings
- Program-Env: a feedback-driven pipeline that separates online dependency preparation from offline compilation, repairs sandbox builds, and generates black-box behavioral tests across modules to produce reference executables and validated test suites.
- Scale: constructs 1,648 executable training environments spanning C, C++, Go, and Rust, each including runtime sandboxes, usage documentation, and black-box tests suitable for supervised and reinforcement-style training.
- Program-Agent: a role-isolated iterative workflow (analysis → implementation → review) with persistent evidence records, offline build checks, and revision management to support long-horizon development and failure recovery.
- Empirical impact: fine-tuning on ProgramGym trajectories raises Qwen3.8-27B from 34.87% to 42.84% Tests Passed; using Program-Agent further increases base and SFT models to 55.71% and 56.25%, demonstrating substantial gains on program reproduction benchmarks.
Who it's for and trade-offs
Great fit if you need realistic, verifiable training environments for coding agents, want to train or evaluate long-horizon program synthesis workflows, or require a repeatable pipeline to convert real projects into sandboxed tasks. It is especially relevant for researchers building, fine-tuning, or benchmarking coding agents on multi-file, dependency-heavy codebases.
Look elsewhere if you only need short, isolated algorithmic problems or simple single-file code generation: ProgramGym focuses on engineering-scale tasks and the overhead of building sandboxes, tests, and offline compilation may be unnecessary for toy or micro-benchmarking scenarios.
Where it fits
ProgramGym sits between synthetic programming benchmarks and full software engineering practice: it provides scaled, verifiable environments that preserve repository complexity while adding automated test and sandbox scaffolding to enable supervised trajectories and agent-in-the-loop development research.
