Why this matters Most prior work for training coding agents relies on development artifacts (issues, commits, docs) that limit task diversity. CodeMidas flips the script: it treats source code itself as the task spec, using agentic procedures to surface requirements, synthesize tests grounded in running the original implementation, and filter tasks via execution checks and adversarial rollouts. This lets practitioners scale diverse, verifiable RL environments directly from real codebases.
Key Findings
- Large, verifiable task bank: constructs 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains, providing varied training signals for coding agents.
- Execution-grounded verification: tests and expected behaviors are derived from executing the original implementation, improving verifier reliability while allowing alternative correct solutions.
- Empirical gains from RL training: training MiMo-V2.5 with GRPO on CodeMidas tasks improves performance on multiple external benchmarks (notable gains reported on DeepSWE, ProgramBench, Terminal-Bench), indicating transfer beyond the constructed tasks.
- Quality over scale: ablations show that increasing the number of high-quality, filtered tasks yields consistent performance improvements and that careful filtering outperforms larger uncleaned corpora.
Who it's for and tradeoffs
Great fit if you need diverse, execution-checked coding tasks to train or evaluate RL-based coding agents, or if you want dataset construction that preserves real project structure and dependencies. Look elsewhere if you require human-curated problem statements, formal specifications beyond behavioral tests, or guaranteed reproducibility in highly nondeterministic scientific code—CodeMidas focuses on extracting behavioral specs and verifiers from existing implementations and filters tasks empirically, which may still miss domain-specific subtlety.
Method sketch
The pipeline allocates agentic compute across stages: agents explore codebases to identify implemented functionality and propose behavioral task statements; they synthesize tests by running the original code to infer expected outputs and edge behavior; environment adaptations preserve project structure while removing the target implementation; finally, post-rollout filtering uses adversarial rollouts, solution reviews, and success-rate thresholds to select robust tasks. These steps aim to ensure verifiers accept correct alternative implementations and reject incorrect ones.