Most progress in model tuning comes from many small, incremental experiments; autoresearch asks a provocative question: what if the loop of propose→implement→train→evaluate could be handed to an LLM agent and run unattended overnight?
What Sets It Apart
- Closed, time‑bounded experiments: every experiment runs for a fixed 5‑minute wall‑clock budget, making experiments comparable across different local hardware and encouraging changes that yield quick, measurable improvements (val_bpb is the optimization metric). This design yields roughly 12 experiments per hour on a single GPU and ~100 overnight iterations.
- Single editable surface for automation: the agent is permitted to modify only one file (train.py). That keeps the search space small and diffs reviewable while letting the agent change architecture, optimizer, hyperparams, and batch sizing.
- Human-in-the-loop research org via markdown: humans write and iterate program.md to shape agent behavior, constraints, and objectives rather than hand-editing Python code for each experiment. This separates strategy (human) from brute‑force exploration (agent).
Who This Fits — and Tradeoffs
Great fit if you want to: run rapid, automated hyperparameter/architecture searches on a single GPU; explore emergent edits an agent might try; or prototype ideas overnight without manual iteration. Look elsewhere if you need: production‑scale training, multi‑GPU/distributed experiments, guaranteed reproducibility across different hardware stacks, or formal experiment tracking at enterprise scale. The repo intentionally favors simplicity and quick feedback over platform generality.
Where It Sits in the Tooling Landscape
It complements single‑GPU training sandboxes (like nanochat) and lightweight local ML workflows: think of it as an automation layer that turns your small training harness into an autonomous experiment generator. Compared with full MLOps platforms, it trades telemetry, orchestration, and cross‑run comparability for extreme simplicity and a very small attack surface.
How It Works (High Level)
Agents receive context from program.md, run a prepare step (data/tokenizer), then iterate: modify train.py, run a 5‑minute training job, evaluate on val_bpb, and keep or discard changes. The repo is self‑contained (PyTorch + minimal deps) and was tested on an NVIDIA H100; smaller‑compute forks exist to tune defaults for CPUs and MPS.
Created by Andrej Karpathy (repo created 2026‑03‑06). Use it as a research sandbox to discover short‑horizon improvements and to prototype agent‑driven experiment workflows rather than as a drop‑in replacement for large‑scale training pipelines.
