Small, repeatable wins are often buried in manual tweak-and-measure loops. Automating that loop — propose, run, measure, keep or revert — both accelerates discovery and preserves auditability. This extension equips a terminal-native AI coding agent with that exact workflow so micro-optimizations can be explored continuously and reproducibly.
What Sets It Apart
- Extension + skill separation: the extension provides run/log tools and a live widget, while the skill encodes domain knowledge (how to form experiments, session files, and finalize kept changes). This keeps the core loop domain-agnostic while letting you swap domain logic.
- Persistent, branch-aware history: experiments append to autoresearch.jsonl and a session document so a fresh agent can resume a loop without cached context. Kept experiments can be split into independent reviewable branches, making autonomous changes review-friendly.
- Statistical guardrails: computes a MAD-based confidence score after a few runs to highlight likely real improvements vs noise, and supports optional backpressure checks to block unsafe keeps (tests/types/lint).
- Practical scope: designed for optimization targets that are easy to run locally or in short jobs — test runtime, bundle size, build speed, lighthouse scores, or short training runs — with tooling to time commands and capture structured metric lines.
Who It's For (and tradeoffs)
Great fit if you already use the pi terminal agent and want to automate repetitive optimization cycles without building orchestration yourself. It’s useful for maintainers who want machine-assisted idea generation plus automated measurement and safe git workflows (auto-commit, revert, branch finalization).
Look elsewhere if you need large-scale distributed experiment orchestration, guaranteed low-variance ML training at production scale, or a web GUI-first experience — this is terminal-first and assumes short, repeatable measurements and an LLM provider configured for pi. Autonomous loops can also increase API/token spend and may require extra run repetition to overcome noisy metrics.
Where it fits
Best used as a lightweight, developer-facing layer on top of existing projects: start sessions with a focused objective (metric, command, files in scope) and let the agent run iterative experiments. Use hooks and checks to integrate external reporting, notifications, or stricter correctness gates.
How it works (brief)
The skill creates session artifacts (autoresearch.md, autoresearch.sh, autoresearch.jsonl). The extension offers tools: init_experiment, run_experiment (times commands and parses METRIC lines), and log_experiment (records, auto-commits, updates widget). After a session, a finalize skill can group kept changes and create independent branches from the merge-base for human review.
