Most LLM fine-tuning uses gradient-based RL, but long-horizon agentic tasks stretch credit assignment and require heavy backprop memory—blocking full-parameter tuning on modest hardware. The core insight here is pragmatic: use evolution strategies (ES) to treat agent evaluation as a black-box, enabling full-parameter updates with only inference-time memory while keeping trajectory-level rewards intact.
Key Findings
- ES-based full-parameter fine-tuning scales to large LLM agents with much lower GPU memory requirements, enabling experiments that would be infeasible with RL's backprop stack.
- On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B with Agentic ESOpt improves the No Skill baseline by 6.69%.
- In online test-time heuristic/prompt co-evolution, Agentic ESOpt outperforms its matched baseline in 28 of 36 evaluated settings, showing practical gains from joint prompt–parameter adaptation.
- Method design choices include an online reward-weighted update for sampled perturbations and a cosine decay schedule for the perturbation scale σ to trade off exploration vs. rapid adaptation.
Who it's for and trade-offs
Great fit if you need to fine-tune large LLM agents on long-horizon, sparse-reward tasks but only have one or a few GPUs; the approach preserves trajectory-level credit without needing a learned critic or large GPU memory. Look elsewhere if you require lowest possible wall-clock GPU-hours for short single-turn tasks where gradient-based RL or LoRA-style methods may converge faster; ES can demand many forward-rollouts and careful population/evaluation budgeting. The paper highlights practical gains on agentic benchmarks and demonstrates scenarios (prompt–parameter co-evolution) where black-box perturbation is an advantage, but practitioners should plan evaluation budgets and perturbation schedules to control wall-clock cost.