AIAny
Icon for item

Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

Fine-tunes long-horizon LLM agents with evolution strategies so full-model updates run at inference-level GPU memory. Emphasizes trajectory-level credit via black-box rewards, online prompt–parameter co-evolution, and a cosine decay for perturbation scale to balance exploration and adaptation; suited for limited-GPU settings.

Introduction

Most LLM fine-tuning uses gradient-based RL, but long-horizon agentic tasks stretch credit assignment and require heavy backprop memory—blocking full-parameter tuning on modest hardware. The core insight here is pragmatic: use evolution strategies (ES) to treat agent evaluation as a black-box, enabling full-parameter updates with only inference-time memory while keeping trajectory-level rewards intact.

Key Findings
  • ES-based full-parameter fine-tuning scales to large LLM agents with much lower GPU memory requirements, enabling experiments that would be infeasible with RL's backprop stack.
  • On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B with Agentic ESOpt improves the No Skill baseline by 6.69%.
  • In online test-time heuristic/prompt co-evolution, Agentic ESOpt outperforms its matched baseline in 28 of 36 evaluated settings, showing practical gains from joint prompt–parameter adaptation.
  • Method design choices include an online reward-weighted update for sampled perturbations and a cosine decay schedule for the perturbation scale σ to trade off exploration vs. rapid adaptation.
Who it's for and trade-offs

Great fit if you need to fine-tune large LLM agents on long-horizon, sparse-reward tasks but only have one or a few GPUs; the approach preserves trajectory-level credit without needing a learned critic or large GPU memory. Look elsewhere if you require lowest possible wall-clock GPU-hours for short single-turn tasks where gradient-based RL or LoRA-style methods may converge faster; ES can demand many forward-rollouts and careful population/evaluation budgeting. The paper highlights practical gains on agentic benchmarks and demonstrates scenarios (prompt–parameter co-evolution) where black-box perturbation is an advantage, but practitioners should plan evaluation budgets and perturbation schedules to control wall-clock cost.

Information

  • Websitearxiv.org
  • AuthorsZhi Zheng, Rongsheng Chen, Yunpeng Ba, Zhenkun Wang, Yee Whye Teh, Wee Sun Lee
  • Published date2026/08/18

More Items

Blocks tool-using agents from directing emails, payments, and other side effects to unauthorized targets by checking typed tool-return fields. Deterministic rules also remove instruction-like text before the model reads it.

Provides six open language models from 0.9B to 375B-A23B for reasoning and agentic tasks. The paper also introduces sparse attention and a decoding speedup that needs no separate draft model.

Reduces Transformer parameter count by repeatedly applying shared blocks, then combines their updates with geometric decay and learned attention. On ClimbMix, it beats standard baselines in validation loss with fewer Transformer parameters.