This dataset surfaces an often-overlooked insight: agentic long-context workloads are dominated by cached prefill tokens, not fresh KV compute. The traces were produced by a Codex agent running swebenchpro-style workloads across many open-source repos and include per-call token breakdowns, cache statistics, timing, and pass/fail outcomes — data that helps quantify where LLM compute is actually spent.
What Sets It Apart
- High-resolution per-call telemetry: includes input, cached, computed (uncached) and output token counts for 20,230 LLM calls across 610 successful trials, letting you measure compute concentration and heavy-tail behavior. (So what: enables empirical analysis of which calls drive uncached compute and cost.)
- Explicit cache analysis: an overall cache hit rate of 94.2% with turn-by-turn and cross-trial breakdowns. (So what: lets researchers evaluate caching strategies and estimate real KV compute under realistic agent workloads.)
- Workload & timing signals: inter-call delays, trial durations, and per-repo success/failure statistics (731 trials total, 610 successful) are included. (So what: supports simulation of agent latency, throughput, and failure modes.)
- Compact, machine-friendly format: JSON with dataset tags indicating pandas/polars compatibility and MIT license. (So what: easy to load, filter, and integrate into analysis pipelines.)
Who It's For & Tradeoffs
Great fit if you are studying LLM operational cost, cache strategies, agent orchestration, or long-context behavior and want grounded, per-call telemetry rather than synthetic benchmarks. It is also useful for validating simulator assumptions about context growth, output sizes, and inter-call timing.
Look elsewhere if you need very large-scale public telemetry (this dataset is compact with modest download counts) or labeled human evaluations of output quality; the traces focus on token/timing/cache metrics and pass/fail outcomes from automated trials rather than nuanced human judgments.
Where It Fits
Use this dataset to: (a) validate cache-utility models and cost estimators, (b) reproduce heavy-tail uncached-compute analyses, and (c) build realistic agent workload generators for benchmarking inference systems. It complements model-centric benchmarks by exposing system-level and agent-driven dynamics that determine real-world LLM costs.
