Kernel benchmarks can reward fast code without showing whether an agent can improve an operator that matters in a deployed workload—or how much it spent finding that improvement. The key shift here is to connect each optimization task to real model execution contexts and score gains alongside token budget, making practical optimization efficiency part of the result.
Key Findings
- The benchmark covers 48 kernels in three difficulty levels, from basic memory-access optimizations to fused and attention computations. This progression tests more than the ability to generate an isolated kernel.
- Test cases use shapes and data types drawn from real training and inference tasks, with model, phase, hardware, and parallelism context. Measured kernel gains can therefore be interpreted in relation to actual operator invocations.
- It supports both model evaluation with a fixed built-in harness and model–harness evaluation with coding agents or custom systems. This helps separate model capability from the effects of an agent’s orchestration and tools.
- The Speedup–Token Score rewards final speedup while penalizing improvements obtained late in the token budget. It distinguishes systems that reach similar performance with different optimization costs.
Who It’s For—and the Trade-off
A good fit for researchers comparing LLMs and coding agents on Ascend kernel optimization, and for teams assessing whether local kernel gains justify their search cost. It is less useful if you need a hardware-agnostic benchmark or cannot access an Ascend execution environment. The report’s initial results cover Batch 1’s 17 tasks; the remaining batches are described as future evaluation work.