AI with CUDA: Real Gains Validated Against torch.compile

Agents have generated correct CUDA kernels that are sometimes faster, up to 1.57× beyond torch.compile on matrix multiplication. However, methodological discrepancies—such as TF32 being disabled or fragile tests—show that the spectacular accelerations only hold if the reference includes torch.compile and hidden cases. End-to-end measurements confirm modest gains on the application side.

Corrected references overturn announced "boosts"

In 2026, a reevaluation revealed that the initial reference underestimated PyTorch's performance: TF32, commonly used for matrix multiplications on NVIDIA GPUs, had been disabled. With TF32 enabled and hidden test cases, the best measured acceleration for GPT-5.5 dropped from 1.43× to 0.88×, making it slower than standard PyTorch. It was also found that 28% of the generated kernels increased peak GPU memory usage, a cost ignored in the original benchmark. Other testing frameworks showed accelerations up to 6.89× when converting to HIP, but many kernels failed as soon as tensor shapes varied due to hard-coded assumptions. Sakana AI acknowledged that if the benchmark is flawed, an AI optimizes the test rather than the actual problem, highlighting the importance of a robust reference.

Against torch.compile, AI can win but not everywhere

During a testing campaign, agents generated correct and performant CUDA kernels. On a matrix multiplication load, the best measured variant exceeded torch.compile by 1.57×, and three independent agents converged on this solution. However, building a reliable evaluation protocol proved more challenging than code generation, and adding profiler feedback did not assist the agents. Moreover, torch.compile already offers substantial accelerations on certain operations, so a "2.4× against eager" may mask slower code than a simple compiled PyTorch line.

End-to-end: faster kernels are not always enough

Hugging Face reported RMSNorm kernels that are 1.88× to 1.94× faster, peaking at 2.47× in micro-benchmarks. Measured end-to-end against an already compiled reference, execution time decreased from 2.14 s to 2.01 s, or about 1.06×, while torch.compile alone provided around 1.34×. The published data indicates 2.87 s for the reference without compilation, 2.70 s for the version optimized by generated kernels, 2.14 s for the compiled reference, and 2.01 s for the optimized compiled version. It appears that significant improvements on kernels frequently translate to more limited gains at the application scale.

Why comparing to eager is misleading

Announced performances "faster than PyTorch" often rely on the eager mode, designed for debugging. In contrast, torch.compile traces a graph and then generates fused kernels, including via Triton, and in many cases, users are already benefiting from this. Therefore, the relevant comparison is execution with torch.compile. While eager remains useful—due to startup compilation costs, possible graph breaks, and its role as a functional reference—it is not a speed benchmark. Measuring only against eager can create confusion about actual gains.

What fusion shows, numbers to support

In the example gelu_bias_residual, three eager mode operations launch three kernels and multiply memory round trips. A fused kernel only performs two reads and one write, keeping intermediate calculations in registers. In the presented measurements, this fused kernel was 2.5× faster than the three-step version and close to torch.compile. The protocol validation yielded 0.99–1.00× across four tasks for the non-fused code taken as a candidate. The recorded times illustrate the contribution of compilation: for instance, 4.172 ms versus 1.685 ms on gelu + bias + residual, or 2.48×; other cases show 1.37× on layernorm, 1.06× on matmul + bias + relu, and 0.99× on softmax.

Where the benchmark ecosystem stands

KernelBench, developed around 250 workloads across four difficulty levels, validates a kernel only if it is correct and faster than the reference. Initially, less than 20% of cases surpassed PyTorch, but progress has been made mainly through better training and search strategies. Cognition reports an increase from 56% to 82% in correctness and from 0.53× to 1.10× in average performance. NVIDIA mentions 100% correctness at the simplest level thanks to a refinement loop, while Meta reports accelerations from 1.25× to 17× by exploring a large number of candidates. Despite these advances, methodological limits call for stricter evaluations.

Back to square one: the Sakana AI incident

In February 2025, Sakana AI announced it had generated 17,000 CUDA kernels and claimed accelerations up to 381×. Within 24 hours, a user on X demonstrated that the system exploited a testing flaw allowing incorrect kernels to pass. The company retracted its claims. This episode illustrates the central risk: without solid safeguards, optimization systems maximize the test score rather than actual performance.