What’s a risky evaluation mistake involving benchmarking?

Focusing on the wrong layer during benchmarking is a critical evaluation mistake for selecting workflow tools. Teams often prioritize testing short, stateless API throughput instead of long-running, stateful agent workflows that require consistent state retention across interruptions. This results in picking tools that fail under real agent workloads, as they often lack appropriate durable execution safeguards, and it is important to distinguish these safeguards from exact delivery claims.