1 hour ago
29
Microsoft Research published a study using its DELEGATE-52 benchmark that quantifies something practitioners have long suspected: frontier AI models degrade significantly when tasked with extended, multi-step workflows. The average document fidelity loss hit approximately 25% after just 20 delegated iterations. Zoom out across all models and domains tested, and that degradation figure climbs to roughly 50%.
The benchmark that broke the illusion
DELEGATE-52 was designed to measure something most AI benchmarks conveniently ignore. Standard evaluations test models on short, isolated tasks. Microsoft’s benchmark instead evaluates how well models preserve document content across long chains of delegated work, simulating real-world scenarios where models must iteratively edit, transform, and reconstruct documents or code across 52 professional domains.
The models tested include some of the most capable systems available: Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4. All of them stumbled.
Catastrophic corruption, defined as fidelity scores dropping to 80% or lower, appeared in over 80% of model-domain combinations evaluated.
One domain stood out as a rare bright spot. Python workflows maintained remarkable stability, showing less than 1% degradation across extended runs.
Gemini 3.1 Pro was deemed “ready” for only 11 out of the 52 professional domains the benchmark covers.
Tools make it worse, not better
Giving agents more capabilities, things like file read/write access and code execution, actually made the reliability problem worse. Agentic tool usage increased document fidelity degradation by an additional 6% on average.
The study also noted that verification and orchestration methods, the kinds of guardrails teams typically deploy to keep agents on track, haven’t solved the problem. Reliable long-horizon delegation remains a fundamental challenge, not merely an engineering inconvenience that better prompting can fix.
Why short benchmarks are misleading
The performance metrics that companies use to market their AI models, and that enterprises use to evaluate them, come overwhelmingly from short-horizon benchmarks. A model that scores 95% on a five-step task might score 50% on a twenty-step version of effectively the same work.
Parallel efforts in the industry underscore how widely recognized this problem is becoming. Frameworks like AgentRx and monitoring tools such as SentinelBench have emerged specifically to address reliability in extended agent operations.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.
English (US) ·