DeepSeek’s V4 Flash struggles with real-world tasks despite topping AI leaderboards

DeepSeek’s V4 Flash has been called a “total monster” by developers since its July 31 launch. The model shot to the top of multiple AI leaderboards, and its pricing, at $0.14 per million input tokens and $0.28 per million output tokens, undercuts comparable models by roughly tenfold.

In practice, the monster has a limp. When testing firm Composio ran V4 Flash through a battery of real-world agent tasks, the model managed a 53.8% pass rate. Out of 240 total runs spanning 30 deliberately difficult, multi-step workflows, only 129 passed. Just six of the 30 workflows were completed successfully by every agent harness tested.

What the Composio tests actually measured

Composio’s evaluation tested V4 Flash across eight different agent harnesses, including Claude Code, Codex, and OpenCode. The 30 tasks involved live tools that developers actually use every day: Gmail, GitHub, Slack, and Google Sheets.

Results varied dramatically depending on which harness was used. Pi Agent emerged as the strongest performer, completing 20 out of 30 tasks. The same underlying model can look brilliant or mediocre depending entirely on how it’s integrated into a workflow.

The benchmark-to-reality gap persists

DeepSeek is currently offering V4 Flash in public beta, with a broader adjustment to API pricing scheduled for August 16, 2026. The beta label matters here. It signals that even DeepSeek views the model as a work in progress, not a finished product ready for mission-critical deployment.

Why cost disruption doesn’t automatically equal adoption

At $0.14 per million input tokens, V4 Flash is roughly a tenth the cost of comparable models from major US competitors. But if a model fails half the time on complex agent tasks, the savings evaporate quickly. Every failed task means wasted compute, wasted developer time debugging, and potentially wasted end-user trust.

For developers evaluating V4 Flash, the Composio data offers a useful framework. The choice of agent harness matters as much as the choice of model. Pi Agent’s 20-out-of-30 success rate versus the weaker harnesses shows that thoughtful integration can partially compensate for model limitations.

Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.