2 hours ago
25
Microsoft just opened the hood on GitHub Copilot at a scale nobody has attempted before. A new research paper from Microsoft Azure Research and the University of Illinois Urbana-Champaign analyzes 13.5 million coding sessions from 3.2 million users, and the findings suggest the infrastructure running AI coding agents needs a serious rethink.
The study, titled “Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale,” draws on one week of production traces from June 2026. During that window, Copilot generated 761 million LLM calls, 775 million tool invocations, and processed 95 trillion tokens. Those numbers make this the first large-scale empirical look at how AI coding agents actually behave in the wild, not in controlled benchmarks or synthetic tests.
The cache problem nobody planned for
In a typical Copilot session, users issue sparse “turns,” which are essentially prompts or instructions. Each turn kicks off an autonomous loop where the LLM makes calls and executes tools at roughly a 1:1 ratio. The median session involves about 15 LLM calls per turn, with the mean exceeding 40 in some scenarios. About 87% of all LLM calls are agent-initiated, not user-initiated.
KV-cache, the mechanism that lets language models avoid recomputing previous context, hits at roughly 90% within a single turn. But once a user issues a new turn, the hit rate drops to around 55%. Switch models or compact the context window, and it craters to 8%.
The paper argues for turn-level or even session-level scheduling, keeping cache-hot workloads together rather than scheduling at the individual request level.
Five types of users, fifty-fold differences
The researchers identified five distinct user archetypes, and the gap between the lightest and heaviest users spans a 50-fold range in token consumption.
The study also uncovered tool-failure-driven retry loops. When a tool call fails, the agent retries, sometimes aggressively. These retry cascades can amplify compute workload demands by up to four times.
Idle time is the hidden opportunity
The researchers found substantial idle time baked into sessions, and that 86-90% of total idle time could be captured by lightweight predictors. That predictability opens the door to proactive resource reclamation, freeing up GPU memory and compute for other users during those gaps rather than holding everything in reserve.
What this means for AI infrastructure
Microsoft’s findings suggest three concrete shifts. First, scheduling should move from per-request to per-turn or per-session granularity, keeping cache-hot workloads together. Second, infrastructure needs to account for the retry amplification problem, building headroom for tool-failure cascades rather than treating them as anomalies. Third, idle-time prediction should become a first-class infrastructure concern, not an afterthought.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.
English (US) ·