In July 2025, METR — an AI research nonprofit, not a vendor with something to sell — ran a randomized controlled trial on experienced open-source developers working on codebases they already knew well. Half their tasks, they could use AI tools (mostly Cursor with Claude 3.5/3.7 Sonnet). Half, they couldn't. The result surprised almost everyone, including the researchers: developers were 19% slower when allowed to use AI.
Here's the part that should actually change how you think about this: before the study, those same developers predicted AI would make them 24% faster. After finishing — having just been measurably slower — they still estimated AI had made them 20% faster. Measured reality and self-reported experience pointed in opposite directions, and the gap didn't close even with the result sitting right in front of them.
Why This Isn't the "AI Is Overhyped" Story It Sounds Like
It would be easy to read that stat and conclude AI coding tools are snake oil. That's not what the study shows, and it's not what we've seen building software with AI assistance daily at DIGIT. A few things matter here that the headline number doesn't capture:
The developers were experts on the specific codebase already. These weren't unfamiliar tasks — they were maintainers working in repositories they knew intimately, often with 5+ years of prior context. AI has the least relative advantage exactly there: when a human's own memory of the codebase is already faster than explaining that context to a model or reviewing its output against it. That is a genuinely different situation from a developer working in unfamiliar code, prototyping something new, or handling a repetitive pattern they'd otherwise have to write by hand.
Review and correction time is invisible until you measure it. The time cost the study picked up — and that developers didn't feel while it was happening — is the tax of reading AI-generated suggestions, catching the subtly wrong ones, and correcting course. That tax is real. It's also exactly the kind of cost that's easy to underestimate in the moment and only shows up when someone puts a stopwatch on the whole workflow, not just the parts that feel fast.
METR itself updated the finding. This is the detail we find most credible about the whole study: in February 2026, METR published a follow-up noting that 30–50% of invited developers had declined to participate without AI access — a selection bias that likely skewed the original sample toward developers who'd struggle more without their usual tools. Their revised methodology, with a larger cohort, found a much smaller effect: roughly 4% slower, with a confidence interval wide enough to include a small speedup. A research group that publicly revises its own headline finding when it finds a flaw is a research group worth taking seriously — including on the original result.
What This Actually Means If You're Deciding Whether to Use AI Coding Tools
The honest, unglamorous answer: it depends entirely on the task, and pretending otherwise — in either direction — is the actual mistake. AI coding assistants tend to help most on unfamiliar code, boilerplate-heavy work, and well-specified, narrow tasks where the model's output is easy to verify quickly. They tend to help least — and can genuinely slow you down — on deep, familiar codebases where an experienced engineer's own mental model is faster than any tool's, and where verifying a subtly wrong suggestion costs more than writing the code directly would have.
This tracks with what we cover in our read on agentic AI adoption data across the industry: the gap between AI adoption and AI ROI is wide, and it's wide for the same reason the METR result is nuanced — the tools help unevenly, and treating them as a uniform productivity multiplier regardless of task is where both the overhype and the "AI doesn't work" backlash both go wrong.
How We Actually Use AI Coding Tools at DIGIT
We didn't stop using Claude and other AI coding assistants after reading this study, and we're not telling clients to avoid them either — that would be its own kind of dishonesty, reacting to one data point as hard as the original hype reacted to none. What changed is where we lean on them. Boilerplate, test scaffolding, first-draft implementations of well-specified features, and unfamiliar-library integration work are where AI assistance measurably speeds up our delivery. Deep debugging in a codebase our engineers already know cold, and architecture decisions with long-term consequences, stay human-led with AI as a second opinion at most — because that's where the METR data, and our own experience, both say the juice isn't worth the squeeze.
The Real Lesson Isn't About AI
The METR study is as much a lesson about measurement as it is about AI: humans are bad at accurately sensing their own productivity in the moment, in either direction. That should make you skeptical of any confident claim about AI's impact on developer output — including ours — that isn't backed by something more rigorous than a gut feeling or a vendor's case study. We'd rather point you at a study that complicates our own pitch than pretend the data is simpler than it is.
Frequently Asked Questions
Does AI actually slow down developers? For experienced developers working in codebases they already know deeply, a rigorous randomized trial found a measurable slowdown in one cohort (19%), though a later, larger cohort from the same research group found a much smaller effect (around 4%) after correcting for selection bias. The effect is real but highly task-dependent — it is not evidence that AI coding tools are broadly unhelpful.
Why did developers think AI made them faster when it didn't? METR's study found developers predicted a 24% speedup beforehand and still estimated a 20% speedup afterward, despite being measurably slower during the study. This gap between felt experience and measured outcome is a known pattern in productivity research generally, not something specific to this one study.
Should my company still invest in AI coding assistants? Yes, selectively. The clearest gains show up on unfamiliar code, boilerplate, and well-specified narrow tasks with easily verifiable output — not on deep, familiar-codebase work where a human's own context is already faster than explaining it to a model.
If you want an honest read on where AI-assisted development actually helps your team versus where it's adding overhead disguised as speed, reach out at info@digit.com.pk — we'll tell you the boring truth over the exciting pitch.