AI safety research organization METR conducted an experiment with a surprising conclusion: experienced developers using AI coding tools completed tasks 19% slower than those working without them.
Experiment Design
METR recruited a group of experienced software developers and had them complete identical programming tasks under two conditions:
- Experimental group: Free to use AI coding tools
- Control group: No AI tools allowed
Tasks covered common development scenarios including bug fixes, feature development, and code refactoring.
Why It Was Slower
The study identified several key reasons:
1. Review costs
AI-generated code cannot be accepted blindly; it requires line-by-line review. Experienced developers, with high standards for code quality, spent more time reviewing and verifying than the AI saved in writing.
2. Context switching
Frequent switching between thinking independently and reviewing AI output disrupted developers' flow state.
3. Over-reliance tendency
Some developers found themselves spending time "tuning" the AI after receiving suggestions, rather than writing the code directly — which was often faster.
Key Limitations
The study has several important caveats:
- Small sample size
- Limited task types
- The AI tools used may not have been the latest or most capable versions
- "Experienced" developers are already fast, leaving less room for improvement
For junior developers, the efficiency impact of AI tools may be entirely different. METR's study focused on the "experienced" group, and its findings cannot be generalized to all developers.
How to Interpret This
This research does not mean AI coding tools are useless. Rather, it shows that efficiency gains are not automatic. They require learning proper usage, applying tools in appropriate scenarios, and establishing effective human-AI collaboration workflows.
The best developers are not those who "use AI the most," but those who know when to use it and when not to.
Source: CocoLoop, METR research report