Agoda's engineering team spent considerable time promoting AI coding tools and then conducted a retrospective. The conclusion is unsettling:
Individual developer productivity improved, but overall project delivery speed barely moved.
This isn't just Agoda's problem. Faros AI aggregated data across companies and produced a striking set of numbers:
Teams with high AI adoption saw PR merge volume increase by 98% and task completion increase by 21%. But at the same time, PR review time increased by 91%.
More output, but slower reviews. The two effects almost cancel each other out.
What the real bottleneck is
Agoda's engineering team concluded that the problem lies in the fact that "coding was never the main bottleneck in software delivery."
AI autocompletion makes writing code faster. But writing code is just one step in the pipeline. Before it, there's requirements analysis, system design, and technical alignment. After it, there's code review, testing, integration, and deployment.
Tripling the speed of the middle step doesn't mean the entire pipeline runs three times faster.
When AI causes a surge in code output, the ability to "digest that code" becomes the new chokepoint:
- Each PR contains more code (AI-generated code is often less concise)
- Code review has more to examine
- But senior engineers' time is limited
The bottleneck has shifted from "slow coding" to "slow code review," and actual delivery pace hasn't improved.
The deeper issue: Specification
Agoda engineer Leonardo Stern goes further, arguing that the truly scarce resource has moved up: it's not review, but specification—the ability to translate vague business requirements into clear, executable technical specifications.
Previously, engineers spent a lot of time writing code, so specifications could be a bit fuzzy because "you'd figure it out as you wrote."
Now, if you throw a vague requirement at AI, it generates code that "looks usable" but goes in the wrong direction. Then you go back to revise the spec, AI regenerates, and you repeat the cycle. On the surface, code output is high, but in reality, you're trial-and-erroring with code, and efficiency is low.
High-fidelity requirement specifications are becoming the core engineering deliverable. Not the code itself, but "how to describe what you want."
Stern's "Gray Box" approach
Stern proposed a working method called the "Gray Box," sitting between two extremes:
- White Box: Review every line of AI-generated code; you are fully responsible for implementation details
- Black Box: Deploy AI-written code directly; you are responsible for the outcome but not the process
- Gray Box: Intervene at two key points—writing precise specs and verifying results against those specs
The core logic of the Gray Box is: you don't need to understand how AI solves a problem, but you must be able to clearly state "what counts as solved" and verify that it is indeed solved.
The essence of this method is to focus engineers' time and attention on the parts AI is worst at—defining intent and verifying results—rather than implementation details.
Where the engineer's role is heading
The traditional software engineer role assumes that "writing good code" is the core competency.
That assumption is now being challenged. When AI can write decent code, where does an engineer's scarcity come from?
Based on practices at companies like Agoda, the answer is moving in this direction:
- People who can translate business problems into precise technical specifications
- People who can design effective verification plans
- People who can judge whether AI-generated code truly solves the right problem
This doesn't mean "the ability to write code is no longer important"—rather, it has shifted from a scarce resource to a baseline requirement, with true differentiation happening further upstream.
In Stern's words: Human authority is migrating from "writing code" to "defining intent." This is an upward shift in abstraction level, not a disappearance of skills.
What management should think about this
For engineering managers, Agoda's research offers a direct action item: Don't measure AI's value by code output.
PR counts, lines of code, and task completions will all look better thanks to AI tools. But if the features delivered haven't increased, haven't gotten faster, and quality hasn't improved, these numbers are just numbers.
The metrics truly worth tracking are: feature launch cycle time, production defect rate, and end-to-end time from requirement to delivery. These numbers tell you whether AI is actually helping the team or just helping the team inflate GitHub stats.
Next time someone presents PR merge volume as ROI for AI tools, you can ask: And what about review time?
Sources: CocoLoop, AI Coding Assistants Haven't Sped Up Delivery Because Coding Was Never the Bottleneck (InfoQ)