The first challenge was to get skeptical team members to adopt AI at all. Many technology leaders are looking to measure adoption established KPIs for AI usage. Oh, the irony. The good news is your teams are using AI – a lot of AI.
Gartner recently found that AI costs $200 to $500 per developer per month, with costs expected to rise sharply. Engineers are burning through tokens while demanding access to the newest (and most expensive) models. Software engineers have become so attached to AI that METR had to abandon an AI productivity study because they couldn’t find engineers willing to work without AI. It’s time to stop worrying about adoption and start measuring the value you’re getting in return.
Let’s start with what you’re not getting: laid off. While there have been examples of companies citing AI adoption as the reason behind engineering layoffs, it is becoming clear our robot overlords are being scapegoated. New York’s WARN Act paperwork now requires companies to check a box if a layoff is due to AI. After one year and 160 filings, zero companies claimed an AI-related layoff.
What about time-to-market? DORA’s measurements over time show AI has transitioned from a slight drain on productivity to a slight boost. Slight. And considering that a study by Tidelift and The New Stack found developers spend only 32% of their time writing new code, opportunities to boost coding productivity are already watered down.
Now the challenge with AI is how to measure its impact. The low-hanging fruit is obvious: team velocity, lines of code, number of pull requests, etc. And there may be some value there. But it’s hard to draw a straight line from lines of code per day to EBIT. The key is to take a step back and look at your pain points.
Ask your engineers what actually hurts
Nobody will say coding. They will say documentation. They will say ambiguous requirements and the resulting rework. They will say technical debt still sitting on the backlog from the last release. That list is nothing new to anyone who has spent a decade or two in software.
Let’s start with ambiguous requirements. Every ambiguity caught before design starts is rework that never happens. AI can analyze a full set of requirements for inconsistency, gaps, untestable language, and missing traceability. Are you using user stories instead? Transcribe your grooming sessions and have AI write complete, well-worded stories with clear acceptance criteria.
How about documentation? AI can reverse engineer design docs from code and turn disorganized thoughts into structured drafts, enabling engineers to focus on substance.
And paying down technical debt starts to show real ROI when AI makes short work of refactoring and replacing dated libraries.
AI usage in engineering is soaring - and so is the token bill. (Image generated with AI, cost: 1,250 tokens)Full Spectrum
Measure what moves the needle for the business
Performance against schedule is the most underrated metric available to you. Not whether you shipped fast, but whether you hit committed dates and whether variance against plan is shrinking. Requirements churn and documentation lag are common sources of schedule slip, and both are directly addressable with AI.
Defect rates are key as well, with attention to where defects surface. A defect caught in verification costs a fraction of one caught in the field, so if AI-assisted requirements analysis and test generation are working, your escape rate will fall.
Mean time to resolution on field issues captures what no productivity metric does: how fast your team can diagnose a problem, fix it, verify it, and release. Especially when the original developers are no longer available, or documentation has fallen out of date, AI can quickly analyze code and suggest both where the bug is and how to fix it.
Rethink the team, not the headcount
This is where project cost comes down. The goal is not eliminating the engineers; it’s composing teams differently. When AI handles first-draft documentation, requirements analysis, and test scaffolding, work that once demanded a large team weighted toward senior engineers can be staffed leaner: fewer people, a different seniority mix, materially less manual verification labor.
Protect the rigor on the way in. Anything touching requirements, design documentation, or verification needs a named human reviewer whose approval is recorded – AI output is a draft, not a deliverable! Baseline your metrics before rollout, because you cannot demonstrate improvement against a number you never captured.
Now make the case
Your engineers have already concluded that AI is indispensable. The token bill proves it. What engineering leaders need to do now is explain what it bought. The answer is going to be different in every organization. The challenge is to focus your AI efforts on the areas that will show the greatest payoff.
Jonathan Burk is the software engineering director at Full Spectrum, a medical device development company based in Westborough, MA.
