Four metrics today, outcomes on top
Most AI adoption metrics count activity: tokens, pull requests opened, lines suggested. There are four engineering metrics you already have the data for (testing, harness changes, understanding the business and review time), and outcome metrics on top of them that take real work to get.
Every team I talk to wants to know whether AI is working, and how they measure it can be a bit hit and miss.
Tokens spent. Pull requests opened. Lines suggested, lines accepted. Hours saved, usually from a survey. Those numbers are easy to get, and they go up and to the right, which is why they end up on the slide. But they measure activity, so an agent that opens forty pull requests a day, half of which get reverted, looks brilliant.
Add seats and logins and you’ve got the usual adoption dashboard. It tells you people have the tools and are using them, but not whether anything got better. If you’re the one rolling this out, you want to know whether the way the work gets done has actually changed. Those are engineering and implementation metrics. I think they fall into two categories, and the easiest place to start is with the ones you already have the data for.
The second category is the one I argued for in the last post: did the work the business asked for actually get done, with evidence each team can stand behind? I still think that’s the real measure. But outcome metrics need an outcome written down, an owner named and gates agreed, and most organisations don’t have any of that yet, so you can’t wait for them.
The four you can start today
The first category needs no new platform, because it all comes from CI, git and your ticketing system. If you can, tag which changes were AI-assisted so you can compare like with like. Claude Code adds a Co-Authored-By trailer to its commits by default, so a rough split is one command away:
git log --since="3 months ago" --oneline --grep="Co-Authored-By: Claude" | wc -lIt’s rough because people can turn the trailer off, and other tools tag their commits differently or not at all. If you can’t tag anything, measure anyway, because the baseline matters more than the split.
Testing: does the change come with proof?
Count the share of merged changes that add or update a test alongside the code, and your flaky test rate: tests that fail and then pass on a rerun with nothing changed.
I think agents are good at writing code that passes the tests you have, but less good at writing the test that would have caught the thing you didn’t think of. So if the share of changes with tests drops as AI usage goes up, the agent is writing code faster than anyone is proving it works.
Flaky tests matter more than they used to, because an agent that hits a red build will try to fix it. A flaky test sends it off fixing something that isn’t broken, so every flake is wasted tokens and a confused agent.
Harness changes: is anyone teaching the agent?
Change failure rate is the obvious quality metric, and it’s one of the DORA metrics. Honestly, I think DORA belongs to a bygone era, when people wrote every line and the release was the risky part. But change failure rate lags, and when it moves you rarely know whether the agent, the team or the release had anything to do with it.
The harness gives you an earlier signal. Count the changes to CLAUDE.md, AGENTS.md, skills, hooks, agent definitions and harness settings, and who made them. For one team’s repository over the last three months, that’s one command:
git log --since="3 months ago" --format='%an' -- CLAUDE.md AGENTS.md .claude/ | sort | uniq -cIf one person is editing the harness for a team of twelve, that team is running someone else’s setup. If five people are editing it, the team owns its workflow.
Every one of those changes is a lesson written down: the agent got something wrong, someone noticed, and the fix went into the file the agent reads every session. That’s the “mistake made twice” rule from Anthropic’s AI-native SDLC playbook, which I wrote about recently, and it’s the closest thing you’ll find to a quality metric that moves before the incident does.
So what about a CLAUDE.md that nobody has touched in three months? It’s either perfect or ignored, and I’ve never seen a perfect one. Either the team stopped using the agent for real work, or they’re putting up with its mistakes, fixing the same thing by hand every day instead of once. Both are adoption failing quietly, and neither shows up in a token count. The licences are assigned, the dashboard says people are logged in, and the harness has gone stale.
Is a stale harness the earliest signal you’ll get that adoption has stalled? I think it might be.
Treat it as a signal, though, and don’t make it a target. A team churning its CLAUDE.md every day, with nobody agreeing how the agent should behave, has a different problem. What you’re looking for is steady change that follows real work.
Understanding the business: does anyone know why?
Count the share of merged changes that link to a stated intent: a ticket with acceptance criteria, an intent.md, a PRD, anything that says what the change is for in plain words. A pull request that links to a ticket titled “fix bug” doesn’t count.
A link only tells you the ticket exists, though. So each week, pick a handful of merged changes and ask the engineer who shipped them, or the agent’s own summary, what business problem they solved. If the answer is a description of the code, the team is shipping without understanding why.
AI makes this worse by default. When writing the code was slow, people had time to ask why, and when it takes ten minutes, the question gets skipped. It’s the participation gap again: the business knows what done means, and it isn’t in the room where the code gets written. An intent check in the pull request is one way to close it, and once you have one, you can count how often it passes.
Review time: is anyone actually reading it?
Measure the time from a pull request opening to its approval, and look at the spread across the team as well as the average.
Too short, and nobody’s reading. An approval two minutes after a 600-line change opened is a rubber stamp. Whoever signed it off probably doesn’t know what’s in it, and that’s how agent-written code nobody understood ends up in production.
Too long, and nothing’s moving. Changes sit for days while the agents keep producing more, and the queue grows faster than anyone can read it. The agents are busy, but the team isn’t shipping.
The healthy range depends on the size of the change, so track review time against lines changed. If review time falls while changes get bigger, people have stopped reading. If it keeps climbing, a faster agent won’t help. Smaller changes, better pull request summaries or more people who can review will.
Start today
Solves a baseline from tools you already run
- Testing: changes with tests, flaky rate
- Harness: changes to CLAUDE.md, AGENTS.md, skills
- Understanding: work linked to an intent
- Review time: against the size of the change
Work towards
Solves proof the work got done
- Time to outcome
- Trust: gates that returned evidence
- Open agent slots
- Cost per outcome
The ones you need to work on
Outcome metrics measure the whole line of work, across every team it touches. You can’t pull them from git, because git doesn’t know what the work was for.
To get them, you need the work defined first: an outcome in one sentence, a named owner who answers for it, the teams it crosses and the gate each step has to pass. That’s what a Work Pack is, and whatever you call it, it’s organisational work before it’s technical work.
Time to outcome
Time from the thing arriving to the thing being done in the business’s terms. In the CVE example from the last post, that’s from the finding landing to every gate signed off, including legal, the pen test and the threat model, and the pull request merging is somewhere in the middle.
That’s why a team can halve its review time and barely move time to outcome: the work was never waiting on engineering.
Trust
The share of gates that came back with evidence. A run that finished fast with three gates missing their evidence isn’t done. Trust is the number the outcome owner answers for, and it’s the one that tells you whether you can believe the rest.
Open agent slots
Count the steps in a workflow or an outcome that no agent runs yet. In the CVE example, the pen test has a person in it, while the end-of-life check and the threat model have nobody. All three are open agent slots, but only the last two are gaps.
Licences tell you who could use an agent, but open agent slots tell you where the work still doesn’t have one. The number should go down as teams get ready, and if it doesn’t move, your agents are running somewhere other than the work that matters.
Not every slot should close, though. A legal sign-off might stay with a person for good, so mark the slots you’ve decided to keep human and leave them out of the count. Report the slots with nobody in them separately, because those are the steps you can’t prove yet.
Cost per outcome
Tokens, licences, people’s time and vendor spend, divided by outcomes delivered rather than pull requests opened. It’s the hardest of the outcome metrics to get right, and the only cost number that means anything to a finance director.
Someone said to me recently that AI is “the most amount of money we’ve ever spent without a business justification”. They weren’t joking, and I don’t think they’re alone. Most of these budgets were signed on a promise, and the bill is now big enough that someone is going to ask what it bought.
Cost per outcome is the answer to that question. “We spent this much and closed this many vulnerabilities, each one faster than last year” holds up in a budget review, but “we used a lot of tokens” doesn’t. After all, someone’s got to have confidence at renewal time, right? Ideally not just the vendor.
Getting there
Most of what I’ve written about so far is engineering, and that’s only half the picture. Surface areas get complicated quickly, because where work comes from, who contributes and how they qualify success look different in every team. People still treat all of this as an engineering activity, but the same techniques apply to business units and knowledge workers. If Anthropic has its way, with claude.ai and Cowork for the people who aren’t engineers, that’s where most of the agents will end up. And I think managed platforms are deciding to become the answer for that. You don’t need one to start counting the four, but you do need one to keep going. Work that crosses domains and business units needs one place where everyone’s agents, and everyone’s numbers, live. That’s what makes this sustainable, and it’s what gets you from the numbers you have today to the outcomes you want.
So getting there is less a rollout plan than a practical change in how people see the work and take part in it. The four translate outside engineering, too. Does the output get checked? Does anyone update the instructions and skills the agent reads? Does the work link to a reason? Does anyone actually read what comes back? A finance team can answer those for its own work just as well as a platform team can.
Honestly, I think the answer is live mentoring. Someone who’s already working this way sits alongside someone who isn’t, on their real work, as it happens, and shows them where the agent fits, what to check and what to write down when it gets something wrong. It doesn’t scale like a training programme, but it spreads, because the person who was mentored goes on to mentor the next one.
Baseline the four
Who: EngineeringBusiness units
Pull the last three months, and don’t fix anything yet.
Make it sustainable
Who: PlatformBusiness units
Put engineering and the business on the same managed platform, so the four mean the same thing everywhere and the outcomes have somewhere to live.
Pair people up
Who: MentorsKnowledge workers
Live mentoring, on real work. When the agent gets something wrong, they fix it together.
Measure one outcome
Who: Outcome owner
Pick work that crosses teams, name its owner and track it end to end.
Final thought
Start with the four, because you can pull them this week, and write down where you are before you try to improve any of it.
The outcome metrics take longer, because someone has to write down what done means and put their name next to it. But when the two meet, you’ll know whether any of it was for anything.
Getting from one to the other isn’t new, either. A shared platform is what makes it sustainable, and live mentoring is how people have always learned a craft: by sitting next to someone who’s done it. They’re old tricks, and they need to apply to a new world.