Can telemetry tell you if a skill helps?

Can you use OpenTelemetry to tell whether a skill or plugin is actually making your AI better?

Last time I wondered if telemetry could tell you who to hire, which is a big question with a lot of maybes in it. This one is smaller. You install a skill because someone said it helps, and then you just sort of trust it. I wanted to know if the numbers would back that up, so I ran a small test on my own laptop.

The setup

Claude Code can already export OpenTelemetry metrics and events, so most of the work is giving them somewhere to land. I ran the OpenTelemetry Collector in Docker with a file exporter, which writes everything it receives to a JSON file.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  file:
    path: /data/otlp.jsonl

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [file]
    logs:
      receivers: [otlp]
      exporters: [file]

Save that as collector.yaml, make a data folder next to it, and start the collector:

docker run -d --name otel-exp -p 4317:4317 -p 4318:4318 \
  -v "$PWD/collector.yaml:/etc/otelcol-contrib/config.yaml" \
  -v "$PWD/data:/data" \
  otel/opentelemetry-collector-contrib:0.165.0

Claude Code sends its events as logs, which is why the config needs a logs pipeline as well as metrics. Then point Claude Code at it:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_METRIC_EXPORT_INTERVAL=1000
export OTEL_LOGS_EXPORT_INTERVAL=1000
export OTEL_LOG_TOOL_DETAILS=1
export OTEL_RESOURCE_ATTRIBUTES="experiment.arm=with_skill,experiment.run=1"

The export intervals default to a minute for metrics, and a short headless run can finish before then, so I dropped both to a second. And OTEL_RESOURCE_ATTRIBUTES stamps every metric and event with whatever tags you give it, which is how you split the runs into two groups afterwards.

A tea shop with house rules

The test needed a codebase, so I made one up: a tiny orders API for a tea shop, in plain Node with no dependencies. It has two routes, a home-made router, two tests and a changelog. The job was the kind you’d hand an agent on a normal day:

Add an endpoint GET /orders/:id/total that returns the order’s total including 20% VAT. Add tests for it.

The skill came in a small plugin, which is just a folder with a manifest and the skill inside it:

teashop-conventions/
  .claude-plugin/plugin.json
  skills/new-endpoint/SKILL.md

The manifest only needs a name, a version and a description. The skill is a Markdown file, and its description is what Claude reads when it decides whether to use it. This is a cut-down version of mine:

---
name: new-endpoint
description: Use when adding or changing an HTTP endpoint in the teashop orders API. Gives the route layout, response and error shapes, money and VAT rules, test conventions and the changelog rule.
---

# New endpoint

Everything you need is here. Don't explore the repo to rediscover it.

- Order routes go in `orderRoutes` in `src/routes/orders.js`.
  Handlers take `{ params }` and return `{ status, body, headers? }`.
- Errors use `notFound()` from `src/errors.js`.
- Any response with order data sets `cache-control: no-store`.
- Money is integer pence. VAT is 20%, worked out once on the
  order's net total and rounded with `Math.round`, never per line.
- Totals respond with exactly `{ orderId, netPence, vatPence, grossPence }`.
- Tests use `node:test`, one happy path and one 404 per endpoint.
- Add a line to `CHANGELOG.md` under `## Unreleased`.

Most of that is written down in the code already, if you go looking. Four rules I made up for the test: the exact response shape, the no-store header, rounding VAT on the order rather than per line, and the changelog line. The code doesn’t spell any of them out, though the changelog file itself is a hint.

The rounding rule has a trap in it. One of the orders is three 52p biscuits, and 20% of 52p is 10.4p. Round each line and the VAT comes to 30p. Round the order and it comes to 31p.

Then I gave Claude Code the job ten times on Sonnet 5.5, in a fresh copy of the project each time. Five runs loaded the plugin and five didn’t, alternating so neither group got a quieter afternoon. Each run looked like this, with the tags in OTEL_RESOURCE_ATTRIBUTES changed to match:

claude -p "Add an endpoint GET /orders/:id/total that returns the order's total including 20% VAT. Add tests for it." \
  --model sonnet --max-turns 30 \
  --setting-sources project \
  --allowedTools "Read Write Edit Glob Grep Skill Bash(node *) Bash(npm test)" \
  --plugin-dir ./teashop-conventions

The runs without the skill just drop the last line. --setting-sources project keeps my own installed plugins out of both groups. Everything ran headless, so nobody was there to approve anything, and Claude Code’s own permission settings decided what was allowed.

Afterwards, a few lines of Node read the collector’s file, grouped every event by experiment.arm and experiment.run, and added up cost, tokens, model calls and tool calls for each run.

Ten runs, side by side

	Without the skill	With the skill
Cost per run	$0.17 ($0.145 to $0.173)	$0.12 ($0.124 to $0.126)
Calls to the model	6 (5 to 6)	4 (every run)
Tool calls	14 (13 to 15)	10 (every run)
Files read	8 (7 to 9)	4 (every run)
Time	39 seconds (30 to 45)	26 seconds (25 to 28)
Output tokens	4,185	2,708

Those are medians, with the range in brackets. Costs are Claude Code’s own figures at API list prices.

The skill made each run about a quarter cheaper and a third quicker. That’s less than I expected, and the tool calls show why. Claude has to read a file before it can edit it, so even with the skill it read the four files it was about to change: the orders route, its test, the data file and the changelog. Without the skill it started by listing the whole project, then read the README, package.json, the products route and the error helpers as well, just to work out how things were done.

What surprised me was how steady the skill made things. The five runs with it all made the same ten tool calls, and their costs landed within a quarter of a cent of each other. Without it, every run went looking in a slightly different way.

Then I tried Opus

So I wondered whether a bigger model would change the picture, and ran the whole thing again on Opus 5.5: same job, same skill, five runs each way.

	Sonnet, no skill	Sonnet, skill	Opus, no skill	Opus, skill
Cost per run	$0.17	$0.12	$0.25	$0.26
Calls to the model	6	4	6	7
Tool calls	14	10	7	11
Time	39 seconds	26 seconds	40 seconds	40 seconds
Response shapes	4 in 5 runs	1	4 in 5 runs	1

On Opus, the skill saved nothing. It cost a cent more per run, and the time didn’t move at all.

The tool calls explain it. Without the skill, Opus didn’t open files one at a time like Sonnet did. It listed the project, then printed every source file in a single command, and read the whole codebase in one go. With the skill it loaded the skill first and then read four or five files separately, because it still has to read a file before it can edit it. So the skill added round trips, and on Opus that made it slightly dearer.

That only works because the tea shop is tiny. Nobody is reading a real codebase in one command. But it does mean the same skill can save a quarter of the bill on one model and nothing on another, and you’d never know without measuring both.

What the telemetry gave me for free

I didn’t have to instrument anything. Claude Code sent all of this on its own:

1. Skill activated

   Who: skill_activated event

   An event every time a skill loads, including what triggered it. Every run with the skill, on both models, said claude-proactive, so the skill’s description was enough to trigger it and nobody had to type a slash command.

2. Spend by skill

   Who: cost.usagetoken.usage

   Every cost and token data point carries skill.name and plugin.name. Without OTEL_LOG_TOOL_DETAILS=1, my local plugin and its skill both came through as “third-party”, and the skill_activated event just said “custom_skill”. So set it if you want to see your own names.

3. What it did

   Who: tool_resultapi_request

   Each tool call and each model call, with durations, token counts and, with the details flag on, which file it touched. That’s where the list of files read came from, not a guess from the bill.

What it couldn’t tell me

Telemetry says what happened, not whether it was any good. So after each run I ran the project’s tests, called the new endpoint myself, and checked the result against the house rules.

On the things you can work out from the code, both groups were perfect. Every run registered the route, returned the right total for a normal order, returned a proper 404 for a missing one, added tests, and left every test passing.

The made-up rules split them. Without the skill, Sonnet found the changelog on its own and added a line to it every time. It also got the biscuit order right in all five runs, rounding the VAT on the order to 31p. It never added the no-store header, which is fair, because nothing in the code hinted at it. And it invented the response shape. Five runs gave me four different shapes:

{ "id": "A101", "netPence": 156, "vatPence": 31, "totalPence": 187 }
{ "id": "A101", "vatPercent": 20, "netPence": 156, "vatPence": 31, "totalPence": 187 }
{ "orderId": "A101", "netPence": 156, "vatPence": 31, "totalPence": 187 }
{ "subtotalPence": 156, "vatPence": 31, "totalPence": 187 }

Every one of them has the right number in it, and every one would pass its own tests. But if you’re the team building the till that calls this API, four shapes for the same endpoint is a bug. Opus was the same on every count. It found the changelog, got the 31p right and skipped the header. And it gave me four shapes in five runs, two of them tucked inside a total object. With the skill, every run on both models returned the same shape with the header set.

That’s the quality half, and none of it showed up in the telemetry. The cost and tool calls would have looked exactly the same if every response had been wrong.

The bigger worry is what bought the saving in the first place. The skill tells Claude not to explore, and that’s part of why it’s cheaper. But if the skill goes stale, say someone moves the error helpers or renames a field, Claude will follow the old rules without checking. The telemetry would show that run as cheap, quick and consistent, and it would be wrong.

Cheaper and more consistent is a signal. It isn’t the outcome.

It’s the same problem I keep coming back to in the outcome post. Tokens and tool calls are activity. The outcome is an endpoint that’s right, and only a check that knows what right looks like can tell you that. Claude Code’s plugin evals are built for that half: fixed tasks, graders, and a comparison with no plugin loaded. Telemetry covers the other half, which is whether that holds up in a real week of work.

This was a very narrow view of what telemetry can do: twenty runs of one task on two models, on a made-up project with rules I wrote into the skill myself. It was still an interesting experiment, but your real week is messier, and a month of mixed work would tell you more than a neat A/B test ever will. And every event carries your email address and account IDs by default, so think about where you send it before you point it at a shared backend.

Final thought

Yes, you can use OpenTelemetry to see whether a skill is helping, and all it takes is a short collector config and a handful of environment variables. It’ll tell you what the skill changed, and that depends on the model. On Sonnet it meant fewer calls, less reading and a cost that stopped wandering about, and on Opus it barely moved the bill. What it can’t tell you is whether the change was right, so pair it with a check that can. And tag your sessions, because the most useful setting in the whole test was the one that said which group each run was in.