Would you query a database without an index?

Every result I get out of a model comes down to what’s in its context when it answers. A better prompt, a skill and a bigger window are all ways of managing that context. Companies everywhere are working on it, too. Some put everything into ever bigger context windows, some train models on their own code, and some build retrieval.

The other way is to hand the model very little and let it go looking. That’s what grep and the other tools in a harness do, because they decide what the model reads, one search or one file at a time. So it’s all about managing context at the end of the day, but can tools be enough on their own?

I think a database already answered that. A wrong harness behaves like a full table scan in SQL instead of an index lookup. The query works, and it gives the right answer on your test data. Then the table grows. A database has two things that stop that happening: a schema, which says where the data lives, and an index, which finds the row without reading the rest.

So I built a test to see if a harness behaves the same way, with grep as the index and a CLAUDE.md as the schema.

The setup

A script generates a fake monorepo at three sizes: 50, 500 and 5,000 services. Each service has four files: a base config, a production override, a handler and a README. Every production override sets a retry limit between 1 and 12, except one, which sets 25. That service sits about three-quarters of the way down the alphabet, so an agent reading in order has to get through most of the repo before it finds it.

Every run got the same question:

One service in this repository retries messages 25 times in production before sending them to the dead letter queue. Which service is it? Answer with the service name and the path of the file that sets it.

The service name doesn’t mention retries, so the only way to be sure of the answer is through the contents of the files. The generator is a short Node script, and this is the part that plants it:

const sorted = [...names].sort();
const target = sorted[Math.floor(sorted.length * 0.73)];

for (const name of sorted) {
  const baseRetry = int(1, 9);
  const prodRetry = name === target ? 25 : int(1, 12);
  // ...writes config/base.yaml, config/overrides/prod.yaml, handler.py and README.md
}

Then I ran Claude Code on Sonnet 5.5 with two harnesses, and the only difference between them is one tool:

const tools = arm === "grep" ? "Read,Grep,Glob" : "Read,Glob";
const args = ["-p", "--model", "sonnet", "--setting-sources", "", "--strict-mcp-config",
  "--no-session-persistence", "--output-format", "stream-json", "--verbose",
  "--max-budget-usd", "3", "--tools", tools, "--allowedTools", tools];

const r = spawnSync("claude", args, { cwd: repo, input: QUESTION, encoding: "utf8" });

None of these tools are new. Glob and grep both come from early Unix, long before Claude, and Claude Code just gives the agent a way to use them. Glob lists files by name, and Read opens one file at a time. Grep is the only one that searches what’s inside them. Take it away and the agent can still find the answer, but only by opening files until it hits the right one.

A script checked every reply for the service name and the path of its prod override:

const correct = result.includes(target) &&
  new RegExp(`${target}[\\\\/]config[\\\\/]overrides[\\\\/]prod\\.yaml`).test(result);

Each run was headless, in a clean folder with no settings, hooks or MCP servers. claude -p prints every tool call as it happens, then the turns, tokens and cost at the end, so every number below comes straight from its output. It’s one run per cell, so this is a demonstration rather than a benchmark.

The repo isn’t meant to look like reality either. Real repos are messier than four tidy files per service, and nobody hides one odd retry limit on purpose. What matters is what each set of tools can do as the repo grows, and how closely that follows what SQL does with and without an index.

The first six runs

Services	Harness	Turns	Files read	Time	Cost	Answer
50	Grep	4	1	9s	$0.02	Right
50	No grep	54	51	70s	$0.17	Right
500	Grep	4	1	9s	$0.02	Right
500	No grep	270	263	6¼ min	$0.92	Right
5,000	Grep	4	1	14s	$0.03	Right
5,000	No grep	6	1	22s	$0.05	Gave up

Grep is the index

The grep harness did the same thing at every size: two searches for a retry setting of 25, then one read of the file they found to confirm it. Four turns, one file and 2 or 3 cents, whether the repo had 50 services or 5,000. That’s an index. The cost follows the question, not the size of the table.

Without grep, the agent started out doing what a database does without an index. At 50 services it opened one base config and then every prod override, 51 reads and 54 turns, and it got there. That cost 17 cents, about eight times the grep run, and took 70 seconds instead of nine. Honestly, at 50 services nobody would notice.

At 500 it ran for over six minutes and 270 turns and cost 92 cents, more than 40 times the grep run. And it didn’t read the whole table. It decided retries probably only matter for consumers, workers, syncs and webhooks, read about 260 of their prod overrides, and skipped the rest. It got the right answer because the planted service happened to be a webhook. If it had been an API, that guess would have missed it.

At 5,000 it didn’t try. After four globs and one read it worked out there were about 20,000 files, said its tools couldn’t search file contents, and stopped. It was honest about it, but it gave no answer.

CLAUDE.md is the schema

Each repo then got a CLAUDE.md at its root that says where retry limits live, the kind of note a team writes for new starters. It doesn’t contain the answer:

# Repository notes

Every service lives in `services/<name>/`.

Queue retry limits are set as `queue.max_retries`. The default is in `config/base.yaml`, and each environment can override it in `config/overrides/<env>.yaml`. Production values are in `config/overrides/prod.yaml`.

The runs were the same apart from --setting-sources project, so Claude Code would load the file.

Services	Harness	Turns	Files read	Time	Cost	Answer
50	Grep + CLAUDE.md	2	0	6s	$0.04	Right
50	No grep + CLAUDE.md	52	50	54s	$0.15	Right
500	Grep + CLAUDE.md	3	0	8s	$0.02	Right
500	No grep + CLAUDE.md	388	379	9 min	$1.12	Right
5,000	Grep + CLAUDE.md	3	0	11s	$0.02	Right
5,000	No grep + CLAUDE.md	5	3	15s	$0.04	Gave up

With grep, the schema saved a step. The agent knew the setting was called max_retries, so it searched for that and read the answer straight out of the search results, with no file opened to confirm it. At 50 services one broad search printed every retry line in the repo, which is why that run cost 4 cents rather than 2.

Without grep, the schema didn’t stop the scan. At 50 services it went straight to the prod overrides and opened all 50. At 500 it stopped guessing which kinds of service mattered and read the prod overrides in order: 379 of them over nine minutes, for $1.12. That’s more than the run without a CLAUDE.md, so at 500 the schema made the scan more thorough, not cheaper.

At 5,000 it gave up again, and it blamed the CLAUDE.md. Its first read was config/overrides/prod.yaml at the root of the repo, which doesn’t exist, and its answer said the notes were wrong about where the config lives. A person reads that path as being inside each service. The model read it literally. So a schema only helps if it’s written for the reader that uses it.

A test and a scalable artefact

Every harness passed at 50 services. If that’s where you stopped testing, you’d happily ship any of them. It’s the same trap as a query that does a full table scan on a dev database with a hundred rows: it isn’t wrong, it just hasn’t met production yet.

A test asks whether the answer is right. A scalable artefact has to keep answering as the repo, the docs or the ticket history grows, and the only way to know is to grow it. So run your harness at 10 and 100 times today’s data, and watch turns and tokens rather than just the answer. If they stay flat, you’ve got an index. If they grow with the data, you’ve got a scan, and somewhere above your test data it’ll start guessing or give up.

I’ll admit grep isn’t the best answer either. A better one is probably a research pattern, where a subagent goes looking and hands back just the answer. But one tool from the 1970s was the difference between 3 cents and no answer, and nothing about the model changed.

Final thought

Would you query a database without an index? Nobody would, but a harness without a search tool does exactly that. Can tools be enough, then? Here the tool did nearly all the work. Grep kept every run to 4 cents or less, and the CLAUDE.md only saved it one read. Without grep, the same CLAUDE.md made the scan more thorough, cost more at 500 services, and still gave up at 5,000. Context tells the model where to look, and a tool is what makes looking cheap. So take one harness you’ve built this year, run it against ten times the data, and see which number grows.