Skip to content

Operator's Tape #1 — Same Task, With and Without a CLAUDE.md

A captured experiment, run for real on the Practice Repo. One coding agent (Claude), one task, two conditions. Everything below — the action logs, the diffs, the outcomes — comes from the actual sessions.

The task, verbatim in both runs: "Add a confidence score to the classification output."

The two conditions:

  • Run A — cold start. The bare Triage code. No CLAUDE.md, no README. The agent navigates, decides what done means, and picks its own verification.
  • Run B — operated. Identical code, plus the Great CLAUDE.md from the Module 1 lesson, injected at session start the way Claude Code loads it.

Methodology, honestly

Same agent model, same wording, run concurrently. The README was removed from both runs because it names the exercise files (it would leak the answer to the cold run). Each agent followed the same logging protocol — append one line to a log before every action — so the logs add identical overhead to both arms. This is one run per condition: a demonstration of the mechanism you can reproduce yourself, and the lesson's the Guided Lab has you do exactly that.

The logs, side by side

1. LIST: .
2. LIST: app, eval, tests, web (recursive)
3. READ: app/triage/classify.py
4. READ: app/models.py
5. READ: app/api/routes.py
6. READ: tests/test_classify.py
7. GREP: confidence / classification usage across repo
8. READ: tests/test_api.py
9. READ: web/static/app.js
10. EDIT: app/models.py
11. EDIT: app/triage/classify.py
12. EDIT: tests/test_classify.py
13. EDIT: tests/test_api.py
14. RUN: pytest
15. RUN: manual smoke + python -m eval.run

1. LIST: .
2. LIST: app (recursive)
3. READ: classify.py, models.py, routes.py, eval/run.py
4. READ: test_classify.py, test_api.py, draft.py, conftest.py
5. READ: eval/cases.jsonl (head), test_draft.py
6. EDIT: app/models.py (add confidence field)
7. EDIT: app/triage/classify.py (compute confidence)
8. EDIT: tests/test_classify.py (update confidence test)
9. RUN: pytest
10. RUN: python -m eval.run
11. RUN: sanity check to_dict output

The measurements

Measure Run A (cold) Run B (operated)
Wall clock ~3.5 min ~2.5 min
Total tool actions (harness count) 25 19
Exploration steps before the first edit 9 5
Files changed 4 3
pytest after the change all green all green
python -m eval.run held at baseline held at baseline
Found the eval gate by exploring told by the rule
Tripped by the seeded guard test no — replaced it, flagged the decision no — replaced it, flagged the decision

What to notice

1. The honest headline: the cold agent succeeded too. On a repo this small — a couple dozen files — a strong agent recovers from a missing instructions file by exploring. Context engineering is a lever on cost and reliability rather than a magic on/off switch. What the CLAUDE.md bought here: roughly a third less wall clock, a quarter fewer actions, and about half the pre-edit exploration. Now scale the repo. The exploration tax grows with the tree; the instructions file's cost stays one page. This repo is the floor of the effect, and it still showed up on every measure.

2. Verification as a rule beats verification as a discovery. Both runs ran the eval — but for different reasons. Run A found eval/ while exploring and decided it was relevant. That's good instinct, and it is luck-shaped: on a messier repo, the eval is the thing a cold agent skips. Run B ran it because the CLAUDE.md states the rule: a classifier change requires python -m eval.run to hold before you call it done. The through-line metric of this course — how long can the agent run before it needs you? — moves exactly here: a rule the agent follows unprompted is an intervention you never have to make.

3. The same vague task produced two different designs — and that's a specification lesson. The task never said how to compute confidence. Run A chose one formula (a 0.7 base per hit, capped below 1.0, because keyword matching is never certain); Run B chose another (a higher base, saturating at 1.0). Both defensible, both documented, both different. Nothing in either context could settle it, because the decision belonged to the spec that was never written — which is Module 2's whole argument compressed into one diff.

4. The seeded guard test caught neither agent — and both handled it right. The shipped test suite asserts the confidence field does not exist yet. Both agents noticed the collision, replaced the test deliberately, and flagged the decision in their report instead of silently deleting it. That flag is what you want from an agent touching a test: the change is visible, reviewable, and argued for. An agent that buries it is the Module 3 failure mode.

Reproduce it

The Module 1 exercise is this experiment: run the cold task on your clone, write the CLAUDE.md, run it again in a fresh session, and compare your own two logs. Your numbers will differ; the shape shouldn't.