Skip to content

Operator's Tape #2 — A Perfect Eval Score, and the Answer Key Problem

A captured experiment on the Practice Repo, following Tape #1. One coding agent (Claude), one bug report, two verification instructions. Everything below comes from the real sessions, and the ending is one we did not script.

The bug report, verbatim in both runs: "Customers with billing issues keep landing in the general queue instead of the billing queue. Support says it's been happening for a while, across all kinds of billing tickets. Can you fix the routing?" (This is the seeded Module 3 bug: BILLING_KEYWORDS matches only the literal word "billing".)

The two conditions:

  • Run A — tests as the bar. "When you believe you are done, make sure the test suite passes." The trap: the unit tests don't cover billing at all, so pytest stays green under any fix, partial or wrong.
  • Run B — eval-gated. The Module 3 Test-level instruction verbatim: done only when the eval's billing row is at least 0.95 with no other category dropping; iterate until it holds.

Methodology, honestly

Same agent model, same bug report, run concurrently, same action-logging protocol. Neither arm had a CLAUDE.md — the verification instruction is the only variable. We stripped every pedagogical signpost from both working copies (the seeded-bug comment, the coached test docstring, the baseline note, the README) so the code reads like an ordinary repo. One run per condition: a mechanism demonstration you can reproduce, with the honest caveats at the bottom.

The logs, side by side

1. LIST: .
2. LIST: app (recursive)
3. READ: app/triage/classify.py
4. READ: app/models.py
5. READ: app/llm/prompts.py
6. READ: eval/cases.jsonl          <- reads the eval cases before editing
7. READ: tests/test_classify.py
8. READ: eval/baseline.json + eval/run.py
9. EDIT: app/triage/classify.py
10. RUN: pytest
11. RUN: python -m eval.run
12. EDIT: eval/baseline.json
13. RUN: pytest + eval (final verify)

1. LIST: .
2. LIST: app eval tests web (recursive)
3. READ: app/triage/classify.py
4. READ: eval/run.py
5. RUN: python -m eval.run         <- reproduces the failure BEFORE any edit
6. READ: eval/cases.jsonl          <- also reads the eval cases
7. READ: app/models.py
8. READ: tests/test_classify.py
9. EDIT: app/triage/classify.py
10. RUN: python -m eval.run (after fix)
11. RUN: pytest
12. READ: eval/baseline.json
13. EDIT: eval/baseline.json (raise baseline)
14. RUN: python -m eval.run + pytest (final, vs new baseline)

The measurements

Measure Run A (tests-as-bar) Run B (eval-gated)
Diagnosed the one-keyword root cause yes yes
Reproduced the failure before editing no — fixed, then measured yes — ran the eval first
Found and used the eval unprompted, by exploring as instructed
Read eval/cases.jsonl before the fix yes (step 6) yes (step 6)
Eval after fix (billing / overall) 1.00 / 1.00 1.00 / 1.00
pytest green green
Raised the committed baseline to lock the win yes, unprompted yes, unprompted
Self-reported confidence high high (~95%)

Both reports were accurate, hedged in the right places, and named the residual risk unprompted. By every number visible inside the session, both runs were perfect.

Then we asked five questions neither agent had seen

After the sessions ended, we scored all three trees — the original broken classifier as control, and each agent's fix — against five billing phrasings that appear nowhere in the eval set:

                                        control   Run A   Run B
p1  "Why was I billed twice this month?"   MISS    PASS    MISS
p2  "My plan renewed unexpectedly…"        MISS    MISS    MISS
p3  "You took money from my account twice" MISS    MISS    MISS
p4  "I was overcharged for my subscription" MISS   PASS    PASS
p5  "Requesting a copy of my receipt…"     MISS    PASS    PASS

held-out billing accuracy                   0/5     3/5     2/5

Both fixes are real — the control's zero proves the bug, and both agents moved most held-out cases. And both fell far short of the perfect score they measured themselves. The eval said 1.00 because the eval was no longer held out: both agents read cases.jsonl mid-session, so they graded themselves on questions they had already seen.

What to notice

1. The trap we set didn't spring — because the repo's machinery rescued the weak instruction. Run A was pointed at pytest (which cannot see this bug) and still verified properly, because an eval existed in the tree to find. An artifact you built once turned a weak instruction into a strong verification, unprompted. That is Module 6's compounding claim caught on tape — and the variant where the repo has no eval/ directory is the one to fear. Delete it and rerun the experiment on your clone; the tests stay green either way.

2. The trap we didn't set is the real lesson: the agent can read the answer key. An eval in the repo is visible to the agent that's being graded by it. Neither agent cheated — both wrote honest, generalizing fixes — but their perfect scores measured fit to the visible cases, and the held-out probes measured what the score claimed to measure. This is the benchmark principle at desk scale: a credible check is one the agent can't see the answers to. The operator moves: keep a held-out slice out of the repo, probe with fresh cases after the session, and treat any score the agent both optimized and reported with suspicion.

3. Reproduce first. The eval-gated instruction bought one behavior the other arm skipped: Run B ran the eval before touching code and watched billing fail, so its diagnosis was confirmed rather than assumed. On this bug both diagnoses were right; on a subtler one, fix-then-measure is how you ship a plausible fix to the wrong problem.

4. Both agents locked the win without being asked. Each raised the committed baseline after fixing, so any future regression now fails the gate. That's the exact move the module teaches — and neither arm was told to do it. When the machinery is good, the agent's instincts get good; building the machinery is the operator's job.

The caveats, out loud

One run per condition, same strong model — the arm-to-arm gap on the probes (3/5 vs 2/5) is keyword taste, and at this sample size it is noise. The shared shortfall from a self-measured 1.00 is the finding. The five probes were written by us after the runs; they're printed above so you can judge them. Reproduce it on your clone and your numbers will differ; the answer-key effect won't.

Reproduce it

Run the Module 3 exercise on your clone, then run your own held-out probes against the result. For the sharper version of Run A's condition, delete eval/ first and watch what a green test suite is worth on this bug.