3.1 · Verification — The Trust Layer¶

An agent will confidently produce code that looks right and is wrong. The operator has to work with that output, but cannot read every line of it; reading does not scale, and it fails worst exactly when the operator is tired. The way through is to make correctness something a machine checks rather than something a person eyeballs. This lesson covers the deterministic part of that problem: the checks that give a clean pass or fail. When the output has no pass or fail (a prompt that gets better or worse), the same idea moves up a layer to eval-driven development. When the agent can act on a wrong belief, you add guardrails.

Verify with machinery rather than eyeballs. Tests, loops, and gates scale; reading every line does not. The deterministic stack has three rungs: a test makes correctness a fact, a self-correcting loop makes the agent run the check, and a gate makes the check unskippable.

Concept¶

The failure being defended against is specific: output that passes the eye test and is wrong anyway. The three rungs run from weakest to strongest, and each one removes a dependency on the operator remembering to look.

Tests are the spec. The most reliable instruction you can give an agent is a failing test. It turns "make it work", which is unverifiable, into "make this go green", which a machine can check. Write the test — or have the agent write it and confirm yourself that it fails for the right reason — before the implementation. Correctness is then a fact rather than an opinion, and it is not yours to eyeball.
The self-correcting loop. The operator need not be the one who runs the check. Give the agent the command and the bar: run pytest, and if anything fails, read the failure and fix it, repeating until green. The agent runs, reads its own error, fixes, and re-runs. The object of inspection shifts from the output to the loop, and the loop closes itself. This is the largest single leverage move in the module.
Gates that run without you. A self-correcting loop still depends on the agent choosing to run the check. A gate removes the choice: a pre-commit hook or continuous-integration step that blocks the change when the check fails, whether the agent or the operator remembered it. The agent cannot ship past a gate. This is what makes verification survive a tired operator late in the day.

The rungs stack, and each one removes a different dependency on you. By the gate, correctness no longer needs your eye on the diff, your hand on the check, or anyone remembering to run it.

flowchart TB
    E["Eyeball<br/>you read every diff — scales with your attention"]
    T["Rung 1 · Test<br/>correctness becomes a fact a machine checks"]
    L["Rung 2 · Self-correcting loop<br/>the agent runs the check and fixes until green"]
    G["Rung 3 · Gate<br/>the check is unskippable"]
    E -->|"removes: your eye as the judge"| T
    T -->|"removes: you running the check"| L
    L -->|"removes: anyone remembering to look"| G

Eyeball → Test → Gate — the same fix, three rigor levels¶

The same task in Triage: find and fix the bug sending billing tickets to the wrong queue. The gap between Test and Gate is the lesson.

EyeballTestGate

Ask the agent to fix the misrouted tickets. It edits a file, you read the
diff, it looks reasonable, you ship.

Why it fails: you checked the cases you thought of, on a diff you skimmed, once. The change's effect on the other categories is unknown, and nothing stops the next change from silently undoing this one. This is the "I'll carefully review everything" strategy, and it does not scale past today.

Write a test that asserts a billing ticket gets the `billing` label. Confirm
it's RED. Then tell the agent: "make this test pass — run `pytest`, read
failures, fix the behavior (don't edit the test), repeat until green."

Why it is solid: correctness is now a fact the machine checks, the agent owns the loop, and a regression in another category would surface as a different red test instead of shipping blind. Most of the value is here.

Install a pre-commit hook that runs `pytest` (and the eval) and exits non-zero
on failure. The commit is blocked until the checks hold — for the agent and
for you. Reintroduce the bug and watch the commit get refused.

Why it is great: verification no longer depends on anyone remembering to check. The machinery refuses the bad change, which turns a habit into a guarantee.

The through-line of this course is one question: how long can the agent run before it needs you? Verification buys that runway. A test and a loop let the agent check its own work instead of stopping to wait for your eyes, and a gate stops a regression without you watching at all.

Guided Lab¶

Triage ships with a seeded classification bug: billing tickets get labeled general. You'll catch it with a test you didn't have to write the code for, drive it green through a loop the agent runs, then gate the fix so it can't come back. You will not read the buggy code to find it — the machinery will.

On the Practice Repo, work through the following.

Clone Triage, build the offline venv, and confirm the failure is real:
```
git clone https://github.com/mrfelixwong/agentic-engineering-triage triage && cd triage
python3 -m venv .venv && .venv/bin/pip install -e . && .venv/bin/pip install -e '.[dev]'
.venv/bin/pytest -q
```
Expected: one test fails — test_billing_ticket_is_labeled_billing (two parametrized cases) — and the rest pass. That red test is the spec for this lab.
Read only the failing test — leave the app code alone for now. Open tests/test_classify.py and find the test marked RED by design.

Expected: you can state the behavior it demands (a billing ticket must be labeled billing) and what the app does instead (general) — without yet knowing why.
Run the self-correcting loop. Start the agent in the repo root and give it the failing test as the goal — keep it honest, it must not edit the test to pass:
```
claude
```
.venv/bin/pytest -q has a failing test in tests/test_classify.py. Investigate the root cause in the app/ code, fix it, and re-run the tests until they pass. Don't edit the test — find why the behavior is wrong and fix the behavior.

Expected: the agent traces the wrong label to the routing seam in app/triage/classify.py (a single aliased mapping folds billing into general), fixes that one line, and re-runs pytest itself until it's green. You inspected the loop the agent ran rather than the diff it produced.
Verify with both checks yourself — don't trust the agent's say-so:
```
.venv/bin/pytest -q
.venv/bin/python -m eval.run
```
Expected: pytest all green; the eval reports 10/10 = 1.00 and exits 0. (Before the fix the eval read 7/10 = 0.70, below its committed 0.80 baseline — the three billing cases were the misses. That held-out eval is the subject of 3.2.)
Install the gate so the fix can't regress, then prove it fires. Look at the hook before you wire it in:
```
cat scripts/pre-commit
```
Expected: a short shell script that runs pytest and then the eval (python -m eval.run), exiting non-zero if either fails. That is the check the gate will enforce on every commit.

Now install it as the repo's pre-commit hook:
```
ln -sf ../../scripts/pre-commit .git/hooks/pre-commit
```
Then restore the bug deterministically — no code-reading needed, since the buggy value is still what the repo has committed — stage it, and try to commit:
```
git checkout HEAD -- app/triage/classify.py
git add app/triage/classify.py
git commit -m "test gate"
```
Expected: the hook runs pytest and the eval, the checks fail, and git refuses the commit. Reapply the fix and commit cleanly. You did not have to be watching; the repo caught the regression itself.

▶ Coached mode

Want this walked through live? Coached mode walks the lab with you. The /lesson-3-1-verification instructor skill (it ships in practice-repo/triage/.claude/skills/lesson-3-1-verification/) coaches you step by step from a second terminal — it watches your session, stops at each checkpoint, and asks a leading question before it hands you the next move. It never names the bug or edits your repo for you.

On your own codebase, repeat it on a repo you actually work in: take a bug you can reproduce, write a failing test that pins the correct behavior, and hand the agent the loop (run the suite, read failures, fix, repeat until green, and don't touch the test). Then add a pre-commit hook that runs that suite and confirm it refuses a commit you deliberately break.

Keep the artifact¶

This lesson leaves two reusable things in the repo:

a failing-test-then-loop habit you can rerun on any future bug, and
a pre-commit gate that runs your check and blocks the bad change automatically (rung 3).

The gate carries to every future task in the repo; once it is installed, every commit pays for itself. That compounding is Module 6.

Self-check¶

You did it right if:

the seeded bug is caught by a test you wrote, not by you reading code, via a loop the agent ran itself; and
your gate automatically refuses the bad change when you reintroduce it, without you remembering to check.

If you found the bug by reading the diff, or you had to remember to run the check, you built a habit instead of a system; push it up a rung.

Recall¶

Before moving on: why is "I'll carefully review everything it writes" a failing verification strategy at scale — and what replaces it?

Answer

It scales with your attention, which is finite and worst exactly when you're tired; it checks the cases you thought of, once. Replace eyeballs with machinery: a failing test makes correctness a fact, a self-correcting loop makes the agent run the check, and a gate makes the check unskippable.